#
e6a4b572 |
|
14-Apr-2024 |
Gleb Smirnoff <glebius@FreeBSD.org> |
mbuf: restore m_uiotombuf() feature of returning a zero length mbuf PR: 278340 Fixes: aba79b0f4a3f69f070ace6effd5700d65226bd20
|
#
aba79b0f |
|
08-Apr-2024 |
Gleb Smirnoff <glebius@FreeBSD.org> |
mbuf: provide mc_uiotomc() a function to copy from uio(9) to mchain Implement m_uiotombuf() as a wrapper around mc_uiotomc(). The M_EXTPG is left untouched. The m_uiotombuf() is left as a compat KPI. New code should use either mc_uiotomc() or m_uiotombuf_nomap(). Reviewed by: markj, tuexen Differential Revision: https://reviews.freebsd.org/D44150
|
#
fd01798f |
|
08-Apr-2024 |
Gleb Smirnoff <glebius@FreeBSD.org> |
mbuf: add mc_split() that works on two struct mchain It preserves tail points and all length/memory accounting, so that caller doesn't need to do any extra traversals. It doesn't respect M_PKTHDR but it may be improved if needed. It respects M_EOR, though. First consumer will be the new unix(4) SOCK_STREAM and SOCK_SEQPACKET. Also provide much more simple mc_concat() that glues two chains back. Reviewed by: markj Differentail Revision: https://reviews.freebsd.org/D44148
|
#
f7d5900a |
|
28-Dec-2023 |
John Baldwin <jhb@FreeBSD.org> |
sys: Style fix for M_EXT | M_EXTPG Add a space around the | operator in places testing for either M_EXT or M_EXTPG. Reviewed by: imp, glebius Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D43216
|
#
29363fb4 |
|
23-Nov-2023 |
Warner Losh <imp@FreeBSD.org> |
sys: Remove ancient SCCS tags. Remove ancient SCCS tags from the tree, automated scripting, with two minor fixup to keep things compiling. All the common forms in the tree were removed with a perl script. Sponsored by: Netflix
|
#
685dc743 |
|
16-Aug-2023 |
Warner Losh <imp@FreeBSD.org> |
sys: Remove $FreeBSD$: one-line .c pattern Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
|
#
c414347b |
|
29-Aug-2022 |
Gleb Smirnoff <glebius@FreeBSD.org> |
mbufs: isolate max_linkhdr and max_protohdr handling in the mbuf code o Statically initialize max_linkhdr to default value without relying on domain(9) code doing that. o Statically initialize max_protohdr to a sane value, without relying on TCP being always compiled in. o Retire max_datalen. Set, but not used. o Don't make the domain(9) system responsible in validating these values and updating max_hdr. Instead provide KPI max_linkhdr_grow() and max_protohdr_grow(). o Call max_linkhdr_grow() from IEEE802.11 and max_protohdr_grow() from TCP. Those are the only protocols today that may want to grow. Reviewed by: tuexen Differential revision: https://reviews.freebsd.org/D36376
|
#
9b841b0e |
|
24-Jun-2022 |
Gleb Smirnoff <glebius@FreeBSD.org> |
m_uiotombuf: write total memory length of the allocated chain in pkthdr Data allocated by m_uiotombuf() usually goes into a socket buffer. We are interested in the length of useful data to be added to sb_acc, as well as total memory used by mbufs. The later would be added to sb_mbcnt. Calculating this value at allocation time allows to save on extra traversal of the mbuf chain. Reviewed by: markj Differential revision: https://reviews.freebsd.org/D35301
|
#
b8394039 |
|
07-Jun-2022 |
Hans Petter Selasky <hselasky@FreeBSD.org> |
mbuf(9): Fix size of mbuf for all 32-bit platforms (i386, ARM, PowerPC and RISCV) Do this by reducing the size of the MBUF_PEXT_MAX_PGS, causing "struct mbuf" to be bigger than M_SIZE, and also add a missing padding field to ensure 64-bit alignment. Reviewed by: gallatin@ Reported by: Elliott Mitchell Differential revision: https://reviews.freebsd.org/D35339 MFC after: 1 week Sponsored by: NVIDIA Networking
|
#
4d88d81c |
|
25-May-2022 |
Hans Petter Selasky <hselasky@FreeBSD.org> |
mbuf(9): Implement a leaf network interface field in the mbuf packet header. When packets are received they may traverse several network interfaces like vlan(4) and lagg(9). When doing receive side offloads it is important to know the first network interface entry point, because that is where all offloading is taking place. This makes it possible to track receive side route changes for multiport setups, for example when lagg(9) receives traffic from more than one port. This avoids having to install multiple offloading rules for the same stream. This field works similar to the existing "rcvif" mbuf packet header field. Submitted by: jhb@ Reviewed by: gallatin@ and gnn@ Differential revision: https://reviews.freebsd.org/D35339 Sponsored by: NVIDIA Networking Sponsored by: Netflix
|
#
17cbcf33 |
|
25-Jan-2022 |
Hans Petter Selasky <hselasky@FreeBSD.org> |
mbuf(9): Assert receive mbufs don't carry a send tag. Else we would start leaking reference counts. Discussed with: jhb@ MFC after: 1 week Sponsored by: NVIDIA Networking
|
#
c6c52d8e |
|
26-Dec-2021 |
Alexander Motin <mav@FreeBSD.org> |
kern: Remove CTLFLAG_NEEDGIANT from some more sysctls. MFC after: 2 weeks
|
#
3c39566d |
|
14-Dec-2021 |
Mateusz Guzik <mjg@FreeBSD.org> |
mbuf: plug set-but-not-used vars Sponsored by: Rubicon Communications, LLC ("Netgate")
|
#
ac2b5444 |
|
10-Nov-2021 |
Mark Johnston <markj@FreeBSD.org> |
mbuf: Fix an offset calculation in m_apply_extpg_one() We were not including the requested starting offset in the page offset. Reviewed by: jhb Fixes: 3c7a01d773ac ("Extend m_apply() to support unmapped mbufs.") Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D32922
|
#
dd31400c |
|
04-Nov-2021 |
Hans Petter Selasky <hselasky@FreeBSD.org> |
Factor out flags preserved during mbuf demote into a separate define. This define will later on be used by coming TLS RX hardware offload patches. No functional change intended. Reviewed by: jhb@ MFC after: 1 week Sponsored by: NVIDIA Networking
|
#
a4667e09 |
|
19-Oct-2021 |
Mark Johnston <markj@FreeBSD.org> |
Convert vm_page_alloc() callers to use vm_page_alloc_noobj(). Remove page zeroing code from consumers and stop specifying VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to simply use VM_ALLOC_WAITOK. Similarly, convert vm_page_alloc_domain() callers. Note that callers are now responsible for assigning the pindex. Reviewed by: alc, hselasky, kib MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D31986
|
#
be79f30d |
|
26-Jul-2021 |
John Baldwin <jhb@FreeBSD.org> |
m_dup: Handle unmapped mbufs as an input mbuf. Use m_copydata() instead of a direct bcopy() when copying data out of a source mbuf into a newly-allocated mbuf. PR: 256610 Reported by: Niels Bakker <niels=freebsd@bakker.net> Reviewed by: markj MFC after: 2 weeks
|
#
c2c34ee5 |
|
06-Jul-2021 |
Mateusz Guzik <mjg@FreeBSD.org> |
mbuf: add m_get_raw and m_gethdr_raw The intent is to eliminate the MT_NOINIT flag and consequently a branch from the constructor. Reviewed by: gallatin Sponsored by: Rubicon Communications, LLC ("Netgate") Differential Revision: https://reviews.freebsd.org/D31080
|
#
aa341db3 |
|
25-May-2021 |
John Baldwin <jhb@FreeBSD.org> |
Rename m_unmappedtouio() to m_unmapped_uiomove(). This function doesn't only copy data into a uio but instead is a variant of uiomove() similar to uiomove_fromphys(). Reviewed by: gallatin, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D30444
|
#
3f9dac85 |
|
25-May-2021 |
John Baldwin <jhb@FreeBSD.org> |
Extend m_copyback() to support unmapped mbufs. Reviewed by: gallatin, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D30133
|
#
3c7a01d7 |
|
25-May-2021 |
John Baldwin <jhb@FreeBSD.org> |
Extend m_apply() to support unmapped mbufs. m_apply() invokes the callback function separately on each segment of an unmapped mbuf: the TLS header, individual pages, and the TLS trailer. Reviewed by: markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D30132
|
#
e243367b |
|
12-Feb-2021 |
Konstantin Belousov <kib@FreeBSD.org> |
mbuf: add a way to mark flowid as calculated from the internal headers In some settings offload might calculate hash from decapsulated packet. Reserve a bit in packet header rsstype to indicate that. Add m_adj_decap() that acts similarly to m_adj, but also either clear flowid if it is not marked as inner, or transfer it to the decapsulated header, clearing inner indicator. It depends on the internals of m_adj() that reuses the argument packet header for the result. Use m_adj_decap() for decapsulating vxlan(4) and gif(4) input packets. Reviewed by: ae, hselasky, np Sponsored by: Nvidia Networking / Mellanox Technologies MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D28773
|
#
608c44f9 |
|
22-Feb-2021 |
Mark Johnston <markj@FreeBSD.org> |
m_uiotombuf_nomap(): Stop clearing PG_ZERO in newly allocated pages The caller should not be passing M_ZERO in the first place, so PG_ZERO will not be preserved by the page allocator and clearing it accomplishes nothing. Reviewed by: gallatin, jhb MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D28808
|
#
c2a8fd6f |
|
13-Oct-2020 |
John Baldwin <jhb@FreeBSD.org> |
Permit sending empty fragments for TLS 1.0. Due to a weakness in the TLS 1.0 protocol, OpenSSL will periodically send empty TLS records ("empty fragments"). These TLS records have no payload (and thus a page count of zero). m_uiotombuf_nomap() was returning NULL instead of an empty mbuf, and a few places needed to be updated to treat an empty TLS record as having a page count of "1" as 0 means "no work to do" (e.g. nothing to encrypt, or nothing to mark ready via sbready()). Reviewed by: gallatin Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26729
|
#
6fed89b1 |
|
01-Sep-2020 |
Mateusz Guzik <mjg@FreeBSD.org> |
kern: clean up empty lines in .c and .h files
|
#
edde7a53 |
|
05-Aug-2020 |
Andrey V. Elsukov <ae@FreeBSD.org> |
Add m__getjcl SDT probe. Obtained from: Yandex LLC MFC after: 1 week Sponsored by: Yandex LLC
|
#
f57d153e |
|
07-May-2020 |
Brandon Bergren <bdragon@FreeBSD.org> |
[PowerPC] Fix powerpcspe build failure after r360569 On powerpcspe, vm_paddr_t is 64 bit despite it being a 32 bit platform. Adjust compile time assertion to compensate.
|
#
61664ee7 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
365e8da4 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Mechanically rename MBUF_EXT_PGS_ASSERT() to M_ASSERTEXTPG() to match classical M_ASSERTPKTHDR. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
6edfd179 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
7b6c99d0 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
bccf6e26 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Step 2.5: Stop using 'struct mbuf_ext_pgs' in the kernel itself. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
c4ee38f8 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Step 2.3: Rename mbuf_ext_pg_len() to m_epg_pagelen() that uses mbuf argument. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
7433a5a9 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Start moving into EPG_/epg_ namespace. There is only one flag, but next commit brings in second flag, so let them already be in the future namespace. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
0c103266 |
|
02-May-2020 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
|
#
23feb563 |
|
14-Apr-2020 |
Andrew Gallatin <gallatin@FreeBSD.org> |
KTLS: Re-work unmapped mbufs to carry ext_pgs in the mbuf itself. While the original implementation of unmapped mbufs was a large step forward in terms of reducing cache misses by enabling mbufs to carry more than a single page for sendfile, they are rather cache unfriendly when accessing the ext_pgs metadata and data. This is because the ext_pgs part of the mbuf is allocated separately, and almost guaranteed to be cold in cache. This change takes advantage of the fact that unmapped mbufs are never used at the same time as pkthdr mbufs. Given this fact, we can overlap the ext_pgs metadata with the mbuf pkthdr, and carry the ext_pgs meta directly in the mbuf itself. Similarly, we can carry the ext_pgs data (TLS hdr/trailer/array of pages) directly after the existing m_ext. In order to be able to carry 5 pages (which is the minimum required for a 16K TLS record which is not perfectly aligned) on LP64, I've had to steal ext_arg2. The only user of this in the xmit path is sendfile, and I've adjusted it to use arg1 when using unmapped mbufs. This change is almost entirely mechanical, except that we change mb_alloc_ext_pgs() to no longer allow allocating pkthdrs, the change to avoid ext_arg2 as mentioned above, and the removal of the ext_pgs zone, This change saves roughly 2% "raw" CPU (~59% -> 57%), or over 3% "scaled" CPU on a Netflix 100% software kTLS workload at 90+ Gb/s on Broadwell Xeons. In a follow-on commit, I plan to remove some hacks to avoid access ext_pgs fields of mbufs, since they will now be in cache. Many thanks to glebius for helping to make this better in the Netflix tree. Reviewed by: hselasky, jhb, rrs, glebius (early version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24213
|
#
7029da5c |
|
26-Feb-2020 |
Pawel Biernacki <kaktus@FreeBSD.org> |
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
|
#
3ff65f71 |
|
30-Jan-2020 |
Mateusz Guzik <mjg@FreeBSD.org> |
Remove duplicated empty lines from kern/*.c No functional changes.
|
#
b2dba663 |
|
27-Sep-2019 |
Andrew Gallatin <gallatin@FreeBSD.org> |
kTLS: Fix a bug where we would not encrypt anon data inplace. Software Kernel TLS needs to allocate a new destination crypto buffer when encrypting data from the page cache, so as to avoid overwriting shared clear-text file data with encrypted data specific to a single socket. When the data is anonymous, eg, not tied to a file, then we can encrypt in place and avoid allocating a new page. This fixes a bug where the existing code always assumes the data is private, and never encrypts in place. This results in unneeded page allocations and potentially more memory bandwidth consumption when doing socket writes. When the code was written at Netflix, ktls_encrypt() looked at private sendfile flags to determine if the pages being encrypted where part of the page cache (coming from sendfile) or anonymous (coming from sosend). This was broken internally at Netflix when the sendfile flags were made private, and the M_WRITABLE() check was added. Unfortunately, M_WRITABLE() will always be false for M_NOMAP mbufs, since one cannot just mtod() them. This change introduces a new flags field to the mbuf_ext_pgs struct by stealing a byte from the tls hdr. Note that the current header is still 2 bytes larger than the largest header we support: AES-CBC with explicit IV. We set MBUF_PEXT_FLAG_ANON when creating an unmapped mbuf in m_uiotombuf_nomap() (which is the path that socket writes take), and we check for that flag in ktls_encrypt() when looking for anon pages. Reviewed by: jhb Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21796
|
#
fee2a2fa |
|
09-Sep-2019 |
Mark Johnston <markj@FreeBSD.org> |
Change synchonization rules for vm_page reference counting. There are several mechanisms by which a vm_page reference is held, preventing the page from being freed back to the page allocator. In particular, holding the page's object lock is sufficient to prevent the page from being freed; holding the busy lock or a wiring is sufficent as well. These references are protected by the page lock, which must therefore be acquired for many per-page operations. This results in false sharing since the page locks are external to the vm_page structures themselves and each lock protects multiple structures. Transition to using an atomically updated per-page reference counter. The object's reference is counted using a flag bit in the counter. A second flag bit is used to atomically block new references via pmap_extract_and_hold() while removing managed mappings of a page. Thus, the reference count of a page is guaranteed not to increase if the page is unbusied, unmapped, and the object's write lock is held. As a consequence of this, the page lock no longer protects a page's identity; operations which move pages between objects are now synchronized solely by the objects' locks. The vm_page_wire() and vm_page_unwire() KPIs are changed. The former requires that either the object lock or the busy lock is held. The latter no longer has a return value and may free the page if it releases the last reference to that page. vm_page_unwire_noq() behaves the same as before; the caller is responsible for checking its return value and freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is introduced for use in pmap_extract_and_hold(). It fails if the page is concurrently being unmapped, typically triggering a fallback to the fault handler. vm_page_wire() no longer requires the page lock and vm_page_unwire() now internally acquires the page lock when releasing the last wiring of a page (since the page lock still protects a page's queue state). In particular, synchronization details are no longer leaked into the caller. The change excises the page lock from several frequently executed code paths. In particular, vm_object_terminate() no longer bounces between page locks as it releases an object's pages, and direct I/O and sendfile(SF_NOCACHE) completions no longer require the page lock. In these latter cases we now get linear scalability in the common scenario where different threads are operating on different files. __FreeBSD_version is bumped. The DRM ports have been updated to accomodate the KPI changes. Reviewed by: jeff (earlier version) Tested by: gallatin (earlier version), pho Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20486
|
#
9fb7c918 |
|
21-Aug-2019 |
Mark Johnston <markj@FreeBSD.org> |
Remove manual wire_count adjustments from the unmapped mbuf code. The original code came from a desire to minimize the number of updates to v_wire_count, which prior to r329187 was updated using atomics. However, there is no significant benefit to batching today, so simply allocate pages using VM_ALLOC_WIRED and rely on system accounting. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D21323
|
#
82334850 |
|
28-Jun-2019 |
John Baldwin <jhb@FreeBSD.org> |
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
|
#
fb3bc596 |
|
24-May-2019 |
John Baldwin <jhb@FreeBSD.org> |
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
|
#
50575ce1 |
|
25-Apr-2019 |
Andrew Gallatin <gallatin@FreeBSD.org> |
Track TCP connection's NUMA domain in the inpcb Drivers can now pass up numa domain information via the mbuf numa domain field. This information is then used by TCP syncache_socket() to associate that information with the inpcb. The domain information is then fed back into transmitted mbufs in ip{6}_output(). This mechanism is nearly identical to what is done to track RSS hash values in the inp_flowid. Follow on changes will use this information for lacp egress port selection, binding TCP pacers to the appropriate NUMA domain, etc. Reviewed by: markj, kib, slavash, bz, scottl, jtl, tuexen Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20028
|
#
f8e73c47 |
|
22-May-2018 |
Fabien Thomas <fabient@FreeBSD.org> |
Add a SPD cache to speed up lookups. When large SPDs are used, we face two problems: - too many CPU cycles are spent during the linear searches in the SPD for each packet - too much contention on multi socket systems, since we use a single shared lock. Main changes: - added the sysctl tree 'net.key.spdcache' to control the SPD cache (disabled by default). - cache the sp indexes that are used to perform SP lookups. - use a range of dedicated mutexes to protect the cache lines. Submitted by: Emeric Poupon <emeric.poupon@stormshield.eu> Reviewed by: ae Sponsored by: Stormshield Differential Revision: https://reviews.freebsd.org/D15050
|
#
51369649 |
|
20-Nov-2017 |
Pedro F. Giffuni <pfg@FreeBSD.org> |
sys: further adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 3-Clause license. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Special thanks to Wind River for providing access to "The Duke of Highlander" tool: an older (2014) run over FreeBSD tree was useful as a starting point.
|
#
66234298 |
|
16-Oct-2017 |
Andriy Voskoboinyk <avos@FreeBSD.org> |
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
|
#
07e87a1d |
|
09-Oct-2017 |
Gleb Smirnoff <glebius@FreeBSD.org> |
In mb_dupcl() don't copy full m_ext, to avoid cache miss. Respectively, in mb_free_ext() always use fields from the original refcount holding mbuf (see. r296242) mbuf. Cuts another cache miss from mb_free_ext(). However, treat EXT_EXTREF mbufs differently, since they are different - they don't have a refcount holding mbuf. Provide longer comments in m_ext declaration to explain this change and change from r296242. In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
|
#
f5b7359a |
|
27-Jun-2017 |
Conrad Meyer <cem@FreeBSD.org> |
Fix one more place uio_resid is truncated to int A follow-up to r231949 and r194990. Reported by: pho@ Reviewed by: kib@, markj@ Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D11373
|
#
14984031 |
|
07-Mar-2017 |
Gleb Smirnoff <glebius@FreeBSD.org> |
m_mbuftouio() doesn't modify the mbuf.
|
#
d53d6fa9 |
|
14-Jan-2017 |
Mark Johnston <markj@FreeBSD.org> |
Suppress a warning about m_assertbuf being unused. MFC after: 1 week
|
#
28323add |
|
08-Nov-2016 |
Bryan Drewery <bdrewery@FreeBSD.org> |
Fix improper use of "its". Sponsored by: Dell EMC Isilon
|
#
69a28758 |
|
15-Sep-2016 |
Ed Maste <emaste@FreeBSD.org> |
Renumber license clauses in sys/kern to avoid skipping #3
|
#
cee4a056 |
|
07-Sep-2016 |
Kevin Lo <kevlo@FreeBSD.org> |
In m_devget(), if the data fits in a packet header mbuf, check the amount of data is less than or equal to MHLEN instead of MLEN when placing initial small packet header at end of mbuf. Reviewed by: glebius MFC after: 3 days
|
#
b85f65af |
|
15-Apr-2016 |
Pedro F. Giffuni <pfg@FreeBSD.org> |
kern: for pointers replace 0 with NULL. These are mostly cosmetical, no functional change. Found with devel/coccinelle.
|
#
fddd4f62 |
|
26-Mar-2016 |
Navdeep Parhar <np@FreeBSD.org> |
Plug leak in m_unshare. m_unshare passes on the source mbuf's flags as-is to m_getcl and this results in a leak if the flags include M_NOFREE. The fix is to clear the bits not listed in M_COPYALL before calling m_getcl. M_RDONLY should probably be filtered out too but that's outside the scope of this fix. Add assertions in the zone_mbuf and zone_pack ctors to catch similar bugs. Update netmap_get_mbuf to not pass M_NOFREE to m_getcl. It's not clear what the original code was trying to do but it's likely incorrect. Updated code is no different functionally but it avoids the newly added assertions. Reviewed by: gnn@ Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D5698
|
#
dcd070d8 |
|
24-Mar-2016 |
George V. Neville-Neil <gnn@FreeBSD.org> |
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
|
#
480f4e94 |
|
22-Mar-2016 |
George V. Neville-Neil <gnn@FreeBSD.org> |
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
|
#
56a5f52e |
|
29-Feb-2016 |
Gleb Smirnoff <glebius@FreeBSD.org> |
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
|
#
5e4bc63b |
|
11-Feb-2016 |
Gleb Smirnoff <glebius@FreeBSD.org> |
o Gather all mbuf(9) allocation functions into kern_mbuf.c, and all mbuf(9) manipulation functions into uipc_mbuf.c. This looks like the initial intent, but had diffused in the last decade. o Gather all declarations in mbuf.h in one place and sort them. o Uninline m_clget() and m_cljget(). There are no functional changes in this patch. The patch comes from a larger version, where all mbuf(9) allocation was uninlined, which allowed to make mbuf(9) UMA zones private to kern_mbuf.c. The performance impact of the total uninlining is still unclear, so we are holding on now with larger version. Together with: melifaro, olivier
|
#
2bab0c55 |
|
08-Jan-2016 |
Gleb Smirnoff <glebius@FreeBSD.org> |
New sendfile(2) syscall. A joint effort of NGINX and Netflix from 2013 and up to now. The new sendfile is the code that Netflix uses to send their multiple tens of gigabits of data per second. The new implementation features asynchronous I/O, when I/O operations are launched, but not awaited to be complete. An explanation of why such behavior is beneficial compared to old one is going to be too long for a commit message, so we will skip it here. Additional features of new syscall are extra flags, which provide an application more control over data sent. The SF_NOCACHE flag tells kernel that data shouldn't be cached after it was sent. The SF_READAHEAD() macro allows to specify readahead size in pages. The new syscalls is a drop in replacement. No modifications are required to applications. One can take nginx binary for stable/10 and run it successfully on head. Although SF_NODISKIO lost its original sense, as now sendfile doesn't block, and now means something completely different (tm), using the new sendfile the old way is absolutely safe. Celebrates: Netflix global launch! Sponsored by: Nginx, Inc. Sponsored by: Netflix Relnotes: yes
|
#
86a996e6 |
|
13-Oct-2015 |
Hiren Panchasara <hiren@FreeBSD.org> |
There are times when it would be really nice to have a record of the last few packets and/or state transitions from each TCP socket. That would help with narrowing down certain problems we see in the field that are hard to reproduce without understanding the history of how we got into a certain state. This change provides just that. It saves copies of the last N packets in a list in the tcpcb. When the tcpcb is destroyed, the list is freed. I thought this was likely to be more performance-friendly than saving copies of the tcpcb. Plus, with the packets, you should be able to reverse-engineer what happened to the tcpcb. To enable the feature, you will need to compile a kernel with the TCPPCAP option. Even then, the feature defaults to being deactivated. You can activate it by setting a positive value for the number of captured packets. You can do that on either a global basis or on a per-socket basis (via a setsockopt call). There is no way to get the packets out of the kernel other than using kmem or getting a coredump. I thought that would help some of the legal/privacy concerns regarding such a feature. However, it should be possible to add a future effort to export them in PCAP format. I tested this at low scale, and found that there were no mbuf leaks and the peak mbuf usage appeared to be unchanged with and without the feature. The main performance concern I can envision is the number of mbufs that would be used on systems with a large number of sockets. If you save five packets per direction per socket and have 3,000 sockets, that will consume at least 30,000 mbufs just to keep these packets. I tried to reduce the concerns associated with this by limiting the number of clusters (not mbufs) that could be used for this feature. Again, in my testing, that appears to work correctly. Differential Revision: D3100 Submitted by: Jonathan Looney <jlooney at juniper dot net> Reviewed by: gnn, hiren
|
#
e40e8705 |
|
06-Oct-2015 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Fix regression from r248371. We need to copy packet header to new mbuf. Unlike in the pre-r248371 code, assert that M_PKTHDR is set only on a first mbuf. Reported & tested by: Andriy Voskoboinyk <s3erios gmail.com> Sponsored by: Nginx, Inc.
|
#
640082d4 |
|
06-Oct-2015 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Remove debugging variable from r143761.
|
#
0cbefd30 |
|
08-Aug-2015 |
Alexander V. Chernikov <melifaro@FreeBSD.org> |
Add const-qualifiers for source mbuf argument in m_dup(), m_copym(), m_dup_pkthdr() and m_tag_copy_chain().
|
#
9523d1bf |
|
30-Jun-2015 |
Navdeep Parhar <np@FreeBSD.org> |
Fix leak in tcp_lro_rx. Simply clearing M_PKTHDR isn't enough, any tags hanging off the header need to be freed too. Differential Revision: https://reviews.freebsd.org/D2708 Reviewed by: ae@, hiren@
|
#
089bb672 |
|
07-May-2015 |
Andrey V. Elsukov <ae@FreeBSD.org> |
m_dup() is supposed to give a writable copy of an mbuf chain. It uses m_dup_pkthdr(), that uses M_COPYFLAGS mask to copy m_flags field. If original mbuf chain has M_RDONLY flag, its copy also will have it. Reset this flag explicitly. MFC after: 2 weeks
|
#
ee52391e |
|
17-Feb-2015 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Use anonymous unions and structs to organize shared space in mbuf(9), instead of preprocessor macros. This will make debugger output of 'print *m' exactly match the names we use in code, making life of a kernel hacker way more pleasant. And this also allows to rename struct_m_ext back to m_ext.
|
#
ec9d83dd |
|
17-Feb-2015 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Use anonymous unions to add possibility to put mbufs into queue(3) STAILQs and SLISTs using the same structure field as good old m_next and m_nextpkt linkage occupy. New code is encouraged to use queue(3) macros, instead of implementing the wheel. However, better not to have a mixture of old style and queue(3) in one file or subsystem. Reviewed by: rwatson, rrs, rpaulo Differential Revision: D1499
|
#
3d1a9ed3 |
|
14-Jan-2015 |
Robert Watson <rwatson@FreeBSD.org> |
In order to support ongoing work to implement variable-size mbufs, and more generally make it easier to extend 'struct mbuf in the future', make a number of changes to the data structure: - As we anticipate embedding mbufs headers within variable-size regions of memory in the future, change the definitions of byte arrays embedded in mbufs to be of size [0] rather than [MLEN] and [MHLEN]. In fact, the cxgbe driver already uses 'struct mbuf' on the front of other storage sizes, but we would like the global mbuf allocator do be able to do this as well. - Fold 'struct m_hdr' into 'struct mbuf' itself, eliminating a set of macros that aliased 'mh_foo' field names to 'm_foo' names such as 'm_next'. These present a particular problem as we would like to add new mbuf-header fields -- e.g., 'm_size' -- that, if similarly named via macros, would introduce collisions with many other variable names in the kernel. - Rename 'struct m_ext' to 'struct struct_m_ext' so that we can add compile-time assertions without bumping into the still-extant 'm_ext' macro. - Remove the MSIZE compile-time assertion for 'struct mbuf', but add new assertions for alignment of embedded data arrays (64-bit alignment even on 32-bit platforms), and for the sizes the mbuf header, packet header, and m_ext structure. - Document that these assertions exist in comments in mbuf.h. This change is not intended to cause (non-trivial) behavioural differences, but is a precursor to further mbuf-allocator work. Differential Revision: https://reviews.freebsd.org/D1483 Reviewed by: bz, gnn, np, glebius ("go ahead, I trust you") Sponsored by: EMC / Isilon Storage Division
|
#
a5e9a6d5 |
|
10-Jan-2015 |
Robert Watson <rwatson@FreeBSD.org> |
Garbage collect m_copymdata(), an mbuf utility routine introduced in FreeBSD 7 that has not been used since. It contains a number of unresolved bugs including an inverted bcopy() and incorrect handling of read-only mbufs using internal storage. Removing this unused code is substantially essier than fixing it in order to update it to the coming mbuf world order -- but it can always be restored from revision history if it turns out to prove useful for future work. Pointed out by: jmallett Sponsored by: EMC / Isilon Storage Division
|
#
b66f2a48 |
|
08-Jan-2015 |
Robert Watson <rwatson@FreeBSD.org> |
Replace hand-crafted versions of M_SIZE() and M_START() in uipc_mbuf.c with calls to the centralised macros, reducing direct use of MLEN and MHLEN. Differential Revision: https://reviews.freebsd.org/D1444 Reviewed by: bz Sponsored by: EMC / Isilon Storage Division
|
#
ed6a66ca |
|
05-Jan-2015 |
Robert Watson <rwatson@FreeBSD.org> |
To ease changes to underlying mbuf structure and the mbuf allocator, reduce the knowledge of mbuf layout, and in particular constants such as M_EXT, MLEN, MHLEN, and so on, in mbuf consumers by unifying various alignment utility functions (M_ALIGN(), MH_ALIGN(), MEXT_ALIGN() in a single M_ALIGN() macro, implemented by a now-inlined m_align() function: - Move m_align() from uipc_mbuf.c to mbuf.h; mark as __inline. - Reimplement M_ALIGN(), MH_ALIGN(), and MEXT_ALIGN() using m_align(). - Update consumers around the tree to simply use M_ALIGN(). This change eliminates a number of cases where mbuf consumers must be aware of whether or not mbufs returned by the allocator use external storage, but also assumptions about the size of the returned mbuf. This will make it easier to introduce changes in how we use external storage, as well as features such as variable-size mbufs. Differential Revision: https://reviews.freebsd.org/D1436 Reviewed by: glebius, trasz, gnn, bz Sponsored by: EMC / Isilon Storage Division
|
#
651e4e6a |
|
30-Nov-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Merge from projects/sendfile: extend protocols API to support sending not ready data: o Add new flag to pru_send() flags - PRUS_NOTREADY. o Add new protocol method pru_ready(). Sponsored by: Nginx, Inc. Sponsored by: Netflix
|
#
7ee2d058 |
|
04-Sep-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Change a very strange code in m_demote() to simple assertion. Sponsored by: Nginx, Inc.
|
#
1967edba |
|
04-Sep-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Provide m_catpkt(), a wrapper around m_cat() that deals with M_PKTHDR mbufs. Sponsored by: Netflix Sponsored by: Nginx, Inc.
|
#
c71b4037 |
|
18-Jul-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Use assignment instead of bcopy. Submitted by: jmg
|
#
1fbe6a82 |
|
11-Jul-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Improve reference counting of EXT_SFBUF pages attached to mbufs. o Do not use UMA refcount zone. The problem with this zone is that several refcounting words (16 on amd64) share the same cache line, and issueing atomic(9) updates on them creates cache line contention. Also, allocating and freeing them is extra CPU cycles. Instead, refcount the page directly via vm_page_wire() and the sfbuf via sf_buf_alloc(sf_buf_page(sf)) [1]. o Call refcounting/freeing function for EXT_SFBUF via direct function call, instead of function pointer. This removes barrier for CPU branch predictor. o Do not cleanup the mbuf to be freed in mb_free_ext(), merely to satisfy assertion in mb_dtor_mbuf(). Remove the assertion from mb_dtor_mbuf(). Use bcopy() instead of manual assignments to copy m_ext in mb_dupcl(). [1] This has some problems for now. Using sf_buf_alloc() merely to increase refcount is expensive, and is broken on sparc64. To be fixed. Sponsored by: Netflix Sponsored by: Nginx, Inc.
|
#
fcc34a23 |
|
11-Jul-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Fix style bug: rename the refcount field of m_ext to ext_cnt, to match other members. Sponsored by: Nginx, Inc.
|
#
15c28f87 |
|
11-Jul-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
All mbuf external free functions never fail, so let them be void. Sponsored by: Nginx, Inc.
|
#
c46713e6 |
|
30-May-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Whitespace only.
|
#
94985f74 |
|
16-Jan-2014 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Remove historical macro. Sponsored by: Nginx, Inc.
|
#
77badb18 |
|
14-Nov-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Fix a very bad typo from r248887. Submitted by: art
|
#
f729ede6 |
|
27-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
Pad m_hdr on 32bit architectures to to prevent alignment and padding problems with the way MLEN, MHLEN, and struct mbuf are set up. CTASSERT's are provided to detect such issues at compile time in the future. The #define MLEN and MHLEN calculation do not take actual compiler- induced alignment and padding inside the complete struct mbuf into account. Accordingly appropriate attention is required when changing members of struct mbuf. Ideally one would calculate MLEN as (MSIZE - sizeof(((struct mbuf *)0)->m_hdr) but that doesn't work as the compiler refuses to operate on an as of yet incomplete structure. In particular ARM 32bit has more strict alignment requirements which caused 4 bytes of padding between m_hdr and pkthdr in struct mbuf because of the 64bit members in pkthdr. This wasn't picked up by MLEN and MHLEN causing an overflow of the mbuf provided data storage by overestimating its size. I386 didn't show this problem because it handles unaligned access just fine, albeit at a small performance penalty. On 64bit architectures the struct mbuf layout is 64bit aligned in all places. Reported by: Thomas Skibo <ThomasSkibo-at-sbcglobal-dot-net> Tested by: tuexen, ian, Thomas Skibo (extended patch) Sponsored by: The FreeBSD Foundation
|
#
bb25e5ab |
|
25-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
Give (*ext_free) an int return value allowing for very sophisticated external mbuf buffer management capabilities in the future. For now only EXT_FREE_OK is defined with current legacy behavior. Sponsored by: The FreeBSD Foundation
|
#
1b4381af |
|
24-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
Restructure the mbuf pkthdr to make it fit for upcoming capabilities and features. The changes in particular are: o Remove rarely used "header" pointer and replace it with a 64bit protocol/ layer specific union PH_loc for local use. Protocols can flexibly overlay their own 8 to 64 bit fields to store information while the packet is worked on. o Mechanically convert IP reassembly, IGMP/MLD and ATM to use pkthdr.PH_loc instead of pkthdr.header. o Extend csum_flags to 64bits to allow for additional future offload information to be carried (e.g. iSCSI, IPsec offload, and others). o Move the RSS hash type enumerator from abusing m_flags to its own 8bit rsstype field. Adjust accessor macros. o Add cosqos field to store Class of Service / Quality of Service information with the packet. It is not yet supported in any drivers but allows us to get on par with Cisco/Juniper in routing applications (plus MPLS QoS) with a modernized ALTQ. o Add four 8 bit fields l[2-5]hlen to store the relative header offsets from the start of the packet. This is important for various offload capabilities and to relieve the drivers from having to parse the packet and protocol headers to find out location of checksums and other information. Header parsing in drivers is a lot of copy-paste and unhandled corner cases which we want to avoid. o Add another flexible 64bit union to map various additional persistent packet information, like ether_vtag, tso_segsz and csum fields. Depending on the csum_flags settings some fields may have different usage making it very flexible and adaptable to future capabilities. o Restructure the CSUM flags to better signify their outbound (down the stack) and inbound (up the stack) use. The CSUM flags used to be a bit chaotic and rather poorly documented leading to incorrect use in many places. Bring clarity into their use through better naming. Compatibility mappings are provided to preserve the API. The drivers can be corrected one by one and MFC'd without issue. o The size of pkthdr stays the same at 48/56bytes (32/64bit architectures). Sponsored by: The FreeBSD Foundation
|
#
9a736876 |
|
24-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
Add an mbuf pointer parameter to (*ext_free) to give the external free function access to the mbuf the external memory was attached to. Mechanically adjust all users to include the mbuf parameter. This fixes a long standing annoyance for external free functions. Before one had to sacrifice one of the argument pointers for this. Sponsored by: The FreeBSD Foundation
|
#
894734cb |
|
24-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
dd a 24 bits wide ext_flags field to m_ext by reducing ext_type to 8 bits. ext_type is an enumerator and the number of types we have is a mere dozen. A couple of ext_types are renumbered to fit within 8 bits. EXT_VENDOR[1-4] and EXT_EXP[1-4] types for vendor-internal and experimental local mapping. The ext_flags field is currently unused but has a couple of flags already defined for future use. Again vendor and experimental flags are provided for local mapping. EXT_FLAG_BITS is provided for the printf(9) %b identifier. Initialize and copy ext_flags in the relevant mbuf functions. Improve alignment and packing of struct m_ext on 32 and 64 archs by carefully sorting the fields.
|
#
1227f20d |
|
21-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
Revert r254520 and resurrect the M_NOFREE mbuf flag and functionality. Requested by: np, grehan
|
#
aa3cb8fb |
|
19-Aug-2013 |
Andre Oppermann <andre@FreeBSD.org> |
Remove the unused M_NOFREE mbuf flag. It didn't have any in-tree users for a very long time, if ever. Should such a functionality ever be needed again the appropriate and much better way to do it is through a custom EXT_SOMETHING external mbuf type together with a dedicated *ext_free function. Discussed with: trociny, glebius
|
#
59b9c4f2 |
|
14-Jul-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Nuke mbstat. It wasn't used for mbuf statistics since FreeBSD 5. Now that r253351 moved sendfile() stats to a separate struct, the last field used in mbstat is m_mcfail, which is updated, but never read or obtained from userland.
|
#
21f39848 |
|
29-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Fix bug in m_split() in a case when split len matches len of the first mbuf, and the first mbuf is M_PKTHDR. PR: kern/176144 Submitted by: Jacques Fourie <jacques.fourie gmail.com>
|
#
4f67e143 |
|
17-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
In m_align() add assertions that mbuf is virgin, similar to assertions in M_ALIGN(), MH_ALIGN, MEXT_ALIGN() macros.
|
#
c95be8b5 |
|
16-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
|
#
5368b81e |
|
16-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Contrary to what the deleted comment said, the m_move_pkthdr() will not smash the M_EXT and data pointer, so it is safe to pass an mbuf with external storage procuded by m_getcl() to m_move_pkthdr(). Reviewed by: andre Sponsored by: Nginx, Inc.
|
#
3112ae76 |
|
15-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Make m_get2() never use clusters that are bigger than PAGE_SIZE. Requested by: andre, jhb Sponsored by: Nginx, Inc.
|
#
41a7572b |
|
12-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Functions m_getm2() and m_get2() have different order of arguments, and that can drive someone crazy. While m_get2() is young and not documented yet, change its order of arguments to match m_getm2(). Sorry for churn, but better now than later.
|
#
8c629bdf |
|
11-Mar-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
The m_extadd() can fail due to memory allocation failure, thus: - Make it return int, not void. - Add wait parameter. - Update MEXTADD() macro appropriately, defaults to M_NOWAIT, as before this change. Sponsored by: Nginx, Inc.
|
#
29110f87 |
|
24-Jan-2013 |
Gleb Smirnoff <glebius@FreeBSD.org> |
- Move large functions m_getjcl() and m_get2() to kern/uipc_mbuf.c - style(9) fixes to mbuf.h Reviewed by: bde
|
#
eb1b1807 |
|
05-Dec-2012 |
Gleb Smirnoff <glebius@FreeBSD.org> |
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
|
#
a2c36a02 |
|
29-Oct-2012 |
Kevin Lo <kevlo@FreeBSD.org> |
Since the macro dtom() has been removed, fix comments about the dtom. Reviewed by: glebius
|
#
14d7c5b1 |
|
28-Oct-2012 |
Andre Oppermann <andre@FreeBSD.org> |
Improve m_cat() by being able to also merge contents from M_EXT mbuf's by doing proper testing with M_WRITABLE(). In m_collapse() replace an incomplete manual check for M_RDONLY with the M_WRITABLE() macro that also tests for shared buffers and other cases that make a particular mbuf immutable. MFC after: 2 weeks
|
#
526d0bd5 |
|
20-Feb-2012 |
Konstantin Belousov <kib@FreeBSD.org> |
Fix found places where uio_resid is truncated to int. Add the sysctl debug.iosize_max_clamp, enabled by default. Setting the sysctl to zero allows to perform the SSIZE_MAX-sized i/o requests from the usermode. Discussed with: bde, das (previous versions) MFC after: 1 month
|
#
7e949c46 |
|
26-Jan-2012 |
Kenneth D. Merry <ken@FreeBSD.org> |
Xen netback driver rewrite. share/man/man4/Makefile, share/man/man4/xnb.4, sys/dev/xen/netback/netback.c, sys/dev/xen/netback/netback_unit_tests.c: Rewrote the netback driver for xen to attach properly via newbus and work properly in both HVM and PVM mode (only HVM is tested). Works with the in-tree FreeBSD netfront driver or the Windows netfront driver from SuSE. Has not been extensively tested with a Linux netfront driver. Does not implement LRO, TSO, or polling. Includes unit tests that may be run through sysctl after compiling with XNB_DEBUG defined. sys/dev/xen/blkback/blkback.c, sys/xen/interface/io/netif.h: Comment elaboration. sys/kern/uipc_mbuf.c: Fix page fault in kernel mode when calling m_print() on a null mbuf. Since m_print() is only used for debugging, there are no performance concerns for extra error checking code. sys/kern/subr_scanf.c: Add the "hh" and "ll" width specifiers from C99 to scanf(). A few callers were already using "ll" even though scanf() was handling it as "l". Submitted by: Alan Somers <alans@spectralogic.com> Submitted by: John Suykerbuyk <johns@spectralogic.com> Sponsored by: Spectra Logic MFC after: 1 week Reviewed by: ken
|
#
6bccea7c |
|
21-Feb-2011 |
Rebecca Cran <brucec@FreeBSD.org> |
Fix typos - remove duplicate "the". PR: bin/154928 Submitted by: Eitan Adler <lists at eitanadler.com> MFC after: 3 days
|
#
a7d5f7eb |
|
19-Oct-2010 |
Jamie Gritton <jamie@FreeBSD.org> |
A new jail(8) with a configuration file, to replace the work currently done by /etc/rc.d/jail.
|
#
1a996ed1 |
|
18-Jul-2010 |
Edward Tomasz Napierala <trasz@FreeBSD.org> |
Revert r210225 - turns out I was wrong; the "/*-" is not license-only thing; it's also used to indicate that the comment should not be automatically rewrapped. Explained by: cperciva@
|
#
805cc58a |
|
18-Jul-2010 |
Edward Tomasz Napierala <trasz@FreeBSD.org> |
The "/*-" comment marker is supposed to denote copyrights. Remove non-copyright occurences from sys/sys/ and sys/kern/.
|
#
d5406806 |
|
12-Jul-2010 |
Colin Percival <cperciva@FreeBSD.org> |
Correctly copy the M_RDONLY flag when duplicating a reference to an mbuf external buffer. Approved by: so (cperciva) Approved by: re (kensmith) Security: FreeBSD-SA-10:07.mbuf
|
#
32a8b1d8 |
|
12-Jul-2010 |
Colin Percival <cperciva@FreeBSD.org> |
Correctly copy the M_RDONLY flag when duplicating a reference to an mbuf external buffer. Approved by: so (cperciva) Approved by: re (kensmith) Security: FreeBSD-SA-10:07.mbuf
|
#
60ae52f7 |
|
21-Jun-2010 |
Ed Schouten <ed@FreeBSD.org> |
Use ISO C99 integer types in sys/kern where possible. There are only about 100 occurences of the BSD-specific u_int*_t datatypes in sys/kern. The ISO C99 integer types are used here more often.
|
#
ee17e64f |
|
06-May-2010 |
Marko Zec <zec@FreeBSD.org> |
MFC r207475: Remove a redundant variable assignment. Reviewed by: bz, rwatson
|
#
a83baab6 |
|
01-May-2010 |
Marko Zec <zec@FreeBSD.org> |
Remove a redundant variable assignment. Reviewed by: bz, rwatson MFC after: 3 days
|
#
bc05b2f6 |
|
22-Jun-2009 |
Andre Oppermann <andre@FreeBSD.org> |
Add m_mbuftouio() helper function to copy(out) an arbitrary long mbuf chain into an arbitrary large uio in a single step. It is a functional mirror image of m_uiotombuf(). This function is supposed to be used instead of hand rolled code with the same purpose and to concentrate it into one place for potential further optimization or hardware assistance.
|
#
0cb6db1d |
|
22-Jun-2009 |
Andre Oppermann <andre@FreeBSD.org> |
Update m_demote: - remove HT_HEADER test (MT_HEADER == MT_DATA for some time now) - be more pedantic about m_nextpkt in other than first mbuf - update m_flags to be retained
|
#
bcf11e8d |
|
05-Jun-2009 |
Robert Watson <rwatson@FreeBSD.org> |
Move "options MAC" from opt_mac.h to opt_global.h, as it's now in GENERIC and used in a large number of files, but also because an increasing number of incorrect uses of MAC calls were sneaking in due to copy-and-paste of MAC-aware code without the associated opt_mac.h include. Discussed with: pjd
|
#
e499bd28 |
|
02-Jun-2009 |
Robert Watson <rwatson@FreeBSD.org> |
Remove unneeded include. MFC after: 3 days
|
#
129c5c81 |
|
18-Jan-2009 |
Alexander Motin <mav@FreeBSD.org> |
Teach m_copyback() to use trailing space of the last mbuf in chain.
|
#
d7f03759 |
|
19-Oct-2008 |
Ulf Lilleengen <lulf@FreeBSD.org> |
- Import the HEAD csup code which is the basis for the cvsmode work.
|
#
9128ec21 |
|
04-Sep-2008 |
Andrew Thompson <thompsa@FreeBSD.org> |
Remove the alignment of the align parameter. This is up to the caller to pass in and it breaks tap(4) on strict alignment machines as m_uiotombuf is called with ETHER_ALIGN. Found by: Jared Go Reviewed by: emax MFC after: 3 days
|
#
2182c0cf |
|
30-Apr-2008 |
Julian Elischer <julian@FreeBSD.org> |
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
|
#
6eeac1d9 |
|
29-Apr-2008 |
Julian Elischer <julian@FreeBSD.org> |
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
|
#
ea26d587 |
|
25-Mar-2008 |
Ruslan Ermilov <ru@FreeBSD.org> |
Replaced the misleading uses of a historical artefact M_TRYWAIT with M_WAIT. Removed dead code that assumed that M_TRYWAIT can return NULL; it's not true since the advent of MBUMA. Reviewed by: arch There are ongoing disputes as to whether we want to switch to directly using UMA flags M_WAITOK/M_NOWAIT for mbuf(9) allocation.
|
#
cf827063 |
|
01-Feb-2008 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Give MEXTADD() another argument to make both void pointers to the free function controlable, instead of passing the KVA of the buffer storage as the first argument. Fix all conventional users of the API to pass the KVA of the buffer as the first argument, to make this a no-op commit. Likely break the only non-convetional user of the API, after informing the relevant committer. Update the mbuf(9) manual page, which was already out of sync on this point. Bump __FreeBSD_version to 800016 as there is no way to tell how many arguments a CPP macro needs any other way. This paves the way for giving sendfile(9) a way to wait for the passed storage to have been accessed before returning. This does not affect the memory layout or size of mbufs. Parental oversight by: sam and rwatson. No MFC is anticipated.
|
#
eeb76a18 |
|
17-Jan-2008 |
Sam Leffler <sam@FreeBSD.org> |
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
|
#
457869b9 |
|
06-Oct-2007 |
Kip Macy <kmacy@FreeBSD.org> |
This patch adds an M_NOFREE flag which allows one to mark an mbuf as not being independently freeable. This allows one to embed an mbuf in the cluster itself. This confers the benefits of the packet zone on all cluster sizes. Embedded mbufs currently suffer from the same limitation that packet zone mbufs do in that one cannot disconnect them and pass them around independently of the cluster. It would likely be possible to eliminate this limitation in the future by adding a second reference for the mbuf itself. Approved by: re(gnn)
|
#
d19e16a7 |
|
16-May-2007 |
Robert Watson <rwatson@FreeBSD.org> |
Generally migrate to ANSI function headers, and remove 'register' use.
|
#
21ee3e7a |
|
14-Apr-2007 |
Kip Macy <kmacy@FreeBSD.org> |
remove now invalid check from m_sanity panic on m_sanity check failure with INVARIANTS
|
#
7c32173b |
|
22-Jan-2007 |
Andre Oppermann <andre@FreeBSD.org> |
Unbreak writes of 0 bytes. Zero byte writes happen when only ancillary control data but no payload data is passed. Change m_uiotombuf() to return at least one empty mbuf if the requested length was zero. Add comment to sosend_dgram and sosend_generic(). Diagnoses by: jhb Regression test by: rwatson Pointy hat to. andre
|
#
5288989f |
|
21-Dec-2006 |
Randall Stewart <rrs@FreeBSD.org> |
The prepend function did not handle non-pkthdr's correctly. It always called MH_ALIGN for small lengths being prepended (less than MHLEN). This meant that if you did a prepend on a non M_PKTHDR the system would panic with the KASSERT in MH_ALIGN. Instead we are not aware of this and do a MH_ALIGN or M_ALIGN as appropriate. Reviewed by: andre Approved by: gnn
|
#
5e20f43d |
|
02-Nov-2006 |
Andre Oppermann <andre@FreeBSD.org> |
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
|
#
aed55708 |
|
22-Oct-2006 |
Robert Watson <rwatson@FreeBSD.org> |
Complete break-out of sys/sys/mac.h into sys/security/mac/mac_framework.h begun with a repo-copy of mac.h to mac_framework.h. sys/mac.h now contains the userspace and user<->kernel API and definitions, with all in-kernel interfaces moved to mac_framework.h, which is now included across most of the kernel instead. This change is the first step in a larger cleanup and sweep of MAC Framework interfaces in the kernel, and will not be MFC'd. Obtained from: TrustedBSD Project Sponsored by: SPARTA
|
#
adf5d1c6 |
|
21-Sep-2006 |
Randall Stewart <rrs@FreeBSD.org> |
atomic_fetchadd_int is used by mb_free_ext(), but it returns the previous value that the "add" effected (In this case we are adding -1), afterwhich we compare it to '0'... to see if we free the mbuf... we should be comparing it to '1'... Note that this only effects when there is contention since there is a first part to the comparison that checks to see if its '1'. So this bug would only crop up if two CPU's are trying to free the same mbuf refcount at the same time. This will happen in SCTP but I doubt can happen in TCP or UDP. PR: N/A Submitted by: rrs Reviewed by: gnn,sam Approved by: gnn,sam
|
#
b37ffd31 |
|
10-Jun-2006 |
Robert Watson <rwatson@FreeBSD.org> |
Move some functions and definitions from uipc_socket2.c to uipc_socket.c: - Move sonewconn(), which creates new sockets for incoming connections on listen sockets, so that all socket allocate code is together in uipc_socket.c. - Move 'maxsockets' and associated sysctls to uipc_socket.c with the socket allocation code. - Move kern.ipc sysctl node to uipc_socket.c, add a SYSCTL_DECL() for it to sysctl.h and remove lots of scattered implementations in various IPC modules. - Sort sodealloc() after soalloc() in uipc_socket.c for dependency order reasons. Statisticize soalloc() and sodealloc() as they are now required only in uipc_socket.c, and are internal to the socket implementation. After this change, socket allocation and deallocation is entirely centralized in one file, and uipc_socket2.c consists entirely of socket buffer manipulation and default protocol switch functions. MFC after: 1 month
|
#
47e2996e |
|
15-Mar-2006 |
Sam Leffler <sam@FreeBSD.org> |
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
|
#
45e0d0aa |
|
13-Mar-2006 |
John-Mark Gurney <jmg@FreeBSD.org> |
spell pdata correctly, we now will only dump maxlen of each mbuf in the chain, instead of the entire mbuf... This should probably be reworked so that it prints at max maxlen bytes for the entire chain...
|
#
67c0796c |
|
07-Mar-2006 |
John Baldwin <jhb@FreeBSD.org> |
For consistency sake, use >= MINCLSIZE rather than > MINCLSIZE to determine whether or not to allocate a full mbuf cluster rather than just a plain mbuf when adding on additional mbufs in m_getm(). In practice, there wasn't any resulting mem trashing since m_getm() doesn't ever allocate an mbuf with a packet header, and MINCLSIZE is the available payload in an mbuf with a header rather than the available payload in a plain mbuf. Discussed with: andre (lightly)
|
#
80444f88 |
|
18-Feb-2006 |
Andre Oppermann <andre@FreeBSD.org> |
The sysctls kern.ipc.[max_linkhdr|max_protohdr|max_hdr|max_datalen] can't be changed from userland. Make them read-only and provide descriptions. kern.ipc.max_datalen must never be less than one byte. Enforce this with a panic in net_init_domain(). Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 days
|
#
ec63cb90 |
|
17-Feb-2006 |
Andre Oppermann <andre@FreeBSD.org> |
Replace the 4k fixed sized jumbo mbuf clusters with PAGE_SIZE sized jumbo mbuf clusters. To make the variable size clear they are named MJUMPAGESIZE. Having jumbo clusters with the native PAGE_SIZE is more useful than a fixed 4k size according the device driver writers using this API. The 9k and 16k jumbo mbuf clusters remain unchanged. Requested by: glebius, gallatin Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 days
|
#
63e6f390 |
|
14-Dec-2005 |
Ed Maste <emaste@FreeBSD.org> |
When using m_dup(9) to copy more than MHLEN bytes of data, don't create an mbuf chain that starts with a cluster containing just MHLEN bytes. This happened because m_dup called m_get or m_getcl depending on the amount of data to copy, but then always set the size available in the first mbuf to MHLEN. Submitted by: Matt Koivisto <mkoivisto at sandvine dot com> Approved by: jmg Silence from: rwatson (mentor)
|
#
d5269a63 |
|
08-Dec-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Add an API for jumbo mbuf cluster allocation and also provide 4k clusters in addition to 9k and 16k ones. struct mbuf *m_getjcl(int how, short type, int flags, int size) void *m_cljget(struct mbuf *m, int how, int size) m_getjcl() returns an mbuf with a cluster of the specified size attached like m_getcl() does for 2k clusters. m_cljget() is different from m_clget() as it can allocate clusters without attaching them to an mbuf. In that case the return value is the pointer to the cluster of the requested size. If an mbuf was specified, it gets the cluster attached to it and the return value can be safely ignored. For size both take MCLBYTES, MJUM4BYTES, MJUM9BYTES, MJUM16BYTES. Reviewed by: glebius Tested by: glebius Sponsored by: TCP/IP Optimization Fundraise 2005
|
#
cd5bb63b |
|
05-Nov-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Free only those mbuf+clusters back to the packet zone that were allocated from there. All others get broken up and free'd individually to the mbuf and cluster zones. The packet zone is a secondary zone to the mbuf zone. There is currently a limitation in UMA which prevents decreasing the packet zone stock when the mbuf and cluster zone are drained and all their members are part of packets. When this is fixed this change may be reverted.
|
#
a5f77087 |
|
04-Nov-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Fix a logic error introduced with mandatory mbuf cluster refcounting and freeing of mbufs+clusters back to the packet zone.
|
#
56a4e45a |
|
02-Nov-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
|
#
fdcc028d |
|
30-Aug-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
|
#
e0068c3a |
|
30-Aug-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Unbreak m_demote() and put back the 'all' flag. Without it we cannot correctly test for m_nextpkt in an mbuf chain.
|
#
fbe81638 |
|
30-Aug-2005 |
Andre Oppermann <andre@FreeBSD.org> |
o Remove the 'all' flag from m_demote(). Users can simply call it with m_demote(m->m_next) if they wish to start at the second mbuf in chain. o Test m_type with == instead of &. o Check m_nextpkt against NULL instead of implicit 0. Based on feedback from: sam
|
#
4da84431 |
|
29-Aug-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Add m_copymdata(struct mbuf *m, struct mbuf *n, int off, int len, int prep, int how). Copies the data portion of mbuf (chain) n starting from offset off for length len to mbuf (chain) m. Depending on prep the copied data will be appended or prepended. The function ensures that the mbuf (chain) m will be fully writeable by making real (not refcnt) copies of mbuf clusters. For the prepending the function returns a pointer to the new start of mbuf chain m and leaves as much leading space as possible in the new first mbuf. Reviewed by: glebius
|
#
a048affb |
|
29-Aug-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
|
#
ed111688 |
|
29-Aug-2005 |
Andre Oppermann <andre@FreeBSD.org> |
Add m_demote(struct mbuf *m, int all) to clean up mbuf (chain) from any tags and packet headers. If "all" is set then the first mbuf in the chain will be cleaned too. This function is used before an mbuf, that arrived as packet with m->flags & M_PKTHDR, is appended to an mbuf chain using m->m_next (not m->m_nextpkt). Reviewed by: glebius
|
#
ab8ab90c |
|
29-Jul-2005 |
Sam Leffler <sam@FreeBSD.org> |
add m_align, a function to align any type of mbuf (i.e. it is a superset of M_ALIGN and MH_ALIGN) Reviewed by: several
|
#
75ae2570 |
|
04-May-2005 |
Maksim Yevmenkin <emax@FreeBSD.org> |
Change m_uiotombuf so it will accept offset at which data should be copied to the mbuf. Offset cannot exceed MHLEN bytes. This is currently used to fix Ethernet header alignment problem on alpha and sparc64. Also change all users of m_uiotombuf to pass proper offset. Reviewed by: jmg, sam Tested by: Sten Spans "sten AT blinkenlights DOT nl" MFC after: 1 week
|
#
7ac139a9 |
|
17-Mar-2005 |
John-Mark Gurney <jmg@FreeBSD.org> |
add m_copyup function.. This can be used to help make our ip stack less alignment restrictive, and help performance on some ethernet cards which currently copy the entire packet a couple bytes to get the packet aligned properly... Wordsmithing by: dwhite Obtained from: NetBSD (code only) I'll clean it up later: rwatson
|
#
a4e71429 |
|
08-Mar-2005 |
Sam Leffler <sam@FreeBSD.org> |
allow the destination of m_move_pkthdr to have external storage (e.g. a cluster) Glanced at by: rwatson, silby
|
#
2b2c7a6b |
|
06-Mar-2005 |
Alan Cox <alc@FreeBSD.org> |
The m_ext reference counts are potentially shared and modified asynchronously by different threads. Thus, declare as volatile the reference count that is accessed through m_ext's pointer, ref_cnt. Revert the previous change, revision 1.144, that casts as volatile a single dereference of ref_cnt. Reviewed by: bmilekic, dwhite Problem reported by: kris MFC after: 3 days
|
#
a1d0c3f2 |
|
02-Mar-2005 |
Doug White <dwhite@FreeBSD.org> |
Insert volatile cast to discourage gcc from optimizing the read outside of the while loop. Suggested by: alc MFC after: 1 day
|
#
59d8b310 |
|
23-Feb-2005 |
Sam Leffler <sam@FreeBSD.org> |
change m_adj to reclaim unused mbufs instead of zero'ing m_len when trim'ing space off the back of a chain; this is indirect solution to a potential null ptr deref Noticed by: Coverity Prevent analysis tool (null ptr deref) Reviewed by: dg, rwatson
|
#
9d8993bb |
|
23-Feb-2005 |
Sam Leffler <sam@FreeBSD.org> |
remove dead code Noticed by: Coverity Prevent analysis tool Reviewed by: silby
|
#
3d2a3ff2 |
|
10-Feb-2005 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Optimize the way reference counting is performed with Mbufs. We do not need to perform an extra memory fetch in the Packet (Mbuf+Cluster) constructor to initialize the reference counter anymore. The reference counts are located in a separate memory region (in the slab header, because this zone is UMA_ZONE_REFCNT), so the memory fetch resulted very often in a cache miss. Additionally, and perhaps more significantly, optimize the free mbuf+cluster (packet) case, which is very common, to no longer require an atomic operation on free (to verify the reference counter) if the reference on the cluster has never been increased (also very common). Reduces an atomic on mbuf free on average. Original patch submitted by: Gerrit Nagelhout <gnagelhout@sandvine.com>
|
#
c711aea6 |
|
09-Feb-2005 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Make a bunch of malloc types static. Found by: src/tools/tools/kernxref
|
#
9454b2d8 |
|
06-Jan-2005 |
Warner Losh <imp@FreeBSD.org> |
/* -> /*- for copyright notices, minor format tweaks as necessary
|
#
a37c415e |
|
15-Dec-2004 |
Sam Leffler <sam@FreeBSD.org> |
fix m_append for case where additional mbufs are required
|
#
4873d1754 |
|
07-Dec-2004 |
Sam Leffler <sam@FreeBSD.org> |
add m_append utility function to be used in forthcoming changes
|
#
7b125090 |
|
28-Sep-2004 |
John-Mark Gurney <jmg@FreeBSD.org> |
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
|
#
01e9ccbd |
|
21-Jul-2004 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Back out just a portion of Alfred's last commit. Remove the MBUF_CHECK (WITNESS) for code paths that always call uma_zalloc_arg() shortly after where the check was, because uma_zalloc_arg() already does a similar check. No objections from Alfred. Thanks Alfred.
|
#
063d8114 |
|
21-Jul-2004 |
Alfred Perlstein <alfred@FreeBSD.org> |
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
|
#
96e12413 |
|
11-Jun-2004 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Gah! Plug a mbuf leak I introduced in the last commit. I don the pointy-hat. Problem reported by: Peter Holm <pho@>
|
#
b5b2ea9a |
|
09-Jun-2004 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Plug a race where upon free this scenario could occur: (time grows downward) thread 1 thread 2 ------------|------------ dec ref_cnt | | dec ref_cnt <-- ref_cnt now zero cmpset | free all | return | | alloc again,| reuse prev | ref_cnt | | cmpset, read | already freed | ref_cnt ------------|------------ This should fix that by performing only a single atomic test-and-set that will serve to decrement the ref_cnt, only if it hasn't changed since the earlier read, otherwise it'll loop and re-read. This forces ordering of decrements so that truly the thread which did the LAST decrement is the one that frees. This is how atomic-instruction-based refcnting should probably be handled. Submitted by: Julian Elischer
|
#
931f76ab |
|
09-Jun-2004 |
Maxime Henrion <mux@FreeBSD.org> |
Fix a panic happening when m_getm() is called with len < MCLBYTES. Reported by: ale Tested by: ale Reviewed by: bosko
|
#
099a0e58 |
|
31-May-2004 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
|
#
24665342 |
|
18-Apr-2004 |
Luigi Rizzo <luigi@FreeBSD.org> |
constify the last argument of m_copyback.
|
#
7f8a436f |
|
05-Apr-2004 |
Warner Losh <imp@FreeBSD.org> |
Remove advertising clause from University of California Regent's license, per letter dated July 22, 1999. Approved by: core
|
#
b711d74e |
|
05-Feb-2004 |
Mike Silbersack <silby@FreeBSD.org> |
Style fixes: don't indent variable names. Submitted by: bde
|
#
2ccbe4b5 |
|
04-Feb-2004 |
Mike Silbersack <silby@FreeBSD.org> |
Style fixes Submitted by: bde
|
#
beb699c7 |
|
01-Feb-2004 |
Mike Silbersack <silby@FreeBSD.org> |
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
|
#
8dee2f67 |
|
24-Dec-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Fix another 0 / NULL mixup.
|
#
a89ec05e |
|
22-Dec-2003 |
Peter Wemm <peter@FreeBSD.org> |
Catch a few places where NULL (pointer) was used where 0 (integer) was expected.
|
#
54065297 |
|
16-Dec-2003 |
Bruce M Simpson <bms@FreeBSD.org> |
style(9) pass and type fixups. Submitted by: bde
|
#
37621fd5 |
|
15-Dec-2003 |
Bruce M Simpson <bms@FreeBSD.org> |
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
|
#
3390d476 |
|
31-Aug-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
|
#
f8bf8e39 |
|
19-Jul-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Three fixes: - Make m_prepend use m_gethdr instead of m_get where appropriate - Make m_copym use m_gethdr instead of m_get where appropriate - Add a call to m_fixhdr in m_defrag; m_defrag can't deal with corrupted pkthdr.len counts. MFC after: 3 days
|
#
51710a45 |
|
16-Jun-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
|
#
677b542e |
|
10-Jun-2003 |
David E. O'Brien <obrien@FreeBSD.org> |
Use __FBSDID().
|
#
352d050e |
|
14-Apr-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Add another MBUF_STRESS_TEST feature, m_defragrandomfailures. When enabled, this causes m_defrag to randomly return NULL (following its normal failure case so that extra memory leaks are not introduced.) Code similar to this was used to find / fix a few bugs last week.
|
#
225bff6f |
|
14-Apr-2003 |
Robert Watson <rwatson@FreeBSD.org> |
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
|
#
aa65d9f5 |
|
14-Apr-2003 |
Robert Watson <rwatson@FreeBSD.org> |
Use MBTOM() to convert mbuf allocator flags to malloc() flags, rather than using the same compare/substitute in many places. Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
|
#
fe584538 |
|
08-Apr-2003 |
Dag-Erling Smørgrav <des@FreeBSD.org> |
Introduce an M_ASSERTPKTHDR() macro which performs the very common task of asserting that an mbuf has a packet header. Use it instead of hand- rolled versions wherever applicable. Submitted by: Hiten Pandya <hiten@unixdaemons.com>
|
#
55e9f80d |
|
28-Mar-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
|
#
df8c7fc9 |
|
27-Mar-2003 |
Mike Silbersack <silby@FreeBSD.org> |
Allow m_dup_pkthdr to accept mbufs with attached clusters as targets. Submitted by: bmilekic
|
#
a80cc4e1 |
|
13-Mar-2003 |
Ian Dowse <iedowse@FreeBSD.org> |
In m_dup_pkthdr(), convert the supplied `how' argument into malloc flags when passing it into m_tag_copy_chain(), as m_tag* functions use malloc, not mbuf flags.
|
#
a163d034 |
|
18-Feb-2003 |
Warner Losh <imp@FreeBSD.org> |
Back out M_* changes, per decision of the TRB. Approved by: trb
|
#
44956c98 |
|
21-Jan-2003 |
Alfred Perlstein <alfred@FreeBSD.org> |
Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0. Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.
|
#
9967cafc |
|
30-Dec-2002 |
Sam Leffler <sam@FreeBSD.org> |
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
|
#
5d846453 |
|
15-Oct-2002 |
Sam Leffler <sam@FreeBSD.org> |
Replace aux mbufs with packet tags: o instead of a list of mbufs use a list of m_tag structures a la openbsd o for netgraph et. al. extend the stock openbsd m_tag to include a 32-bit ABI/module number cookie o for openbsd compatibility define a well-known cookie MTAG_ABI_COMPAT and use this in defining openbsd-compatible m_tag_find and m_tag_get routines o rewrite KAME use of aux mbufs in terms of packet tags o eliminate the most heavily used aux mbufs by adding an additional struct inpcb parameter to ip_output and ip6_output to allow the IPsec code to locate the security policy to apply to outbound packets o bump __FreeBSD_version so code can be conditionalized o fixup ipfilter's call to ip_output based on __FreeBSD_version Reviewed by: julian, luigi (silent), -arch, -net, darren Approved by: julian, silence from everyone else Obtained from: openbsd (mostly) MFC after: 1 month
|
#
4a3276d4 |
|
19-Sep-2002 |
Julian Elischer <julian@FreeBSD.org> |
While well intentionned the check to see it there is a packet header and return that length, was misguided. The check itself didn't take into account the fact that the mbuf pointer pased in may be null, and the function is defined specifically for cases where the caller knows what it wants. Rather than fix the check I'm removing it as phk suggested. Submitted by: phk@freebsd.org
|
#
4a49235b |
|
18-Sep-2002 |
Julian Elischer <julian@FreeBSD.org> |
fix style.. Return in the kernel always has () around the arguments.
|
#
1494277d |
|
18-Sep-2002 |
Julian Elischer <julian@FreeBSD.org> |
Compiler was correct: m WAS being used uninitialized..
|
#
e6249771 |
|
18-Sep-2002 |
Darren Reed <darrenr@FreeBSD.org> |
If M_PKTHDR is set then we don't need to do a loop to find the total length.
|
#
bd395ae8 |
|
18-Sep-2002 |
Bosko Milekic <bmilekic@FreeBSD.org> |
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
|
#
4e4425d4 |
|
18-Sep-2002 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Make m_length() and m_fixhdr() return unsigned. Suggested by: arr
|
#
ac6e585d |
|
18-Sep-2002 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Introduce the m_length() function which will return the accumulated length of an mbuf-chain and optionally a pointer to the last mbuf.
|
#
3f2e06c5 |
|
18-Sep-2002 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
|
#
f9d0d524 |
|
01-Aug-2002 |
Robert Watson <rwatson@FreeBSD.org> |
Include file cleanup; mac.h and malloc.h at one point had ordering relationship requirements, and no longer do. Reminded by: bde
|
#
e32a5b94 |
|
30-Jul-2002 |
Robert Watson <rwatson@FreeBSD.org> |
Introduce support for Mandatory Access Control and extensible kernel access control. Invoke additional MAC entry points when an mbuf packet header is copied to another mbuf: release the old label if any, reinitialize the new header, and ask the MAC framework to copy the header label data. Note that this requires a potential allocation operation, but m_copy_pkthdr() is not permitted to fail, so we must block. Since we now use interrupt threads, this is possible, but not desirable. Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
|
#
e37b1fcd |
|
30-Jul-2002 |
Robert Watson <rwatson@FreeBSD.org> |
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
|
#
4151d2e6 |
|
24-Jul-2002 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Move m_freem() from uipc_mbuf.c to subr_mbuf.c so it can take advantage of the inlines, like its cousin, m_free(). Also, make a small (first step?) optimisation of m_free() to use the MBP_PERSIST{,ENT} interface to hold the lock across frees when possible. The thing is that right now, we can only do this easily for at most across one mbuf + one cluster free, as the comment mentions (it also explains why). Anyway, some basic tests revealed a 5-10% overall improvement. Some of the results can be found here: http://people.freebsd.org/~bmilekic/code/measure.txt
|
#
185c2244 |
|
15-Jul-2002 |
Bosko Milekic <bmilekic@FreeBSD.org> |
o Introduce new m_getcl() interface routine that allocates an mbuf and a cluster in one shot. o Introduce MBP_PERSIST and MBP_PERSISTENT control bits to mb_alloc(); MBP_PERSIST means "if you can allocate, then keep the cache lock held on exit," and MBP_PERSISTENT means "a cache lock is alredy held on entry, so allocate from the specified (already locked) cache." They may be used in combination. o m_getcl() uses the MBP_PERSIST/MBP_PERSISTENT interface so that it doesn't drop the cache lock in between the mbuf and cluster allocations. o m_getm(), which takes a size and allocates an mbuf + cluster "best fit" chain, has been moved from uipc_mbuf.c to subr_mbuf.c and shown how to use MBP_PERSIST/MBP_PERSISTENT to attempt to do a grouped allocation without dropping the cache lock in between. Why this is good: much less bus-locked lock acquires/drops when they're not needed. Also, prototype for m_getcl(): struct mbuf * m_getcl(int how, short type, int flags); "how" and "type" are self-explanatory. "flags" may be M_PKTHDR, in which case m_getcl() will make the mbuf a pkthdr-mbuf. While I'm in subr_mbuf.c: o Every exported routine now has a nice comment with a description of the expected arguments. Eventually, mbuf(9) needs to be re-vamped but there's still more code to write/finalize before I get to that. o internal macros have been changed a bit. o consistently use 'short' for "type." This somehow slipped through before (that 'type' was sometimes declared as int). Alfred has been pushing for the MBP_PERSIST{,ENT} thing for almost a year now. Luigi asked for m_getcl(), and will probably MFC that part of this commit. TODO [Related]: teach mb_free() about MBP_PERSIST{, ENT}.
|
#
48d183fa |
|
31-May-2002 |
Archie Cobbs <archie@FreeBSD.org> |
Fix a bug in m_split(): the "m->m_ext.ext_size" field of an mbuf was being set to zero. This field indicates the total space in the external buffer and therefore should not be modified after the external buffer is added. Add a comment warning that the mbufs returned by m_split() might be read-only. Fix M_TRAILINGSPACE() to return zero if !M_WRITABLE(m). Reviewed by: freebsd-net Obtained from: Vernier Networks, Inc. MFC after: 1 week
|
#
40376987 |
|
11-Apr-2002 |
Jeffrey Hsu <hsu@FreeBSD.org> |
Fix corner case where m_len was not being initialized. Submitted by: Maksim Yevmenkin <myevmenk@digisle.net> MFC after: 1 week
|
#
ecde8f7c |
|
04-Feb-2002 |
Matthew Dillon <dillon@FreeBSD.org> |
Get rid of the twisted MFREE() macro entirely. Reviewed by: dg, bmilekic MFC after: 3 days
|
#
a48740b6 |
|
09-Dec-2001 |
David E. O'Brien <obrien@FreeBSD.org> |
Update to C99, s/__FUNCTION__/__func__/.
|
#
a8cfc0ee |
|
18-Aug-2001 |
Julian Elischer <julian@FreeBSD.org> |
Forgot to remove this un-needed test. (M_WAITOK won't fail) I vaguely remember someone once proving it COULD return NULL.. was that changed? Reminded by: BDE MFC after: 2 weeks
|
#
08442f8a |
|
22-Jun-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Introduce numerous SMP friendly changes to the mbuf allocator. Namely, introduce a modified allocation mechanism for mbufs and mbuf clusters; one which can scale under SMP and which offers the possibility of resource reclamation to be implemented in the future. Notable advantages: o Reduce contention for SMP by offering per-CPU pools and locks. o Better use of data cache due to per-CPU pools. o Much less code cache pollution due to excessively large allocation macros. o Framework for `grouping' objects from same page together so as to be able to possibly free wired-down pages back to the system if they are no longer needed by the network stacks. Additional things changed with this addition: - Moved some mbuf specific declarations and initializations from sys/conf/param.c into mbuf-specific code where they belong. - m_getclr() has been renamed to m_get_clrd() because the old name is really confusing. m_getclr() HAS been preserved though and is defined to the new name. No tree sweep has been done "to change the interface," as the old name will continue to be supported and is not depracated. The change was merely done because m_getclr() sounds too much like "m_get a cluster." - TEMPORARILY disabled mbtypes statistics displaying in netstat(1) and systat(1) (see TODO below). - Fixed systat(1) to display number of "free mbufs" based on new per-CPU stat structures. - Fixed netstat(1) to display new per-CPU stats based on sysctl-exported per-CPU stat structures. All infos are fetched via sysctl. TODO (in order of priority): - Re-enable mbtypes statistics in both netstat(1) and systat(1) after introducing an SMP friendly way to collect the mbtypes stats under the already introduced per-CPU locks (i.e. hopefully don't use atomic() - it seems too costly for a mere stat update, especially when other locks are already present). - Optionally have systat(1) display not only "total free mbufs" but also "total free mbufs per CPU pool." - Fix minor length-fetching issues in netstat(1) related to recently re-enabled option to read mbuf stats from a core file. - Move reference counters at least for mbuf clusters into an unused portion of the cluster itself, to save space and need to allocate a counter. - Look into introducing resource freeing possibly from a kproc. Reviewed by (in parts): jlemon, jake, silby, terry Tested by: jlemon (Intel & Alpha), mjacob (Intel & Alpha) Preliminary performance measurements: jlemon (and me, obviously) URL: http://people.freebsd.org/~bmilekic/mb_alloc/
|
#
f5eece3f |
|
20-Jun-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
|
#
db957588 |
|
12-Jun-2001 |
Peter Wemm <peter@FreeBSD.org> |
Patch up a blunder I made a few days ago. nmbcnt was being initialized too late. Noted by: bmilekic Pointy-hat to: peter
|
#
33841545 |
|
10-Jun-2001 |
Hajimu UMEMOTO <ume@FreeBSD.org> |
Sync with recent KAME. This work was based on kame-20010528-freebsd43-snap.tgz and some critical problem after the snap was out were fixed. There are many many changes since last KAME merge. TODO: - The definitions of SADB_* in sys/net/pfkeyv2.h are still different from RFC2407/IANA assignment because of binary compatibility issue. It should be fixed under 5-CURRENT. - ip6po_m member of struct ip6_pktopts is no longer used. But, it is still there because of binary compatibility issue. It should be removed under 5-CURRENT. Reviewed by: itojun Obtained from: KAME MFC after: 3 weeks
|
#
09786698 |
|
07-Jun-2001 |
Peter Wemm <peter@FreeBSD.org> |
"Fix" the previous initial attempt at fixing TUNABLE_INT(). This time around, use a common function for looking up and extracting the tunables from the kernel environment. This saves duplicating the same function over and over again. This way typically has an overhead of 8 bytes + the path string, versus about 26 bytes + the path string.
|
#
4422746f |
|
06-Jun-2001 |
Peter Wemm <peter@FreeBSD.org> |
Back out part of my previous commit. This was a last minute change and I botched testing. This is a perfect example of how NOT to do this sort of thing. :-(
|
#
81930014 |
|
06-Jun-2001 |
Peter Wemm <peter@FreeBSD.org> |
Make the TUNABLE_*() macros look and behave more consistantly like the SYSCTL_*() macros. TUNABLE_INT_DECL() was an odd name because it didn't actually declare the int, which is what the name suggests it would do.
|
#
240ef842 |
|
01-Jun-2001 |
David E. O'Brien <obrien@FreeBSD.org> |
Back out jesper's 2001/05/31 14:58:11 PDT commit. It does not compile.
|
#
e916d96e |
|
31-May-2001 |
Jesper Skriver <jesper@FreeBSD.org> |
Move the definition of NMBCLUSTERS from src/sys/kern/uipc_mbuf.c to <sys/param.h>, so it's available to src/sys/netinet/ip_input.c, and remove the now unneeded includes of "opt_param.h". MFC after: 1 week
|
#
629db604 |
|
23-May-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Increment mbstat.m_mpfail, not mbstat.m_mcfail, when m_pullup() fails. This slipped in accidently a few commits back.
|
#
fb919e4d |
|
01-May-2001 |
Mark Murray <markm@FreeBSD.org> |
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files. Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files. Sort sys/*.h includes where possible in affected files. OK'ed by: bde (with reservations)
|
#
d04d50d1 |
|
18-Apr-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Fix inconsistency in setup of kernel_map: we need to make sure that we also reserve _adequate_ space for the mb_map submap; i.e. we need space for nmbclusters, nmbufs, _and_ nmbcnt. Furthermore, we need to rounddown, and not roundup, so that we are consistent. Pointed out by: bde
|
#
4b8ae40a |
|
02-Apr-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
- Change the msleep()s to condition variables. The mbuf and mcluster free lists now each "own" a condition variable, m_starved. - Clean up minor indentention issues in sys/mbuf.h caused by previous commit.
|
#
5ea487f3 |
|
02-Apr-2001 |
Alfred Perlstein <alfred@FreeBSD.org> |
Use only one mutex for the entire mbuf subsystem. Don't use atomic operations for the stats updating, instead protect the counts with the mbuf mutex. Most twiddling of the stats was done right before or after releasing a mutex. By doing this we reduce the number of locked ops needed as well as allow a sysctl to gain a consitant view of the entire stats structure. In the future... This will allow us to chain common mbuf operations that would normally need to aquire/release 2 or 3 of the locks to build an mbuf with a cluster or external data attached into a single op requiring only one lock. Simplify the per-cpu locks that are planned. There's also some if (1) code that should check if the "how" operation specifies blocking/non-blocking behavior, we _could_ make it so that we hold onto the mutex through calls into kmem_alloc when non-blocking requests are made, but for safety reasons we currently drop and reaquire the mutex around the calls. Also, note that calling kmem_alloc is rare and only happens during a shortage so drop/re-getting the mutex will not be a common occurance. Remove some #define's that seemed to obfuscate the code to me. Remove an extranious comment. Remove an XXX, including mutex.h isn't a crime. Reviewed by: bmilekic
|
#
2ba1a895 |
|
24-Mar-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Move the atomic() mbstat.m_drops incrementing to the MGET(HDR) and MCLGET macros in order to avoid incrementing the drop count twice. Otherwise, in some cases, we may increment m_drops once in m_mballoc() for example, and increment it again in m_mballoc_wait() if the wait fails.
|
#
9612101e |
|
17-Mar-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Fix a couple of things in the internal mbuf allocation interface: - Make sure that m_mballoc() really doesn't allow over nmbufs mbufs to be allocated from mb_map. In the case where nmbufs-reserved space is not an exact multiple of PAGE_SIZE (which it should be, but anyway...), we hold nmbufs as an absolute maximum which need not ever be reached. - Clean up m_clalloc(); make it more consistent in the sense that the first argument `ncl' really means "the number of clusters ensured to be allocated" and not "the number of pages worth of clusters to be allocated," as was previously the case. This also makes it consistent with m_mballoc() as well as the comment that preceeds it. Reviewed by: jlemon
|
#
03137ec8 |
|
21-Feb-2001 |
Boris Popov <bp@FreeBSD.org> |
Fix parameter order in the calls to MGET().
|
#
5fe86675 |
|
20-Feb-2001 |
Luigi Rizzo <luigi@FreeBSD.org> |
Preserve alignment of first mbuf in m_copypacket. This is useful when doing copies of packet where some leading space has been preallocated to insert protocol headers. Note that there are in fact almost no users of m_copypacket. MFC candidate.
|
#
fffd12bd |
|
13-Feb-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Implement m_getm() which will perform an "all or nothing" mbuf + cluster allocation, as required. If m_getm() receives NULL as a first argument, then it allocates `len' (second argument) bytes worth of mbufs + clusters and returns the chain only if it was able to allocate everything. If the first argument is non-NULL, then it should be an existing mbuf chain (e.g. pre-allocated mbuf sitting on a ring, on some list, etc.) and so it will allocate `len' bytes worth of clusters and mbufs, as needed, and append them to the tail of the passed in chain, only if it was able to allocate everything requested. If allocation fails, only what was allocated by the routine will be freed, and NULL will be returned. Also, get rid of existing m_getm() in netncp code and replace calls to it to calls to this new generic code. Heavily Reviewed by: bp
|
#
122a814a |
|
10-Feb-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
|
#
9ed346ba |
|
08-Feb-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Change and clean the mutex lock interface. mtx_enter(lock, type) becomes: mtx_lock(lock) for sleep locks (MTX_DEF-initialized locks) mtx_lock_spin(lock) for spin locks (MTX_SPIN-initialized) similarily, for releasing a lock, we now have: mtx_unlock(lock) for MTX_DEF and mtx_unlock_spin(lock) for MTX_SPIN. We change the caller interface for the two different types of locks because the semantics are entirely different for each case, and this makes it explicitly clear and, at the same time, it rids us of the extra `type' argument. The enter->lock and exit->unlock change has been made with the idea that we're "locking data" and not "entering locked code" in mind. Further, remove all additional "flags" previously passed to the lock acquire/release routines with the exception of two: MTX_QUIET and MTX_NOSWITCH The functionality of these flags is preserved and they can be passed to the lock/unlock routines by calling the corresponding wrappers: mtx_{lock, unlock}_flags(lock, flag(s)) and mtx_{lock, unlock}_spin_flags(lock, flag(s)) for MTX_DEF and MTX_SPIN locks, respectively. Re-inline some lock acq/rel code; in the sleep lock case, we only inline the _obtain_lock()s in order to ensure that the inlined code fits into a cache line. In the spin lock case, we inline recursion and actually only perform a function call if we need to spin. This change has been made with the idea that we generally tend to avoid spin locks and that also the spin locks that we do have and are heavily used (i.e. sched_lock) do recurse, and therefore in an effort to reduce function call overhead for some architectures (such as alpha), we inline recursion for this case. Create a new malloc type for the witness code and retire from using the M_DEV type. The new type is called M_WITNESS and is only declared if WITNESS is enabled. Begin cleaning up some machdep/mutex.h code - specifically updated the "optimized" inlined code in alpha/mutex.h and wrote MTX_LOCK_SPIN and MTX_UNLOCK_SPIN asm macros for the i386/mutex.h as we presently need those. Finally, caught up to the interface changes in all sys code. Contributors: jake, jhb, jasone (in no particular order)
|
#
5dbc7fe2 |
|
07-Feb-2001 |
John Baldwin <jhb@FreeBSD.org> |
Don't bother with acquiring/releasing Giant around kmem_malloc() and kmem_free() for now. Kmem_malloc() and kmem_free() now have appropriate assertions in place, and these checks aren't feasible until more of the networking code is locked down. Also, the extra assertions here should already be caught by the WITNESS code as lock order violations should mutex operations on Giant be reintroduced here later.
|
#
56acb799 |
|
20-Jan-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
When short of mbufs or mbuf clusters, we sleep on appropriate "counters." The counters are incremented when a thread goes to sleep and decremented either when a thread is woken up by another thread or when the sleep times out. There existed a race where the sleep count could be decremented twice resulting in an eventual underflow. Move the decrementing of the "counters" to the thread initiating the sleep and thus remedy the problem.
|
#
35c05ac6 |
|
15-Jan-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Add some KASSERTs valid if WITNESS is defined to verify that the mbuf allocation routines are being called safely. Since we drop our relevant mbuf mutex and acquire Giant before we call kmem_malloc(), we have to make sure that this does not pave the way for a fatal lock order reversal. Check that either Giant is already held (in which case it's safe to grab it again and recurse on it) or, if Giant is not held, that no other locks are held before we try to acquire Giant. Similarily, add a KASSERT valid in the WITNESS case in m_reclaim() to nail callers who end up in m_reclaim() and hold a lock. Pointed out by: jhb
|
#
d113d385 |
|
09-Jan-2001 |
Bosko Milekic <bmilekic@FreeBSD.org> |
In m_mballoc_wait(), drop the mmbfree mutex lock prior to calling m_reclaim() and re-acquire it when m_reclaim() returns. This means that we now call the drain routines without holding the mutex lock and recursing into it. This was done for mainly two reasons: (i) Avoid the long recursion; long recursions are typically bad and this is the case here because we block all other code from freeing mbufs if they need to. Doing that is kind of counter-productive, since we're really hoping that someone will free. (ii) More importantly, avoid a potential lock order reversal. Right now, not all the locks have been added to our networking code; but without this change, we're introducing the possibility for deadlock. Consider for example ip_drain(). We will likely eventually introduce a lock for ipq there, and so ip_freef() will be called with ipq lock held. But, ip_freef() calls m_freem() which in turn acquires the mmbfree lock. Since we were previously calling ip_drain() with mmbfree held, our lock order would be: mmbfree->ipq->mmbfree. Some other code may very well lock ipq first and then call ip_freef(). This would result in the regular lock order, ipq->mmbfree. Clearly, we have deadlock if one thread acquires the ipq lock and sits waiting for mmbfree while another thread calling m_reclaim() acquires mmbfree and sits waiting for the ipq lock. Also, make sure to add a comment above m_reclaim()'s definition briefly explaining this. Also document this above the call to m_reclaim() in m_mballoc_wait(). Suggested and reviewed by: alfred
|
#
2a0c503e |
|
21-Dec-2000 |
Bosko Milekic <bmilekic@FreeBSD.org> |
* Rename M_WAIT mbuf subsystem flag to M_TRYWAIT. This is because calls with M_WAIT (now M_TRYWAIT) may not wait forever when nothing is available for allocation, and may end up returning NULL. Hopefully we now communicate more of the right thing to developers and make it very clear that it's necessary to check whether calls with M_(TRY)WAIT also resulted in a failed allocation. M_TRYWAIT basically means "try harder, block if necessary, but don't necessarily wait forever." The time spent blocking is tunable with the kern.ipc.mbuf_wait sysctl. M_WAIT is now deprecated but still defined for the next little while. * Fix a typo in a comment in mbuf.h * Fix some code that was actually passing the mbuf subsystem's M_WAIT to malloc(). Made it pass M_WAITOK instead. If we were ever to redefine the value of the M_WAIT flag, this could have became a big problem.
|
#
35e0e5b3 |
|
20-Oct-2000 |
John Baldwin <jhb@FreeBSD.org> |
Catch up to moving headers: - machine/ipl.h -> sys/ipl.h - machine/mutex.h -> sys/mutex.h
|
#
181d2a15 |
|
15-Oct-2000 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Add nmbcnt sysctl and make it tunable at boottime; nmbcnt is the number of ext_buf counters that are possibly allocatable. Do this because: (i) It will make it easier to influence EXT_COUNTERS for if_sk, if_ti (or similar) users where the driver allocates its own ext_bufs and where it is important for the mbuf system to take it into account when reserving necessary space for counters. (ii) Facilitate some percentile calculation for netstat(1)
|
#
7d032714 |
|
30-Sep-2000 |
Bosko Milekic <bmilekic@FreeBSD.org> |
Big mbuf subsystem diff #1: incorporate mutexes and fix things up somewhat to accomodate the changes. Here's a list of things that have changed (I may have left out a few); for a relatively complete list, see http://people.freebsd.org/~bmilekic/mtx_journal * Remove old (once useful) mcluster code for MCLBYTES > PAGE_SIZE which nobody uses anymore. It was great while it lasted, but now we're moving onto bigger and better things (Approved by: wollman). * Practically re-wrote the allocation macros in sys/sys/mbuf.h to accomodate new allocations which grab the necessary lock. * Make sure that necessary mbstat variables are manipulated with corresponding atomic() routines. * Changed the "wait" routines, cleaned it up, made one routine that does the job. * Generalized MWAKEUP() macro. Got rid of m_retry and m_retryhdr, as they are now included in the generalized "wait" routines. * Sleep routines now use msleep(). * Free lists have locks. * etc... probably other stuff I'm missing... Things to look out for and work on later: * find a better way to (dynamically) adjust EXT_COUNTERS * move necessity to recurse on a lock from drain routines by providing lock-free lower-level version of MFREE() (and possibly m_free()?). * checkout include of mutex.h in sys/sys/mbuf.h - probably violating general philosophy here. The code has been reviewed quite a bit, but problems may arise... please, don't panic! Send me Emails: bmilekic@freebsd.org Reviewed by: jlemon, cp, alfred, others?
|
#
9579e8c1 |
|
25-Aug-2000 |
Peter Wemm <peter@FreeBSD.org> |
m_mballoc_wait() had a spl/tsleep race. mbufs can be freed in interrupt context, which can cause a wakeup.. which can race with this.
|
#
a5c4836d |
|
19-Aug-2000 |
David Malone <dwmalone@FreeBSD.org> |
Replace the mbuf external reference counting code with something that should be better. The old code counted references to mbuf clusters by using the offset of the cluster from the start of memory allocated for mbufs and clusters as an index into an array of chars, which did the reference counting. If the external storage was not a cluster then reference counting had to be done by the code using that external storage. NetBSD's system of linked lists of mbufs was cosidered, but Alfred felt it would have locking issues when the kernel was made more SMP friendly. The system implimented uses a pool of unions to track external storage. The union contains an int for counting the references and a pointer for forming a free list. The reference counts are incremented and decremented atomically and so should be SMP friendly. This system can track reference counts for any sort of external storage. Access to the reference counting stuff is now through macros defined in mbuf.h, so it should be easier to make changes to the system in the future. The possibility of storing the reference count in one of the referencing mbufs was considered, but was rejected 'cos it would often leave extra mbufs allocated. Storing the reference count in the cluster was also considered, but because the external storage may not be a cluster this isn't an option. The size of the pool of reference counters is available in the stats provided by "netstat -m". PR: 19866 Submitted by: Bosko Milekic <bmilekic@dsuper.net> Reviewed by: alfred (glanced at by others on -net)
|
#
9ad48853 |
|
31-Jul-2000 |
Alfred Perlstein <alfred@FreeBSD.org> |
mbstat should be a read-only sysctl. Submitted by: Bosko Milekic <bmilekic@dsuper.net>
|
#
af0e6bcd |
|
15-Jul-2000 |
Alfred Perlstein <alfred@FreeBSD.org> |
Make mbstat.m_mtypes seperate and viewable via sysctl, also expand the size from short to ulong Submitted by: Ian Dowse <iedowse@maths.tcd.ie> PR: kern/19809
|
#
686cdd19 |
|
04-Jul-2000 |
Jun-ichiro itojun Hagino <itojun@FreeBSD.org> |
sync with kame tree as of july00. tons of bug fixes/improvements. API changes: - additional IPv6 ioctls - IPsec PF_KEY API was changed, it is mandatory to upgrade setkey(8). (also syntax change)
|
#
736e4b67 |
|
27-Dec-1999 |
Mike Smith <msmith@FreeBSD.org> |
Actively limit the allocation of mbufs to NMBUFS/nmbufs and mbuf clusters to NMBCLUSTERS/nmbclusters/kern.ipc.nmbclusters. Add a read-only sysctl kern.ipc.nmbufs matching kern.ipc.nmbclusters. Submitted by: Bosko Milekic <bmilekic@dsuper.net>
|
#
6357e7b5 |
|
20-Dec-1999 |
Eivind Eklund <eivind@FreeBSD.org> |
Make m_print const correct (avoids a warning)
|
#
2cfa173c |
|
18-Dec-1999 |
Brian Feldman <green@FreeBSD.org> |
Woops, I'm so sorry I forgot this! From the last mbuf.h change: m_mballoc_wakeup() (inline) -> MMBWAKEUP() (macro) m_clalloc_wakeup() (inline) -> MCLWAKEUP() (macro) Noticed by: peter
|
#
6c544102 |
|
13-Dec-1999 |
Brian Feldman <green@FreeBSD.org> |
Bug fix: The variables "m_mclalloc_wid" and "m_mballoc_wid" were not in the proper place. They should have been in uipc_mbuf.c and have been global, not in mbuf.h and local per each file that uses mbuf.h. Sorta bug fix: In mbuf.h, the definitions of various things for KERNEL and not KERNEL cases were very screwy. This fixes all of that which I could find.
|
#
f48b807f |
|
11-Dec-1999 |
Brian Feldman <green@FreeBSD.org> |
This is Bosko Milekic's mbuf allocation waiting code. Basically, this means that running out of mbuf space isn't a panic anymore, and code which runs out of network memory will sleep to wait for it. Submitted by: Bosko Milekic <bmilekic@dsuper.net> Reviewed by: green, wollman
|
#
1c38f2ea |
|
01-Dec-1999 |
Archie Cobbs <archie@FreeBSD.org> |
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
|
#
eed9fd42 |
|
17-Nov-1999 |
Peter Wemm <peter@FreeBSD.org> |
Fix a warning.
|
#
ce4a64f7 |
|
01-Nov-1999 |
Poul-Henning Kamp <phk@FreeBSD.org> |
New function: m_print(struct mbuf *); hexdumps a mbuf.
|
#
e0a653dd |
|
13-Oct-1999 |
Alfred Perlstein <alfred@FreeBSD.org> |
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
|
#
c3aac50f |
|
27-Aug-1999 |
Peter Wemm <peter@FreeBSD.org> |
$Id$ -> $FreeBSD$
|
#
134c934c |
|
05-Jul-1999 |
Mike Smith <msmith@FreeBSD.org> |
Move the initialisation/tuning of nmbclusters from param.c/machdep.c into uipc_mbuf.c. This reduces three sets of identical tunable code to one set, and puts the initialisation with the mbuf code proper. Make NMBUFs tunable as well. Move the nmbclusters sysctl here as well. Move the initialisation of maxsockets from param.c to uipc_socket2.c, next to its corresponding sysctl. Use the new tunable macros for the kern.vm.kmem.size tunable (this should have been in a separate commit, whoops).
|
#
9c8b8baa |
|
01-Jul-1999 |
Peter Wemm <peter@FreeBSD.org> |
Slight reorganization of kernel thread/process creation. Instead of using SYSINIT_KT() etc (which is a static, compile-time procedure), use a NetBSD-style kthread_create() interface. kproc_start is still available as a SYSINIT() hook. This allowed simplification of chunks of the sysinit code in the process. This kthread_create() is our old kproc_start internals, with the SYSINIT_KT fork hooks grafted in and tweaked to work the same as the NetBSD one. One thing I'd like to do shortly is get rid of nfsiod as a user initiated process. It makes sense for the nfs client code to create them on the fly as needed up to a user settable limit. This means that nfsiod doesn't need to be in /sbin and is always "available". This is a fair bit easier to do outside of the SYSINIT_KT() framework.
|
#
eca2ddda |
|
12-Apr-1999 |
Dag-Erling Smørgrav <des@FreeBSD.org> |
Typo in comment.
|
#
ce02431f |
|
16-Feb-1999 |
Doug Rabson <dfr@FreeBSD.org> |
* Change sysctl from using linker_set to construct its tree using SLISTs. This makes it possible to change the sysctl tree at runtime. * Change KLD to find and register any sysctl nodes contained in the loaded file and to unregister them when the file is unloaded. Reviewed by: Archie Cobbs <archie@whistle.com>, Peter Wemm <peter@netplex.com.au> (well they looked at it anyway)
|
#
01ddfa33 |
|
26-Jul-1998 |
David Greenman <dg@FreeBSD.org> |
Only call m_reclaim() if M_WAIT since calling it from an interrupt can cause problems. PR: 7403
|
#
3e5e083c |
|
03-Jul-1998 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Update M_EXT support in m_copypacket(). PR: 7122 Reviewed by: phk Submitted by: Castor Fu <castor@geocast.com> Originally forgotten by: julian
|
#
9523f5c1 |
|
05-Jun-1998 |
David Greenman <dg@FreeBSD.org> |
If we are out of mb_map space and we failed to m_reclaim() anything and the alloc is not M_DONTWAIT, then panic with "Out of mbuf clusters". Callers that specify M_WAIT can't deal with getting a NULL buffer, so this is a more graceful failure than randomly page faulting in the socket code or elsewhere.
|
#
d68fa50c |
|
20-Feb-1998 |
Bruce Evans <bde@FreeBSD.org> |
Don't depend on "implicit int".
|
#
64cfdf46 |
|
27-Dec-1997 |
Bruce Evans <bde@FreeBSD.org> |
Restored used include of <sys/malloc.h>. malloc() is not used here, but kmem_malloc() is used and it takes the same "flags" as malloc(). Use the mbuf allocation "flags" M_WAIT and M_DONTWAIT consistently. There is really only one boolean flag, M_DONTWAIT, but the "flags" were always treated as enum-like values, except in some places here where the values are tacitly converted to boolean flags. Treat them as enum-like values everywhere, except where we tacitly assume that there are only two values in order to convert them to the corresponding two kmem_malloc() "flags".
|
#
55b211e3 |
|
28-Oct-1997 |
Bruce Evans <bde@FreeBSD.org> |
Removed unused #includes.
|
#
a1c995b6 |
|
12-Oct-1997 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Last major round (Unless Bruce thinks of somthing :-) of malloc changes. Distribute all but the most fundamental malloc types. This time I also remembered the trick to making things static: Put "static" in front of them. A couple of finer points by: bde
|
#
1fd0b058 |
|
02-Aug-1997 |
Bruce Evans <bde@FreeBSD.org> |
Removed unused #includes.
|
#
639acc13 |
|
24-Feb-1997 |
Garrett Wollman <wollman@FreeBSD.org> |
Create a new branch of the kernel MIB, kern.ipc, to store all of the configurables and instrumentation related to inter-process communication mechanisms. Some variables, like mbuf statistics, are instrumented here for the first time. For mbuf statistics: also keep track of m_copym() and m_pullup() failures, and provide for the user's inspection the compiled-in values of MSIZE, MHLEN, MCLBYTES, and MINCLSIZE.
|
#
d8392c6c |
|
18-Feb-1997 |
Garrett Wollman <wollman@FreeBSD.org> |
uipc_mbuf.c: do a better job of counting how often we have to wait for memory, or are denied a cluster. uipc_socket2.c: define some generic ``operation-not-supported'' entry points for pr_usrreqs.
|
#
30f700e9 |
|
13-Feb-1997 |
Garrett Wollman <wollman@FreeBSD.org> |
Provide an alternative mbuf cluster allocator which permits use of clusters greater than one page in length by calling contigmalloc1(). This uses a helper process `mclalloc' to do the allocation if the system runs out at interrupt time to avoid calling contigmalloc at high spl. It is not yet clear to me whether this works.
|
#
649c409d |
|
15-Jan-1997 |
David Greenman <dg@FreeBSD.org> |
Fix bug related to map entry allocations where a sleep might be attempted when allocating memory for network buffers at interrupt time. This is due to inadequate checking for the new mcl_map. Fixed by merging mb_map and mcl_map into a single mb_map. Reviewed by: wollman
|
#
1130b656 |
|
14-Jan-1997 |
Jordan K. Hubbard <jkh@FreeBSD.org> |
Make the long-awaited change from $Id$ to $FreeBSD$ This will make a number of things easier in the future, as well as (finally!) avoiding the Id-smashing problem which has plagued developers for so long. Boy, I'm glad we're not using sup anymore. This update would have been insane otherwise.
|
#
0885c1d2 |
|
18-Aug-1996 |
Julian Elischer <julian@FreeBSD.org> |
fix handling of external objects referenced by mbufs somehow this got broken between 4.3 tahoe and here, though I've been using these fixes for over a year here..
|
#
7642f474 |
|
12-May-1996 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Ups, I removed NMB_INIT too. Complained about by: asami
|
#
0482730e |
|
11-May-1996 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Nail down NCL_INIT = 1, and put a comment there telling what it is.
|
#
cb7545a9 |
|
10-May-1996 |
Garrett Wollman <wollman@FreeBSD.org> |
Allocate mbufs from a separate submap so that NMBCLUSTERS works as expected.
|
#
6a06dea0 |
|
08-May-1996 |
Garrett Wollman <wollman@FreeBSD.org> |
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
|
#
b74a76ee |
|
06-May-1996 |
Poul-Henning Kamp <phk@FreeBSD.org> |
An old typo MCLBYTES/CLBYTES became more obvious bogus now. Submitted by: wollman
|
#
e911eafc |
|
02-May-1996 |
Poul-Henning Kamp <phk@FreeBSD.org> |
removed: CLBYTES PD_SHIFT PGSHIFT NBPG PGOFSET CLSIZELOG2 CLSIZE pdei() ptei() kvtopte() ptetov() ispt() ptetoav() &c &c new: NPDEPG Major macro cleanup.
|
#
edbfedac |
|
11-Mar-1996 |
Peter Wemm <peter@FreeBSD.org> |
Import 4.4BSD-Lite2 onto the vendor branch, note that in the kernel, all files are off the vendor branch, so this should not change anything. A "U" marker generally means that the file was not changed in between the 4.4Lite and Lite-2 releases, and does not need a merge. "C" generally means that there was a change. [note new unused (in this form) syscalls.conf, to be 'cvs rm'ed]
|
#
87b6de2b |
|
14-Dec-1995 |
Poul-Henning Kamp <phk@FreeBSD.org> |
A Major staticize sweep. Generates a couple of warnings that I'll deal with later. A number of unused vars removed. A number of unused procs removed or #ifdefed.
|
#
efeaf95a |
|
06-Dec-1995 |
David Greenman <dg@FreeBSD.org> |
Untangled the vm.h include file spaghetti.
|
#
98d93822 |
|
02-Dec-1995 |
Bruce Evans <bde@FreeBSD.org> |
Completed function declarations and/or added prototypes.
|
#
d841aaa7 |
|
02-Dec-1995 |
Bruce Evans <bde@FreeBSD.org> |
Finished (?) cleaning up sysinit stuff.
|
#
a98ca469 |
|
29-Oct-1995 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Second batch of cleanup changes. This time mostly making a lot of things static and some unused variables here and there.
|
#
4590fd3a |
|
09-Sep-1995 |
David Greenman <dg@FreeBSD.org> |
Fixed init functions argument type - caddr_t -> void *. Fixed a couple of compiler warnings.
|
#
2b14f991 |
|
28-Aug-1995 |
Julian Elischer <julian@FreeBSD.org> |
Reviewed by: julian with quick glances by bruce and others Submitted by: terry (terry lambert) This is a composite of 3 patch sets submitted by terry. they are: New low-level init code that supports loadbal modules better some cleanups in the namei code to help terry in 16-bit character support some changes to the mount-root code to make it a little more modular.. NOTE: mounting root off cdrom or NFS MIGHT be broken as I haven't been able to test those cases.. certainly mounting root of disk still works just fine.. mfs should work but is untested. (tomorrows task) The low level init stuff includes a total rewrite of init_main.c to make it possible for new modules to have an init phase by simply adding an entry to a TEXT_SET (or is it DATA_SET) list. thus a new module can be added to the kernel without editing any other files other than the 'files' file.
|
#
28f8db14 |
|
29-Jul-1995 |
Bruce Evans <bde@FreeBSD.org> |
Eliminate sloppy common-style declarations. There should be none left for the LINT configuation.
|
#
5eb7d0cd |
|
15-Mar-1995 |
David Greenman <dg@FreeBSD.org> |
Special cased the handling of mb_map in the M_WAITOK case. kmem_malloc() now returns NULL and sets a global 'mb_map_full' when the map is full. m_clalloc() has further been taught to expect this and do the right thing. This should fix the "mb_map full" panics that several people have reported.
|
#
9b4b42d1 |
|
23-Feb-1995 |
David Greenman <dg@FreeBSD.org> |
Implemented mbstat.m_wait and mbstat.m_drops.
|
#
0bc786d0 |
|
05-Feb-1995 |
Bruce Evans <bde@FreeBSD.org> |
Update kmem_malloc() call to new waitflag(s) interface. This might fix recent problems on thud and freefall.
|
#
35c10d22 |
|
09-Oct-1994 |
David Greenman <dg@FreeBSD.org> |
Got rid of map.h. It's a leftover from the rmap code, and we use rlists. Changed swapmap into swaplist.
|
#
c5789ba3 |
|
04-Oct-1994 |
Poul-Henning Kamp <phk@FreeBSD.org> |
Moved m_copyback into uipc_mbuf.c
|
#
797f2d22 |
|
02-Oct-1994 |
Poul-Henning Kamp <phk@FreeBSD.org> |
All of this is cosmetic. prototypes, #includes, printfs and so on. Makes GCC a lot more silent.
|
#
3c4dd356 |
|
02-Aug-1994 |
David Greenman <dg@FreeBSD.org> |
Added $Id$
|
#
26f9a767 |
|
25-May-1994 |
Rodney W. Grimes <rgrimes@FreeBSD.org> |
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
|
#
df8bae1d |
|
24-May-1994 |
Rodney W. Grimes <rgrimes@FreeBSD.org> |
BSD 4.4 Lite Kernel Sources
|