MFH: r303612

netgraph module for reconstructing checksums

PR: 206108
Submitted by: Dmitry Vagin daemon.hammer@ya.ru

MFC r301549, r301550:
ng_mppc(4): Bring netgraph(3) MPPC compression support.

Support for compression has been available from July 2007 but it
was never imported due to concerns with patents once held by
STAC/HiFn. The issues have clearly been resolved so bring it
in now.

Special thanks to Brett Glass for preserving the code and
pointing documentation for the expiration case.

Taken from: mav (through Brett Glass)

MFC r290665:

Build all of sys/modules with SUBDIR_PARALLEL.

MFC r288304:

Enable parallel subdirectory building with sys/modules/netgraph

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Use rt_numfibs variable instead of compile-time RT_NUMFIBS.

Reviewed by: glebius (previous version)
Approved by: kib(mentor), ae(mentor)

Add IPv6 support to the ng_ipfw(4) [1]. Also add ifdefs to be able
build it with and without INET/INET6 support.

Submitted by: Alexander V. Chernikov <melifaro at yandex-team.ru> [1]
Tested by: Alexander V. Chernikov <melifaro at yandex-team.ru> [1]
Approved by: re (bz)
MFC after: 2 weeks

Remove missing include directory in preparation for adding
-Wmissing-include-dirs to CWARNFLAGS.

Add support for NetFlow version 9 into ng_netflow(4) node.

Submitted by: Alexander V. Chernikov <melifaro ipfw.ru>

New netgraph node ng_patch(4). It performs data modification of packets
passing through. Modifications are restricted to a subset of C language
operations on unsigned integers of 8, 16, 32 or 64 bit size.
These are: set to new value (=), addition (+=), subtraction (-=),
multiplication (*=), division (/=), negation (= -), bitwise AND (&=),
bitwise OR (|=), bitwise eXclusive OR (^=), shift left (<<=),
shift right (>>=). Several operations are all applied to a packet
sequentially in order they were specified by user.

Submitted by: Maxim Ignatenko <gelraen.ua at gmail.com>
Vadim Goncharov <vadimnuclight at tpu.ru>
Discussed with: net@
Approved by: mav (mentor)
MFC after: 1 month

Connect ng_pipe to the default build.

Approved by: julian (mentor)

After r193232 rt_tables in vnet.h are no longer indirectly dependent on
the ROUTETABLES kernel option thus there is no need to include opt_route.h
anymore in all consumers of vnet.h and no longer depend on it for module
builds.

Remove the hidden include in flowtable.h as well and leave the two
explicit #includes in ip_input.c and ip_output.c.

Hook ubt and ubtbcmfw back up to the build.

Garbage collect unbuildable and unusable non-MPSAFE network device
drivers that depended on the historic IFF_NEEDSGIANT compatibility
mechanism:

ar(4)
ray(4)
sr(4)

Discussed on: arch@

Remove IFF_NEEDSGIANT, a compatibility infrastructure introduced
in FreeBSD 5.x to allow network device drivers to run with Giant
despite the network stack being Giant-free. This significantly
simplifies calls into ioctl() on network interfaces, especially
in the multicast code, as well as eliminates deferred invocation
of interface if_start routines.

Disable the build on device drivers still depending on
IFF_NEEDSGIANT as they no longer compile. They will be removed
in a few weeks if they haven't been made MPSAFE in that time.
Disabled drivers:

if_ar
if_axe
if_aue
if_cdce
if_cue
if_kue
if_ray
if_rue
if_rum
if_sr
if_udav
if_ural
if_zyd

Drivers that were already disabled because of tty changes:

if_ppp
if_sl

Discussed on: arch@

Hook up new USB modules.

Rejoin ng_tty module to the build.

Hook up the ether_echo node and fix the man page

Add a trivial node to reflect ethernet frames to whence they came.

MFC after: 1 month

Rather than using hidden includes (with cicular dependencies),
directly include only the header files needed. This reduces the
unneeded spamming of various headers into lots of files.

For now, this leaves us with very few modules including vnet.h
and thus needing to depend on opt_route.h.

Reviewed by: brooks, gnn, des, zec, imp
Sponsored by: The FreeBSD Foundation

Per email to arch@ a little while ago (that was greeted with silence),
prefer the more common > ${.TARGET} over > opt_foo.h in modules
makefiles.

Disconnect drivers that haven't been ported to MPSAFE TTY yet.

As clearly mentioned on the mailing lists, there is a list of drivers
that have not been ported to the MPSAFE TTY layer yet. Remove them from
the kernel configuration files. This means people can now still use
these drivers if they explicitly put them in their kernel configuration
file, which is good.

People should keep in mind that after August 10, these drivers will not
work anymore. Even though owners of the hardware are capable of getting
these drivers working again, I will see if I can at least get them to a
compilable state (if time permits).

Hook up Bluetooth SCO sockets code to the build

MFC after: 3 months

Remove netatm from HEAD as it is not MPSAFE and relies on the now removed
NET_NEEDS_GIANT. netatm has been disconnected from the build for ten
months in HEAD/RELENG_7. Specifics:

- netatm include files
- netatm command line management tools
- libatm
- ATM parts in rescue and sysinstall
- sample configuration files and documents
- kernel support as a module or in NOTES
- netgraph wrapper nodes for netatm
- ctags data for netatm.
- netatm-specific device drivers.

MFC after: 3 weeks
Reviewed by: bz
Discussed with: bms, bz, harti

Make ng_h4(4) MPSAFE. Use similar to ng_tty(4) locking strategy.
Reconnect ng_h(4) back to the build.

Reviewed by: kensmith
Approved by: re (kensmith)
MFC after: 1 month

Disconnect netatm from the build as it is not MPSAFE and relies on
NET_NEEDS_GIANT, which will shortly be removed. This is done in a
away that it may be easily reattached to the build before 7.1 if
appropriate locking is added. Specifics:

- Don't install netatm include files
- Disconnect netatm command line management tools
- Don't build libatm
- Don't include ATM parts in rescue or sysinstall
- Don't install sample configuration files and documents
- Don't build kernel support as a module or in NOTES
- Don't build netgraph wrapper nodes for netatm

This removes the last remaining consumer of NET_NEEDS_GIANT.

Reviewed by: harti
Discussed with: bz, bms
Approved by: re (kensmith)

Mark ng_h4(4) as not MPSAFE and disconnect it from the build for now.

Approved by: re (rwatson)

A node that implements various traffic shaping and rate limiting algorithms.

Approved by: glebius (mentor)

Build bits for ng_deflate(4) and ng_pred1(4).

Don't use touch when what is really meant is :> (create an empty file, or
truncate it if it exists) or :>> (ensure the file exists, but don't change
it if it already does)

Reviewed by: ru
MFC after: 2 weeks

A netgraph node that can do different manipulations with
mbuf_tags(9) on packets.

Submitted by: Vadim Goncharov <vadimnuclight tpu.ru>
mdoc(7) reviewed by: ru

Reimplementation of world/kernel build options. For details, see:

http://lists.freebsd.org/pipermail/freebsd-current/2006-March/061725.html

The src.conf(5) manpage is to follow in a few days.

Brought to you by: imp, jhb, kris, phk, ru (all bugs are mine)

Style: Remove blank lines before EOF.

Tested by: cvs diff -B

Add BPF Just-In-Time compiler support for ng_bpf(4).

The sysctl is changed from net.bpf.jitter.enable to net.bpf_jitter.enable
and this controls both bpf(4) and ng_bpf(4) now.

Let modules use the kernel's opt_*.h files if built along with
the kernel by wrapping all targets for fake opt_*.h files in
.if defined(KERNBUILDDIR). Thus, such fake files won't be
created at all if modules are built with the kernel.

Some modules undergo cleanup like removing unused or unneeded
options or .h files, without which they wouldn't build this way
or the other.

Reviewed by: ru
Tested by: no binary changes in modules built alone
Tested on: i386 sparc64 amd64

Attach ng_tcpmss to the build.

Make NETGRAPH_DEBUG a kernel option, so that it can't be turned off
without hacking source.

In collaboration with: ru, julian

Attach ng_nat and libalias to build.

ng_nat - a netgraph(4) node, which does NAT

Connect ng_source(4) to the build.

Connect ng_atmllc(4) to the build.

Don't export symbols, all netgraph modules appear to have
proper module dependencies.

While here, removed stray -Wall from CFLAGS.

Bitrot: the ethernet parse type is long standard.

Add ng_ipfw to kernel module build.

Connect ng_device(4) to the build.

ar and sr (and their netgraph cousins) don't appear to be 64-bit clean, so
disable them on all but i386.

No need to generate vnode_if.h anymore

NOCRYPT -> NO_CRYPT

We already have ng_fec.4 manpage. And sys/modules is not a correct
place for documentation.

Sort SUBDIR.

Major overhaul.

List of functional changes:
- Make a single device per single node with a single hook.
This gives us parrallelizm, which can't be achieved on a single
node with many devices/hooks. This also gives us flexibility - we
can play with a particular device node, not affecting others.
- Remove read queue as it is. Use struct ifqueue instead. This change
removes a lot of extra memcpy()ing, m_devget()ting and m_copymem()ming.
In ng_device_receivedata() we enqueue an mbuf and wake readers.
In ngdread() we take one mbuf from qeueue and uiomove() it to
userspace. If no mbuf is present we optionally block. [1]
- In ngdwrite() we create an mbuf from uio using m_uiotombuf().
This is faster then uiomove() into buffer, and then m_copydata(),
and this is much better than huge m_pullup().
- Perform locking of device
- Perform locking of connection list.
- Clear out _rcvmsg method, since it does nothing good yet.
- Implement NGM_DEVICE_GET_DEVNAME message.
- #if 0 ioctl method, while nothing is done here yet.
- Return immediately from ngdwrite() if uio_resid == 0.

List of tidyness changes:
- Introduce device2priv(), to remove cut'n'paste.
- Use MALLOC/FREE, instead of malloc/free.
- Use unit2minor().
- Use UID_ROOT/GID_WHEEL instead of 0/0.
- Define NGD_DEVICE_DEVNAME, use it.
- Use more nice macros for debugging. [2]
- Return Exxx, not -1.

style(9) changes:
- No "#endif" after short block.
- Break long lines.
- Remove extra spaces, add needed spaces.

[1] Obtained from: if_tun.c
[2] Obtained from: ng_pppoe.c
Reviewed by: marks
Approved by: julian (mentor)
MFC after: 1 month

Attach ng_netflow to kernel build.

Approved by: julian (mentor)

Add the module build stuff for the ATM call control module.

Make bluetooth compile on all platforms

Reviewed by: imp, ru

Build usbdevs.h

Move to generating pccarddevs.h on the fly, both for the kernel and
the modules.

Also generate usbdevs.h automatically now, but a non-kernel file is
stopping that at the moment.

Add ng_sppp(4) to the modules build process.

A simple packet distribution node type that acts like an Ethernet hub.

sppp (4) to netgraph (4) node. As always: I'l connect it to the
system after extra check.

Approved by: imp (mentor)
Approved by: julian (in general)

Add a netgraph node to handle ATM LLC encapsulation. This currently handles
ethernet (tested) and FDDI (not tested). The main use for this is on ADSL (or
other ATM) connections where bridged ethernet is used, PPPoE being a prime
example.

There is no manual page as yet, I will write one shortly.

Reviewed by: harti

Netgraph node type for IEEE 802.1Q VLAN tagging.

Add NO_BLUETOOTH knob to the build process

Requested by: phk
Reviewed by: imp (mentor), ru

Normalize SUBDIR.

MODULE_DEPEND is a C macro, not a make(1).

bsd.kmod.mk does not deal with manpages anymore.

Don't compile with -g by default; there's a better way to build modules with
debug support.

The layer 3 (signalling) of NgATM netgraph node: ng_uni. This node
handles user and network side signaling and partly PNNI.

Put the address handling, traffic descripto handling and the
message encoding and decoding stuff into the base module. All of this
is accessed by several of the NgATM modules and putting this into
atmbase reduceds the memory footprint.

Module build infrastructure for the NgATM SAA layer.

Update Bluetooth code.

Reviewed by: M. Warner Losh <imp@bsdimp.com>; John Hay <jhay@freebsd.org>
Approved by: M. Warner Losh <imp@bsdimp.com> (mentor)

Add ng_atmpif: a HARP physical interface emulation. This allows one
to run the HARP ATM stack without real hardware.

Submitted by: Vincent Jardin <vjardin@wanadoo.fr>

This is a netgraph node to access ATM interfaces. It works with the
hatm(4) and fatm(4) drivers, en(4) will follow soon.

Part one of undating the bluetooth code to the newest version

Submitted by: Maksim Yevmenkin <m_evmenkin@yahoo.com>
Approved by: re@

Add device driver support for the ASIX Electronics AX88172 USB 2.0
ethernet controller. The driver has been tested with the LinkSys
USB200M adapter. I know for a fact that there are other devices out
there with this chip but don't have all the USB vendor/device IDs.

Note: I'm not sure if this will force the driver to end up in the
install kernel image or not. Special magic needs to be done to exclude
it to keep the boot floppies from bloating again, someone please
advise.

Fix module build by adding options to Makefile.

Take the rc4 code out of ng_mppc module so we don't fail to load when
we have the rc4 code already in the kernel (via wlan stuff or awi).
Add a dependency on the rc4 module so if it doesn't exist then load it.

Reviewed by: archie

Make it work -current style.

Add Makefile.inc to include ../Makefile.inc.

Approved by: re (rwatson)

Fix make buildkernel.

These makefiles work when building in the sys/modules directory, but
not with the objdir stuff that buildkernel uses. This is because they
used -I../../../blah rather than -I${.CURDIR}/../../../blah.

# I didn't fix the abuse of CFLAGS to specify -g since I wanted the
# barest minimal change since we're in a code freeze.

Approved by: make buildkernel...
Hat for armchair anarchists: core member fixing src tree damage

Make the bluetooth modules

Approved by: re

The second try a committing the bluetooth code
Has been seen to work on several cards and communicating with
several mobile phones to use them as modems etc.

We are still talking with 3com to try get them to allow us to include
the firmware for their pccard in the driver but the driver is here..
In the mean time
it can be downloaded from the 3com website and loaded using the utility
bt3cfw(8) (supplied) (instructions in the man page)

Not yet linked to the build

Submitted by: Maksim Yevmenkin <myevmenk@exodus.net>
Approved by: re

Add the netgraph 'source' module.
This is NOT YET CONVERTED TO -current.
This node is a source for preprogrammed packets at a known rate for testing.

I will convert it to -current "in place" but will MFC teh original
pre-conversion variant as that is what is originally submitted.
Man page my me, info from Dave's README.

Submitted by: Dave Chapeskie <dchapeskie@SANDVINE.com>
Obtained from: Sandvine inc.
MFC after: 1 week

Finally get around to committing Bill Paul's FEC netgraph nodes.
These are really only partly netgraph nodes as they do not use the
netgraph interfaces for many of the functions for which they could
be used, however they represent important functionality.

Submitted by: wpaul
MFC after: 2 days

Keep subdirectory list sorted.

Add ng_l2tp module.

A node that creates a device entry in /dev (yay devfs)
so that /dev/mumble can be the entrypoint to some networking graph,
e.g. a tunnel or a remote tape drive or whatever...

Not fully tested (by me) yet.

Submitted by: Mark Santcroos <marks@ripe.net>
MFC after: 3 weeks

turn on the ethertype filter module

Export "ng_ether_enaddr_type" because the ng_bridge type uses it.

Drop <bsd.man.mk> support from <bsd.kmod.mk>.

Not objected to by: -current

Turn on unconditional symbol export for modules whose API is
not clear enough, will require additional analysis, or will
require some input from their maintainers.

A node that allows ethernet type packets to be filtered to different
hooks depending on ethertype. Great for prototyping protocols.
connects to the lower and upper hooks of an ethernet type of node.

Obtained from: Monzoon Networks.
Thanks to Andre Oppermann, May 2001.

Add ng_ip_input. A new netgraph node for queuing IP packets into the
main IP input processing code.

/home/brooks/ng_gif.message

KSE Milestone 2
Note ALL MODULES MUST BE RECOMPILED
make the kernel aware that there are smaller units of scheduling than the
process. (but only allow one thread per process at this time).
This is functionally equivalent to teh previousl -current except
that there is a thread associated with each process.

Sorry john! (your next MFC will be a doosie!)

Reviewed by: peter@freebsd.org, dillon@freebsd.org

X-MFC after: ha ha ha ha

Make ng_split part of the module building process.

MFC after: 7 weeks

Put module directories one per line so adding new modules creates readable
diffs and produces less conflicts with local source trees.

Backed out rev.1.8. Rev.1.8 was just to support a bogus unused include
in ng_tty.c.

Move the isa parts to a separate file.

Allow the eiface node to be made as a module (not linked into build yet)

Add a Makefile for the sample node so It can be compiled.
This helps to stop it from geting out of sync.
It is not part of the normal build but I can use it with all the others
when I make changes to netgraph to ensure it is buildable.

Add a 'splitter' node to separate a bidirectional
packet flow into two unidirectional flows.

Part of a suite of nodes developed for packet flow control.
More to follow as I have time to port them to 5.x or
as others do so. The ipfw node will be the hardest..

Submitted by: "Vitaly V. Belekhov" <vitaly@riss-telecom.ru>

Make klds for the ar and sr devices and hook them up.

The normal versions are called if_ar.ko and if_sr.ko and the netgraph
versions ng_sync_ar.ko and ng_sync_sr.ko.

Use a consistent style and one much closer to the rest of /usr/src

Next phase in the PCI subsystem cleanup.

- Move PCI core code to dev/pci.
- Split bridge code out into separate modules.
- Remove the descriptive strings from the bridge drivers. If you
want to know what a device is, use pciconf. Add support for
broadly identifying devices based on class/subclass, and for
parsing a preloaded device identification database so that if
you want to waste the memory, you can identify *anything* we know
about.
- Remove machine-dependant code from the core PCI code. APIC interrupt
mapping is performed by shadowing the intline register in machine-
dependant code.
- Bring interrupt routing support to the Alpha
(although many platforms don't yet support routing or mapping
interrupts entirely correctly). This resulted in spamming
<sys/bus.h> into more places than it really should have gone.
- Put sys/dev on the kernel/modules include path. This avoids
having to change *all* the pci*.h includes.

Fix another callout_init() that I missed.

unhook netgraph/bridge until somebody makes it compile in LINT

Add one2many subdirectory (previously forgotten).

New netgraph node type ng_one2many(4).

Major update to the way synchronization is done in the kernel. Highlights
include:

* Mutual exclusion is used instead of spl*(). See mutex(9). (Note: The
alpha port is still in transition and currently uses both.)

* Per-CPU idle processes.

* Interrupts are run in their own separate kernel threads and can be
preempted (i386 only).

Partially contributed by: BSDi (BSD/OS)
Submissions by (at least): cp, dfr, dillon, grog, jake, jhb, sheldonh

New netgraph node type for Ethernet bridging.
No ipfw support yet.

When manual pages are moved or removed, MANX should be changed
to NOMAN.

Reported by: bde

Move the netgraph-related manual pages into share/man/man4/, as
discussed with archie.

Build ng_ether(4) KLD.

Use .include <bsd.kmod.mk> to get to ../../*/conf/kmod.mk instead of
encoding the relative path.

Update man page to reflect new IPv6 hook. Simplify Makefile.

Supply only one author name per instance of %A, as per mdoc.samples(7).

PR: 18465
Submitted by: Kazu TAKAMUNE <takamune@avrl.mei.co.jp>

AUTHOR -> AUTHORS

Document the new statistics control messages.
Add a note about the IP header asymmetry on the "lower" hook.

Move netgraph node type man pages from section 8 to section 4.

Pull in sys/conf/kmod.mk, rather than /usr/share/mk/bsd.kmod.mk.
This means that the kernel can be totally self contained now and is not
dependent on the last buildworld to update /usr/share/mk. This might
also make it easier to build 5.x kernels on 4.0 boxes etc, assuming
gensetdefs and config(8) are updated.

Zap KMODDEPS line

Update the man page to reflect the recent changes to the kernel API for
netgraph.

Only build the ng_mppc KLD if !NOCRYPT and required crypto sources exist.

Revert to previous state, plus a few cleanups pointed out by bde.
Instead, make the decision as to whether to build this KLD in ../Makefile.

Turn off build of ng_mppc KLD until I can figure out how to have the
Makefile correctly handle all the possible permutations (including
missing crypto sources). Suggestions welcome.

Turn off MPPE encryption if NOCRYPT is defined. If neither compression
nor encryption is enabled, don't build or install anything.

Add ng_mppc(8) netgraph node as a KLD module.

Obtained from: Whistle source tree

Remove duplicate word

- Document updates in iface netgraph node type functionality
- Remove makefile hacks for opt_inet.h, etc: the KLD now supports
whatever address families your kernel was compiled with automagically

Fix typo.

Reported by: Robert Swindells <swindellsr@genrad.co.uk>

Correct man page title.

Add a little more history of when netgraph appeared in FreeBSD.

Reviewed by: asmodai

Work around aparent bug in the .Dv macro by eliminating some spaces.
The closing curly-brace in this line was being omitted somehow.

Fix typos per PR 15649. Also did some rewording for clarity.

PR: docs/15649
Submitted by: Kazuo Horikawa <horikawa@jp.FreeBSD.org>

Revert previous checkin; I incorrectly thought that it was needed
due to having an old version of bsd.kmod.mk.

Caught by: bde

Manual page style work.
Submitted by: Alexey Zelkin <phantom@FreeBSD.org>
thanks!

Add reference to RFC 1702, after learning how to have "T. Li"
as an author without nroff blowing up.

Problem solved by: fenner

Update man page: BPF is supported even in the KLD version.

Fix 'make clean'

New netgraph node type 'pptpgre': this performs GRE encapsulation
for the PPTP protocol as specified in RFC 2637.

Include net/slcompress.c in the build of this module.

forgot .CURDIR
caught by Steven G. Kargl <kargl@troutmask.apl.washington.edu>

New netgraph node type, ng_bpf(8). This node type allows you to
apply bpf(4) filters to data travelling through a netgraph network.

Update documentation to reflect new functionality.

Fix typo.

Add two new generic control messages, NGM_ASCII2BINARY and
NGM_BINARY2ASCII, which convert control messages to ASCII and back.
This allows control messages to be sent and received in ASCII form
using ngctl(8), which makes ngctl a lot more useful.

This also allows all the type-specific debugging code in libnetgraph
to go away -- instead, we just ask the node itself to do the ASCII
translation for us.

Currently, all generic control messages are supported, as well as
messages associated with the following node types: async, cisco,
ksocket, and ppp.

See /usr/share/examples/netgraph/ngctl for an example of using this.

Also give ngctl(8) the ability to print out incoming data and
control messages at any time. Eventually nghook(8) may be subsumed.

Several other misc. bug fixes.

Reviewed by: julian

Removed special rules for building and cleaning device interface files
and empty options files. The rules are now generated automatically in
bsd.kmod.mk. Cleaned up related things ($S and ${CLEANFILES}).

Document the requirement for root privileges.

Document new feature (support for write(2) when 1 hook attached).

Update references.

Miscellaneous edits.

Add reference to ng_pppoe(8).

Update to reflect changed functionality.

Add reference to new netgraph node type "ksocket".

New Netgraph node type "ksocket".

Obtained from: Whistle source tree

Update to reflect changes in the node type.

Update to better reflect reality.

Add missing documentation for the two control messages supported
by this node type.

Add some more clarification.

Stop nroff from complaining about the freeBSD version..
Also some updates to the vjc node man page.

Bring the man page for netgraph socket nodes up to date.

Turn on the pppoe node module

bpf need no longer be an option

Fix some spelling errors.

Revamped and more useful PPP node type, supporting multi-link PPP directly.

Slight touchups noticed after checkin.

Add a man page for the pppoe node. Includes sample code.

Put tty back in, problem was due to cvsup mess on freefall.

Temporarily remove 'tty' netgraph sub module until it can be made
to compile.

Now that Netgraph is in the system there are some cleanups we can do.
Also save a slightly closer to completion version of the PPPOE code.

Submitted by: Archie Cobbs <archie@freebsd.org>

Whistle's Netgraph link-layer (sometimes more) networking infrastructure.
Been in production for 3 years now. Gives Instant Frame relay to if_sr
and if_ar drivers, and PPPOE support soon. See:
ftp://ftp.whistle.com/pub/archie/netgraph/index.html
for on-line manual pages.

Reviewed by: Doug Rabson (dfr@freebsd.org)
Obtained from: Whistle CVS tree

Test commit of first Netgraph file