xref: /haiku/docs/develop/midi/design.rst

Midi Kit design
===============

The Midi Kit consists of the midi_server and two shared libraries,
libmidi2.so and libmidi.so. The latter is the "old" pre-R5 Midi Kit and
has been re-implemented using the facilities from libmidi2, which makes
it fully compatible with the new kit. This document describes the design
and implementation of the OpenBeOS midi_server and libmidi2.so.

The midi_server has two jobs: it keeps track of the endpoints that the
client apps have created, and it publishes endpoints for the devices
from /dev/midi. (This last task could have been done by any other app,
but it was just as convenient to make the midi_server do that.) The
libmidi2.so library also has two jobs: it assists the midi_server with
the housekeeping stuff, and it allows endpoints to send and receive MIDI
events. (That's right, the midi_server has nothing to do with the actual
MIDI data.)

--------------

Ooh, pictures
-------------

The following image shows the center of Midi Kit activity, the
midi_server, and its data structures:

   |image0|

And here is the picture for libmidi2.so:

   |image1|

Note that these diagrams give only a conceptual overview of who is
responsible for which bits of data. The actual implementation details of
the kit may differ.

--------------

Housekeeping
------------

-  The design for our implementation of the midi2 "housekeeping"
   protocol roughly follows `what Be did <oldprotocol.html>`__, although
   there are some differences. In Be's implementation, the BMidiRosters
   only have BMidiEndpoints for remote endpoints if they are registered.
   In our implementation, the BMidiRosters have BMidiEndpoint objects
   for *all* endpoints, including remote endpoints that aren't published
   at all. If there are many unpublished endpoints in the system, our
   approach is less optimal. However, it made the implementation of the
   Midi Kit much easier ;-)

-  Be's libmidi2.so exports the symbols "midi_debug_level" and
   "midi_dispatcher_priority", both int32's. Our libmidi2 does not use
   either of these. But even though these symbols are not present in the
   headers, some apps may use them nonetheless. That's why our libmidi2
   exports those symbols as well.

-  The name of the message fields in Be's implementation of the protocol
   had the "be:" prefix. Our fields have a "midi:" prefix instead.
   Except for the fields in the B_MIDI_EVENT notification messages,
   because that would break compatibility with existing apps.

Initialization
~~~~~~~~~~~~~~

-  The first time an app uses a midi2 class, the
   BMidiRoster::MidiRoster() method sends an 'Mapp' message to the
   midi_server, and blocks (on a semaphore). This message includes a
   messenger to the app's BMidiRosterLooper object. The server adds the
   app to its list of registered apps. Then the server asynchronously
   sends back a series of 'mNEW' message notifications for all endpoints
   on the roster, and 'mCON' messages for all existing connections. The
   BMidiRosterLooper creates BMidiEndpoint objects for these endpoints
   and adds them to its local roster; if the app is watching, it also
   sends out corresponding B_MIDI_EVENT notifications. Finally, the
   midi_server sends an 'mAPP' message to notify the app that it has
   been successfully registered. Upon receipt, BMidiRoster::MidiRoster()
   unblocks and returns control to the client code. This handshake is
   the only asynchronous message exchange; all the other requests have a
   synchronous reply.

-  If the server detects an error during any of this (incorrect message
   format, delivery failure, etc.) it simply ignores the request and
   does not try to send anything back to the client (which is most
   likely impossible anyway). If the app detects an error (server sends
   back meaningless info, cannot connect to server), it pretends that
   everything is hunkey dorey. (The API has no way of letting the client
   know that the initialization succeeded.) Next time the app tries
   something, the server either still does not respond, or it ignores
   the request (because this app isn't properly registered). However, if
   the app does not receive the 'mAPP' message, it will not unblock, and
   remains frozen for all eternity.

-  BMidiRoster's MidiRoster() method creates the one and only
   BMidiRoster instance on the heap the first time it is called. This
   instance is automatically destroyed when the app quits.

Error handling
~~~~~~~~~~~~~~

-  If some error occurs, then the reply message is only guaranteed to
   contain the "midi:result" field with some non- zero error code.
   libmidi2 can only assume that the reply contains other data on
   success (i.e. when "midi:result" is B_OK).

-  The timeout for delivering and responding to a message is about 2
   seconds. If the client receives no reply within that time, it assumes
   the request failed. If the server cannot deliver a message within 2
   seconds, it assumes the client is dead and removes it (and its
   endpoints) from the roster. Of course, these assumptions may be
   false. If the client wasn't dead and tries to send another request to
   the server, then the server will now ignore it, since the client app
   is no longer registered.

-  Because we work with timeouts, we must be careful to avoid
   misunderstandings between the midi_server and the client app. Both
   sides must recognize the timeout, so they both can ignore the
   operation. If, however, the server thinks that everything went okay,
   but the client flags an error, then the server and the client will
   have two different ideas of the current state of the roster. Of
   course, those situations must be avoided.

-  Although apps register themselves with the midi_server, there is no
   corresponding "unregister" message. The only way the server
   recognizes that an app and its endpoints are no longer available is
   when it fails to deliver a message to that app. In that case, we
   remove the app and all its endpoints from the roster. To do this, the
   server sends "purge endpoint" messages to itself for all of the app's
   endpoints. This means we don't immediately throw the app away, but we
   schedule that for some time in the future. That makes the whole event
   handling mechanism much cleaner. There is no reply to the purge
   request. (Actually, we *do* immediately throw away the app_t object,
   since that doesn't really interfere with anything.) (If there are
   other events pending in the queue which also cause notifications,
   then the server may send multiple purge messages for the same
   endpoints. That's no biggie, because a purge message will be ignored
   if its endpoint no longer exists.)

-  As mentioned above, the midi_server ignores messages that do not come
   from a registered app, although it does send back an error reply. In
   the case of the "purge endpoint" message, the server makes sure the
   message was local (i.e. sent by the midi_server itself).

-  Note: BMessage's SendReply() apparently succeeds even if you kill the
   app that the reply is intended for. This is rather strange, and it
   means that you can't test delivery error handling for replies by
   killing the app. (You *can* kill the app for testing the error
   handling on notifications, however.)

Creating and deleting endpoints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  When client code creates a new BMidiLocalProducer or
   BMidiLocalConsumer endpoint, we send an 'Mnew' message to the server.
   Unlike Be's implementation, the "name" field is always present, even
   if the name is empty. After adding the endpoint to the roster, the
   server sends 'mNEW' notifications to all other applications. Upon
   receipt of this notification, the BMidiRosterLoopers of these apps
   create a new BMidiEndpoint for the endpoint and add it to their
   internal list of endpoints. The app that made the request receives a
   reply with a single "midi:result" field.

-  When you "new" an endpoint, its refcount is 1, even if the creation
   failed. (For example, if the midi_server does not run.) When you
   Acquire(), the refcount is bumped. When you Release(), it is
   decremented. When refcount drops to 0, the endpoint object "deletes"
   itself. (So client code should never use an endpoint after having
   Release()'d it, because the object may have just been killed.) When
   creation succeeds, IsValid() returns true and ID() returns a valid ID
   (> 0). Upon failure, IsValid() is false and ID() returns 0.

-  After the last Release() of a local endpoint, we send 'Mdel' to let
   the midi_server know the endpoint is now deleted. We don't expect a
   reply back. If something goes wrong, the endpoint is deleted
   regardless. We do not send separate "unregistered" notifications,
   because deleting an endpoint implies that it is removed from the
   roster. For the same reason, we also don't send separate
   "disconnected" notifications.

-  The 'mDEL' notification triggers a BMidiRosterLooper to remove the
   corresponding BMidiEndpoint from its internal list. This object is
   always a proxy for a remote endpoint. The remote endpoint is gone,
   but whether we can also delete the proxy depends on its reference
   count. If no one is still using the object, its refcount is zero, and
   we can safely delete the object. Otherwise, we must defer destruction
   until the client Release()'s the object.

-  If you "delete" an endpoint, your app drops into the debugger.

-  If you Release() an endpoint too many times, your app *could* drop
   into the debugger. It might also crash, because you are now using a
   dead object. It depends on whether the memory that was previously
   occupied by your endpoint object was overwritten in the mean time.

-  You are allowed to pass NULL into the constructors of
   BMidiLocalConsumer and BMidiLocalProducer, in which case the
   endpoint's name is simply an empty string.

Changing endpoint attributes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  An endpoint can be "invalid". In the case of a proxy this means that
   the remote endpoint is unregistered or even deleted. Local endpoints
   can only be invalid if something went wrong during their creation (no
   connection to server, for example). You can get the attributes of
   invalid objects, but you cannot set them. Any attempts to do so will
   return an error code.

-  For changing the name, latency, or properties of an endpoint,
   libmidi2 sends an 'Mchg' message with the fields that should be
   changed, "midi:name", "midi:latency", or "midi:properties".
   Registering or unregistering an endpoint also sends such an 'Mchg'
   message, because we consider the "registered" state also an
   attribute, in "midi:registered". The message obviously also includes
   the ID of the endpoint in question. Properties are sent using a
   different message, because the properties are not stored inside the
   BMidiEndpoints.

-  After handling the 'Mchg' request, the midi_server broadcasts an
   'mCHG' notification to all the other apps. This message has the same
   contents as the original request.

-  If the 'Mchg' message contains an invalid "midi:id" (i.e. no such
   endpoint exists or it does not belong to the app that sent the
   request), the midi_server returns an error code, and it does not
   notify the other apps.

-  If you try to Register() an endpoint that is already registered,
   libmidi2 does not send a message to the midi_server but simply
   returns B_OK. (Be's implementation *did* send a message, but our
   libmidi2 also keeps track whether an endpoint is registered or not.)
   Although registering an endpoint more than once doesn't make much
   sense, it is not considered an error. Likewise for Unregister()ing an
   endpoint that is not registered.

-  If you try to Register() or Unregister() a remote endpoint, libmidi2
   immediately returns an error code, and does not send a message to the
   server. Likewise for a local endpoints that are invalid (i.e. whose
   IsValid() function returns false).

-  BMidiRoster::Register() and Unregister() do the same thing as
   BMidiEndpoint::Register() and Unregister(). If you pass NULL into
   these functions, they return B_BAD_VALUE.

-  SetName() ignores NULL names. When you call it on a remote endpoint,
   SetName() does nothing. SetName() does not send a message if the new
   name is the same as the current name.

-  SetLatency() ignores negative values. SetLatency() does not send a
   message if the new latency is the same as the current latency. (Since
   SetLatency() lives in BMidiLocalConsumer, you can never use it on
   remote endpoints.)

-  We store a copy of the endpoint properties in each BMidiEndpoint. The
   properties of new endpoints are empty. GetProperties() copies this
   BMessage into the client's BMessage. GetProperties() returns NULL if
   the message parameter is NULL.

-  SetProperties() returns NULL if the message parameter is NULL. It
   returns an error code if the endpoint is remote or invalid.
   SetProperties() does *not* compare the contents of the new BMessage
   to the old, so it will always send out the change request.

Connections
~~~~~~~~~~~

-  BMidiProducer::Connect() sends an 'Mcon' request to the midi_server.
   This request contains the IDs of the producer and the consumer you
   want to connect. The server sends back a reply with a result code. If
   it is possible to make this connection, the server broadcasts an
   'mCON' notification to all other apps. In one of these apps the
   producer is local, so that app's libmidi2 calls the
   BMidiLocalProducer::Connected() hook.

-  You are not allowed to connect the same producer and consumer more
   than once. The midi_server checks for this. It also returns an error
   code if you try to disconnect two endpoints that were not connected.

-  Disconnect() sends an 'Mdis' request to the server, which contains
   the IDs of the producer and consumer that you want to disconnect. The
   server replies with a result code. If the connection could be broken,
   it also sends an 'mDIS' notification to the other apps. libmidi2
   calls the local producer's BMidiLocalProducer::Disconnected() hook.

-  Connect() and Disconnect() immediately return an error code if you
   pass a NULL argument, or if the producer or consumer is invalid.

-  When you Release() a local consumer that is connected, all apps will
   go through their producers, and throw away this consumer from their
   connection lists. If one of these producers is local, we call its
   Disconnected() hook. If you release a local producer, this is not
   necessary.

Watching
~~~~~~~~

-  When you call StartWatching(), the BMidiRosterLooper remembers the
   BMessenger, and sends it B_MIDI_EVENT notifications for all
   registered remote endpoints, and the current connections between
   them. It does not let you know about local endpoints. When you call
   StartWatching() a second time with the same BMessenger, you'll
   receive the whole bunch of notifications again. StartWatching(NULL)
   is not allowed, and will be ignored (so it is not the same as
   StopWatching()).

Thread safety
~~~~~~~~~~~~~

-  Within libmidi2 there are several possible race conditions, because
   we are dealing with two threads: the one from BMidiRosterLooper and a
   thread from the client app, most likely the BApplication's main
   thread. Both can access the same data: BMidiEndpoint objects. To
   synchronize these threads, we lock the BMidiRosterLooper, which is a
   normal BLooper. Anything happening in BMidiRosterLooper's message
   handlers is safe, because BLoopers are automatically locked when
   handling a message. Any other operations (which run from a different
   thread) must first lock the looper if they access the list of
   endpoints or certain BMidiEndpoint attributes (name, properties,
   etc).

-  What if you obtain a BMidiEndpoint object from FindEndpoint() and at
   the same time the BMidiRosterLooper receives an 'mDEL' request to
   delete that endpoint? FindEndpoint() locks the looper, and bumps the
   endpoint object before giving it to you. Now the looper sees that the
   endpoint's refcount is larger than 0, so it won't delete it (although
   it will remove the endpoint from its internal list). What if you
   Acquire() or Release() a remote endpoint while it is being deleted by
   the looper? That also won't happen, because if you have a pointer to
   that endpoint, its refcount is at least 1 and the looper won't delete
   it.

-  It is not safe to use a BMidiEndpoint and/or the BMidiRoster from
   more than one client thread at a time; if you want to do that, you
   should synchronize access to these objects yourself. The only
   exception is the Spray() functions from BMidiLocalProducer, since
   most producers have a separate thread to spray their MIDI events.
   This is fine, as long as that thread isn't used for anything else,
   and it is the only one that does the spraying.

-  BMidiProducer objects keep a list of consumers they are connected to.
   This list can be accessed by several threads at a time: the client's
   thread, the BMidiRosterLooper thread, and possibly a separate thread
   that is spraying MIDI events. We could have locked the producer using
   BMidiRosterLooper's lock, but that would freeze everything else while
   the producer is spraying events. Conversely, it would freeze all
   producers while the looper is talking to the midi_server. To lock
   with a finer granularity, each BMidiProducer has its own BLocker,
   which is used only to lock the list of connected consumers.

Misc remarks
~~~~~~~~~~~~

-  BMidiEndpoint keeps track of its local/remote state with an "isLocal"
   variable, and whether it is a producer/consumer with "isConsumer". It
   also has an "isRegistered" field to remember whether this endpoint is
   registered or not. Why not lump all these different states together
   into one "flags" bitmask? The reason is that isLocal only makes sense
   to this application, not to others. Also, the values of isLocal and
   isConsumer never change, but isRegistered does. It made more sense
   (and clearer code) to separate them out. Finally, isRegistered does
   not need to be protected by a lock, even though it can be accessed by
   multiple threads at a time. Reading and writing a bool is atomic, so
   this can't get messed up.

The messages
~~~~~~~~~~~~

   ::

      Message: Mapp (MSG_REGISTER_APP)
          BMessenger midi:messenger
      Reply: 
          (no reply)

      Message: mAPP (MSG_APP_REGISTERED)
          (no fields)

      Message: Mnew (MSG_CREATE_ENDPOINT)
          bool     midi:consumer
          bool     midi:registered
          char[]   midi:name
          BMessage midi:properties
          int32    midi:port       (consumer only)
          int64    midi:latency    (consumer only)
      Reply:
          int32    midi:result
          int32    midi:id

      Message: mNEW (MSG_ENPOINT_CREATED)
          int32    midi:id
          bool     midi:consumer
          bool     midi:registered
          char[]   midi:name
          BMessage midi:properties
          int32    midi:port       (consumer only)
          int64    midi:latency    (consumer only)

      Message: Mdel (MSG_DELETE_ENDPOINT)
          int32 midi:id
      Reply: 
          (no reply)

      Message: Mdie (MSG_PURGE_ENDPOINT)
          int32 midi:id
      Reply: 
          (no reply)

      Message: mDEL (MSG_ENDPOINT_DELETED)
          int32 midi:id

      Message: Mchg (MSG_CHANGE_ENDPOINT)
          int32    midi:id
          int32    midi:registered (optional)
          char[]   midi:name       (optional)
          int64    midi:latency    (optional)
          BMessage midi:properties (optional)
      Reply:
          int32 midi:result

      Message: mCHG (MSG_ENDPOINT_CHANGED)
          int32 midi:id
          int32    midi:registered (optional)
          char[]   midi:name       (optional)
          int64    midi:latency    (optional)
          BMessage midi:properties (optional)

--------------

MIDI events
-----------

-  MIDI events are always sent from a BMidiLocalProducer to a
   BMidiLocalConsumer. Proxy endpoint objects have nothing to do with
   this. During its construction, the local consumer creates a kernel
   port. The ID of this port is published, so everyone knows what it is.
   When a producer sprays an event, it creates a message that it sends
   to the ports of all connected consumers.

-  This means that the Midi Kit considers MIDI messages as discrete
   events. Hardware drivers chop the stream of incoming MIDI data into
   separate events that they send out to one or more kernel ports.
   Consumers never have to worry about parsing a stream of MIDI data,
   just about handling a bunch of separate events.

-  Each BMidiLocalConsumer has a (realtime priority) thread associated
   with it that waits for data to arrive at the port. As soon as a new
   MIDI message comes in, the thread examines it and feeds it to the
   Data() hook. The Data() hook ignores the message if the "atomic" flag
   is false, or passes it on to one of the other hook functions
   otherwise. Incoming messages are also ignored if their contents are
   not valid; for example, if they have too few or too many bytes for a
   certain type of MIDI event.

-  Unlike the consumer, BMidiLocalProducer has no thread of its own. As
   a result, spraying MIDI events always happens in the thread of the
   caller. Because the consumer port's queue is only 1 message deep,
   spray functions will block if the consumer thread is already busy
   handling another MIDI event. (For this reason, the Midi Kit does not
   support interleaving of real time messages with lower priority
   messages such as sysex dumps, except at the driver level.)

-  The producer does not just send MIDI event data to the consumer, it
   also sends a 20-byte header describing the event. The total message
   looks like this:

      +---------+------------------------------+
      | 4 bytes | ID of the producer           |
      +---------+------------------------------+
      | 4 bytes | ID of the consumer           |
      +---------+------------------------------+
      | 8 bytes | performance time             |
      +---------+------------------------------+
      | 1 byte  | atomic (1 = true, 0 = false) |
      +---------+------------------------------+
      | 3 bytes | padding (0)                  |
      +---------+------------------------------+
      | x bytes | MIDI event data              |
      +---------+------------------------------+

-  In the case of a sysex event, the SystemExclusive() hook is only
   called if the first byte of the message is 0xF0. The sysex end marker
   (0xF7) is optional; only if the last byte is 0xF7 we strip it off.
   This is unlike Be's implementation, which all always strips the last
   byte even when it is not 0xF7. According to the MIDI spec, 0xF7 is
   not really required; any non-realtime status byte ends a sysex
   message.

-  SprayTempoChange() sends 0xFF5103tttttt, where tttttt is
   60,000,000/bpm. This feature is not really part of the MIDI spec, but
   an extension from the SMF (Standard MIDI File) format. Of course, the
   TempoChange() hook is called in response to this message.

-  The MIDI spec allows for a number of shortcuts. A Note On event with
   velocity 0 is supposed to be interpreted as a Note Off, for example.
   The Midi Kit does not concern itself with these shortcuts. In this
   case, it still calls the NoteOn() hook with a velocity parameter of
   0.

-  The purpose of BMidiLocalConsumer's AllNotesOff() function is not
   entirely clear. All Notes Off is a so-called "channel mode message"
   and is generated by doing a SprayControlChange(channel,
   B_ALL_NOTES_OFF, 0). BMidi has an AllNotesOff() function that sends
   an All Notes Off event to all channels, and possible Note Off events
   to all keys on all channels as well. I suspect someone at Be was
   confused by AllNotesOff() being declared "virtual", and thought it
   was a hook function. Only that would explain it being in
   BMidiLocalConsumer as opposed to BMidiLocalProducer, where it would
   have made sense. The disassembly for Be's libmidi2.so shows that
   AllNotesOff() is empty, so to cut a long story short, our
   AllNotesOff() simply does nothing and is never invoked either.

-  There are several types of System Common events, each of which takes
   a different number of data bytes (0, 1, or 2). But
   SpraySystemCommon() and the SystemCommon() hook are always given 2
   data parameters. The Midi Kit simply ignores the extra data bytes; in
   fact, in our implementation it doesn't even send them. (The Be
   implementation always sends 2 data bytes, but that will confuse the
   Midi Kit if the client does a SprayData() of a common event instead.
   In our case, that will still invoke the SystemCommon() hook, because
   we are not as easily fooled.)

-  Handling of timeouts is fairly straightforward. When reading from the
   port, we specify an absolute timeout. When the port function returns
   with a B_TIMED_OUT error code, we call the Timeout() hook. Then we
   reset the timeout value to -1, which means that timeouts are disabled
   (until the client calls SetTimeout() again). This design means that a
   call to SetTimeout() only takes effect the next time we read from the
   port, i.e. after at least one new MIDI event is received (or the
   previous timeout is triggered). Even though BMidiLocalConsumer's
   timeout and timeoutData values are accessed by two different threads,
   I did not bother to protect this. Both values are int32's and
   reading/writing them should be an atomic operation on most processors
   anyway.

.. |image0| image:: midi_server.png
.. |image1| image:: libmidi2.png