From SandboxWiki
Jump to: navigation, search

Contact: sandbox-developers at movial.com

Source code

Source browser is available at http://sandbox.movial.com/gitweb?p=isatis.git;a=summary

Isatis Multimedia Plugin implementation document

The multimedia plugin (mm-plugin) provides media playback properties by integrating GStreamer with the browser enabling multimedia to run in the browser.

The plugin is a dynamic library, and runs only from the browser. It uses a separate player executable to provide the actual multimedia playback. The player uses the GStreamer framework to decode various multimedia formats.


The general functionality

When the browser starts, it enumerates the available plugins, querying the MIME types for that plugin, and registers each plugin library for its MIME types in an internal data structure.

The life cycle of a plugin is completely controlled by the browser and the web page that calls it:

When the user opens a page that contains embedded data of media type that invokes a plugin, the browser checks for a plugin with a matching MIME type. It dynamically loads the plugin code into memory and initializes it (NP_Initialize), if not loaded and initialized already. Finally, the browser creates a new instance of the plugin (NPP_New).

Gecko can load multiple instances of the same plugin on a single page, or on multiple open tabs and windows at the same time, if the user is browsing pages containing several multimedia clips. In the case of !WebKit and Chrome, a plugin is initialized for each tab and all the instances of the same plugin are inside that one tab.

Once the plugin has received the stream URI, window XID and communications socket file descriptor, it launches a new player instance and passes the parameters on the command line.

When the user leaves the instance's page or closes its window or tab, the plugin instance is deleted (NPP_Destroy). When the last instance of a plugin is deleted, the plugin code is unloaded from memory (NP_Shutdown).

The plugin contains a hardcoded list of MIME types it claims to support rather than querying them dynamically because at least Gecko-based browsers will cache the information rather than re-querying every time, and will only update the cache when the timestamp of the plugin library has changed.

Plugin interface

The plugin is a dynamic library written in C. The Netscape Plugin Application Programming Interface (NPAPI) is made up of a set of shared data structures and two groups of functions:

  • the plugin-specific functions, provided by plugin to browser, named with the prefix "NPP_"
  • the browser-specific functions, provided by browser to plugin, named with the prefix "NPN_"

There are also a couple of functions, that are direct library entry points and not related to any particular plugin instance. These functions are named with the prefix "NP_".

The plugin provides the following functions to browser:

  • NP_Initialize
    • initializes the plugin class
    • called after the plugin library has been loaded
  • NPP_New
    • creates a new instance of the plugin
    • called after NP_Initialize and when multimedia content encountered
  • NPP_Destroy
    • called when the user leaves the instance page or closes the instance window
    • frees resources allocated by the plugin instance
    • terminates the player process associated with the plugin instance
  • NPP_SetWindow
    • initially called to inform the plugin of the window XID and other properties
    • may be called to notify the plugin of changed window properties
  • NPP_NewStream
    • notifies a plug-in instance of a new data stream
    • called when a stream is created
  • NPP_DestroyStream
    • called when a data stream is destroyed by the browser
  • NPP_WriteReady
    • determines whether the plugin can receive data and how many bytes it can receive
    • called when data is available, before each call to NPP_Write
  • NPP_Write
    • receives data from an incoming stream
    • may be called after NPP_WriteReady has returned a value indicating more data can be received by the plugin
  • NPP_GetValue
    • called by the browser to query various properties of the plugin instance
  • NP_Shutdown
    • called when the browser is about to unload the plugin library from memory
    • releases any remaining resources used by the plugin
  • NP_GetMIMEDescription
    • tells which MIME types and file extensions the plugin handles
    • called after the plugin is loaded
  • NP_GetValue
    • called by the browser to query properties of the plugin library
    • internally redirected to NPP_GetValue


Implementation

The plugin utilizes a native player executable using GStreamer framework for the actual media playback.

After a new plugin instance has received the initial window parameters (most importantly the XID of the window) and the input stream information (the URI to data) it launches a new process running the actual player binary. All the important information (XID, URL, communications socket filedescriptor) are passed to the player process via commandline arguments.

The plugin (running inside the browser process) and the player (running in its own process) communicate with a two-way local socket. The communication consists of packets composed of 1 byte packet type indicator, a native format 32bit integer for payload length and optionally a variable number of bytes of payload.

Currently defined packet types are:

  • Player to plugin:
    • ISATIS_CMD_START ('1', payload contains the URI to stream from)
    • ISATIS_CMD_STOP ('0', no payload)
  • Plugin to player:
    • ISATIS_CMD_DATA ('D', payload contains stream data)
    • ISATIS_CMD_EOS ('E', no payload)
    • ISATIS_CMD_STOPPED ('S', no payload)
Mm-plugin-integration-new2.png

Browser toolkit integration

The browser plugin will currently only function in browsers supporting XEmbed and using GTK2 as the windowing/event handling toolkit. This is due to the lack of timers or event notifications in NPAPI and the need to have two-way communication between the plugin and the player.

The player

The player runs as a separate process to avoid crashing or hanging the browser with the relatively unstable multimedia codecs.

GStreamer multimedia framework is used in auto-plugging mode by the player, in which the framework will attempt to utilise the best available plugins for decoding and displaying the media. In case of the player being launched from the plugin it associates the HTTP protocol to be handled by its own special URI handler, which ensures that the GStreamer framework still considers the content to be from a streaming source (see buffering) yet allows us to receive the data via the browser plugin.

Mm-plugin-player.png

The player contains the following internal functions:

  • start_playback
    • sets pipeline to playing-state
  • pause_playback
    • sets pipeline to paused-state
  • stop_playback
    • sets pipeline to stopped-state
  • parse_args
    • parses the commandline arguments
  • setup_ui
    • builds the user interface
    • connects callbacks to GTK+ signals
  • main
    • creates a new instance of a player
    • connects callbacks to GStreamer signals

The following are connected to various signals emitted by GStreamer:

  • prepare_xwindow_id_callback
    • connected to a synchronous signal, called in the output thread context
    • called when the pipeline is ready to receive XID of the output window
  • update_progress_callback
    • updates the progress bar and position & duration indicator text
  • state_changed_callback
    • called when the playback pipeline has changed state
  • error_callback
    • called after an unrecoverable error has happened in the playback pipeline
  • eos_callback
    • called when playback has finished (end of stream)
  • buffering_callback
    • called when the pipeline is waiting for more data from network
    • provided with the percentage of the buffer currently filled
  • element_callback
    • called for element-specific events
    • checks for redirects to a new URI

The following functions are connected to signals emitted by GTK+:

  • paint_label_background_callback
    • used for skinning the player UI
  • playbutton_clicked_callback
    • toggles the state between playing and paused
  • stopbutton_clicked_callback
    • stops playback
    • called when stop button has been clicked

The player also contains a custom implementation of a !GstPushSrc source element, which is used when the player is launched from the browser plugin to redirect access to the resource to happen via the browser.

The player has three states: Stopped, Playing and Paused. The player is initially on state Stopped. When user clicks the Play button, the state changes to Playing, in which the user can click Stop to return to Stopped state or Pause to change the state to Paused. In Paused state the user can click Stop to change the state to Stopped or Play to continue in Playing mode. If the user clicks Pause button in Stopped state nothing happens. The transitions are illustrated below.

Mm-plugin-states.png

Buffering

The player relies on the internal buffering mechanisms of GStreamer. Because the list of protocols GStreamer considers to be from networked sources is fixed (at least in current releases), the player installs its own handler for HTTP protocol datasources when launched from the browser plugin. When the GStreamer pipeline needs to buffer more data it sends an event on the GstBus associated with the pipeline, which in turn fires a signal that ends up calling buffering_callback. When this happens, the pipeline is set to a temporary PAUSED state until the buffering is complete (an event with buffer percentage of 100% is received) after which the pipeline will be set back to the desired state (PLAYING or PAUSED).


Redirects

Some multimedia containers may contain redirections to other files (eg. www.apple.com/trailers uses this approach). In this case the pipeline will issue a callback to element_callback signal handler, which will check the message whether it contains a redirection and if so, sets the pipeline to STOPPED state, changes the URI to the new one (resolving relative URIs if necessary) and sets the pipeline back to PLAYING state.

User interface

The player's user interface is very simple. In addition to the video playing screen the UI has:

  • play/pause toggle-button
  • stop button
  • progress bar
  • time played and total length of the media (when available)

Below is a picture of the user interface:

Mm-plugin-screenshot.png

For more information, please see: