pion / mediadevices

Pion MediaDevices Go implementation of the MediaDevices API provides access to media input devices like cameras, microphones, and screen capture. It can also be used to encode your video/audio stream to various codec selections. abstracts away the complexities of interacting with things like hardware and codecs allowing you to focus on building appilcations, interacting only with an amazingly simple, easy, and elegant API! Install Build Tags provides a build tag. Use this tag to build without microphone support. This is particularly useful for: • Cross-compilation where CGO dependencies like are unavailable. • Projects where audio is not required. Usage The following snippet shows how to capture a camera stream and store a frame as a jpeg image: More Examples • Webrtc - Use Webrtc to create a realtime peer-to-peer video call • Face Detection - Use a machine learning algorithm to detect faces in a camera stream • RTP Stream - Capture camera stream, encode it in H264/VP8/VP9, and send it to a RTP server • HTTP Broadcast - Broadcast camera stream through HTTP with MJPEG • Archive - Archive H264 encoded video stream from a camera Available Media Inputs | Input | Linux | Mac | Windows | | :--------: | :---: | :-: | :-----: | | Camera | ✔️ | ✔️ | ✔️ | | Microphone | ✔️ | ✔️ | ✔️ | | Screen | ✔️ | ✔️ | ✔️ | By default, there's no media input registered. This decision was made to allow you to play only what you need. Therefore, you need to import the associated packages for the media inputs. For example, if you want to use a camera, you need to import the camera package as a side effect: Available Codecs In order to encode your video/audio, needs to know what codecs that you want to use and their parameters. To do this, you need to import the associated packages for the codecs, and add them to the codec selector that you'll pass to : Since doesn't implement the video/audio codecs, it needs to call the codec libraries from the system through cgo. Therefore, you're required to install the codec libraries before you can use them in . In the next section, it shows a list of available codecs, where the packages are defined (documentation linked), and installation instructions. Note: we do not provide recommendations on choosing one codec or another as it is very complex and can be subjective. Video Codecs x264 A free software library and application for encoding video streams into the H.264/MPEG-4 AVC compression format. • Package: github.com/pion/mediadevices/pkg/codec/x264 • Installation: • Mac: • Ubuntu: mmal A framework to enable H264 hardware encoding for Raspberry Pi or boards that use VideoCore GPUs. • Package: github.com/pion/mediadevices/pkg/codec/mmal • Installation: no installation needed, mmal should come built in Raspberry Pi devices openh264 A codec library which supports H.264 encoding and decoding. It is suitable for use in real time applications. • Package: github.com/pion/mediadevices/pkg/codec/openh264 • Installation: no installation needed, included as a static binary svtav1 A free software video codec library from the Alliance for Open Media that implements AV1 video coding formats. • Package: github.com/pion/mediadevices/pkg/codec/svtav1 • Installation: • Mac: • Ubuntu: vpx A free software video codec library from Google and the Alliance for Open Media that implements VP8/VP9 video coding formats. • Package: github.com/pion/mediadevices/pkg/codec/vpx • Installation: • Mac: • Ubuntu: vaapi An open source API that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities (currently support VP8/VP9). • Package: github.com/pion/mediadevices/pkg/codec/vaapi • Installation: • Ubuntu: Audio Codecs opus A totally open, royalty-free, highly versatile audio codec. • Package: github.com/pion/mediadevices/pkg/codec/opus • Installation: • Mac: • Ubuntu: Benchmark Result as of Nov 4, 2020 with Go 1.14 on a Raspberry pi 3, can produce video, encode, send across network, and decode at **720p, 30 fps with < 500 ms latency**. The test was taken by capturing a camera stream, decoding the raw frames, encoding the video stream with mmal, and sending the stream through Webrtc. FAQ Failed to find the best driver that fits the constraints provides an automated driver discovery through and . The driver discover algorithm works something like: • Open all registered drivers • Get all properties (property describes what a driver is capable of, e.g. resolution, frame rate, etc.) from opened drivers • Find the best property that meets the criteria So, when returns error, one of the following conditions might have occured: • Driver was not imported as a side effect in your program, e.g. • Your constraint is too strict that there's no driver can fullfil your requirements. In this case, you can try to turn up the debug level by specifying the following environment variable: to see what was too strict and tune that. • Your driver is not supported/implemented. In this case, you can either let us know (file an issue) and wait for the maintainers to implement it. Or, you can implement it yourself and register it through • If trying to use note that you will need to use instead of Failed to find vpx/x264/mmal/opus codecs Since uses cgo to access video/audio codecs, it needs to find these libraries from the system. To accomplish this, pkg-config is used for library discovery. If you see the following error message at compile time: There are 2 common problems: • The required codec library is not installed (vpx in this example). In this case, please refer to the available codecs. • Pkg-config fails to find the files for this codec (reference). In this case, you need to find where the codec library's is stored, and let pkg-config knows with: . Roadmap The library can be used with our WebRTC implementation. Please refer to that roadmap to track our major milestones. Community Pion has an active community on Discord. Follow the Pion Twitt…