DIY video monitoring system – Part IV Video streaming using MJPEG
Now that we know how to serve single images from our Raspberry Pi Camera, it is time to do some video streaming. To make this happen, we’re going to use the MJPEG video format.
If you haven’t seen my previous posts on DIY video monitoring system, I strongly encourage you read them first as some bits of the code below are coming from them.
MJPEG
Before we start modifying our code, let’s quickly talk about MJPEG. M-JPEG, or Motion JPEG, is a video compression format, where each video frame is compressed as a JPEG image. It is a widely supported format, often used by video capturing devices like webcams or IP cameras. What interests us mostly, is the fact that it’s natively supported by modern web browser (Chrome, Firefox, Safari, Edge) so no additional plugins are required to play it back in one of them.
When streaming MJPEG over HTTP, we start with a GET request from the client browser and in response, we get a bunch of headers that tell the browser how to handle it. Most notable response headers are:
Content-Type: multipart/x-mixed-replace;boundary="<boundary_name>"
It’s a special mime-type content type header that tells the client to expect the response to consist of multiple parts, delimited by the boundary equal to <boundary_name>.Connection: keep-alive
This one tells the client to keep the connection open as we’re expecting more data (subsequent frames) to come from the serverCache-Control: no-cache, no-store, max-age=0, must-revalidate
Expires: Thu, Jan 01 1970 00:00:00 GMT
Pragma: no-cache
The last three headers are responsible for cache control – in short, we want do disallow browser caching so that the resource loaded with our initial GET request always gets fetched from the server
Once the connection is established, the server starts providing subsequent frames:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
--<boundary_name> Content-Type: image/jpeg Content-Length: <frame_1_size> <frame_1_data> --<boundary_name> Content-Type: image/jpeg Content-Length: <frame_2_size> <frame_2_data> ... --<boundary_name> Content-Type: image/jpeg Content-Length: <frame_n_size> <frame_n_data> |
Each chunk (frame) starts with the boundary marker, Content-Type header, Content-Length header and the content itself. Chunks can also be extended with some additional headers, like frame dimensions or timestamps.
The streaming will continue until one of the sides closes the connection.
Something, that may have caught your eye is the fact that we don’t see any information about the video frame rate. Strange, isn’t it?
In fact it is one of MJPEG’s advantages – as the server keeps feeding the viewers with frames as soon as there are new ones, it can adapt the frame rate to e.g. low light conditions, where taking each frame may take more time.
Implementation
Let’s head back to our refactored script from the previous post. This time we only need to modify the server part as our camera code is smart enough to handle concurrent capture calls thanks to the Promises.
In the server script – capture.js
– we need to define the boundary to be used in MJPEG stream:
1 2 |
// MJPEG frame boundar const BOUNDARY = 'MYBOUNDARY'; |
Now, we need to update the callback passed to http.createServer
.
We’ll start with replacing static.jpg
with live.jpg
in our HTML template, so that the main page will serve our live stream:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
if (req.url === '/') { res.writeHead(200, { 'Content-Type': 'text/html;charset=utf-8' }); res.write(`<!doctype html> <html> <head> <title>Raspberry Pi Camera</title> <meta charset="utf-8"> </head> <body> <img src="live.jpg"> </body> </html>`); res.end(); } |
Next, we’ll create a new if
statement that will handle requests to /live.jpg
. Let’s have a look at the code first:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
if (req.url === '/live.jpg') { res.writeHead(200, { 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Connection': 'keep-alive', // here we define the boundary marker for browser to identify subsequent frames 'Content-Type': `multipart/x-mixed-replace;boundary="${BOUNDARY}"`, 'Expires': 'Thu, Jan 01 1970 00:00:00 GMT', 'Pragma': 'no-cache' }); // connection closed - finish the response res.on('close', () => res.end()); let loop = () => { return camera.capture() .then((frame) => { if (res.finished) { return; } // serve a single frame res.write(`--${BOUNDARY}\r\n`); res.write('Content-Type: image/jpeg\r\n'); res.write(`Content-Length: ${frame.length}\r\n`); res.write('\r\n'); res.write(Buffer(frame), 'binary'); res.write('\r\n'); // attempt to cache another frame after 50ms = ~20fps setTimeout(loop, 50); }) .catch((e) => { res.writeHead(500, 'Server Error'); res.write(e.message); res.end(); }); }; loop(); } |
We start with writing all the headers mentioned in the previous chapter:
1 2 3 4 5 6 7 8 |
res.writeHead(200, { 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Connection': 'keep-alive', // here we define the boundary marker for browser to identify subsequent frames 'Content-Type': `multipart/x-mixed-replace;boundary="${BOUNDARY}"`, 'Expires': 'Thu, Jan 01 1970 00:00:00 GMT', 'Pragma': 'no-cache' }); |
Notice we’re using an ES2015 template string to pass the boundary constant value to the content-type header.
Next, we bind a listener to response’s close
event. Once the connection closes, e.g. the client disconnects, we need to finish the response using the res.end()
method.
Then we define a loop()
function that captures the frame using our camera instance and responds with a frame once the capture promise fulfills. Before we do that, though, we check if the response hasn’t finished in the meantime. If so, then there is no point in proceeding further with the script. Finally, after 50 milliseconds, we execute the loop()
function once more to serve another frame.
In case when our promise gets rejected, we respond with 500 Server error and finish the response.
Finally, we execute the loop()
function for the first time to kicking off the streaming.
And here’s how it looks in the browser:
Summary
In this post we’ve learned:
- what MJPEG format is and how it works
- how to stream MJPEG video using Node.js
The complete code used in this post can be found in my Github repository.
In the next post we’re going to do something different – we’ll use Node.js to control a servo via Raspberry Pi’s GPIOs.
See you next time!
Hi! It’s good for me! Now I am doing a project with an IP camera, which has a server, i think it is similar with yours! I want to create a server that request to the server of the IP camera, the link like: http://192.168.0.103:8888/videostream.cgi?&loginuse=admin&loginpas=admin (a localhost of IP cam) and transfer this MJPEG to my client. Can you help me?
Some additional information about the request I received:
Response Header:
Accept-Ranges: bytes
Connection: close
Content-Type: multipart/x-mixed-replace;boundary=ipcam264
Date: Thu Apr 12 00:06:14 2018
Server: GoAhead-Webs
Request Header:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/
apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: vi,en-US;q=0.9,en;q=0.8,vi-VN;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Host: 192.168.0.103:9999
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Looking forward your response ^^! Thank you very much!!!