Tutorial: The Real Time API

Tutorial Real Time
Handling Real Time Events

With socket.io integrated in to our API we can turn our attention to handling events in real time.  Most socket.io demonstrations are aimed at a rudimentary chat application but in this tutorial we will solve another API problem — how to transfer large chunks of data.

Traditional API techniques calls for the developer to make use of the HTTP transport’s abilities through multipart uploads which are usually first converted to base64 thus increasing the payload size. Socket.io offers the API a powerful capability in that it can transport binary data natively.  A second challenge is that if during the data transfer the connection was interrupted the data would be permanently lost forcing the transaction to start again.

In this tutorial I will show how socket.io can be used to transfer binary data and resume from the last successfully received chunk in cases of a connection break.  Lastly instead of having the browser report what it thinks it has uploaded we will use socket.io to report to the browser after successfully receiving a chunk of data and writing it to the disk.

To understand the interaction between the client and the API I will be switching between files located in static/index.html (client) and sockets/connection.js (API).

Let’s take a look at the client side.  Socket.io can be included in to our HTML page by loading the JavaScript library and instantiating the functions.

<script src="/js/socket.io.js"></script>

On line 27 the websocket connection is established with the API.

var socket = io.connect('http://127.0.0.1:8086', {transports: ['websocket']});

The connection event is received by the API and is seen on line 45 of sockets/connection.js

io.sockets.on('connection', function (socket) {

The ‘socket’ object contains a wealth of useful information regarding the connection in much the same as the req object in RESTify contains information regarding the HTTP session.

To the ‘socket’ object we bind several event listeners that react on events received by the client.  It is a common notation in socket.io and is thus used in the same context and format between the client and the API.  Therefore the client reacts to events sent by the API and the API reacts to events sent by the client — true bidirectional events.

io.sockets.on('connection', function (socket) {
        socket.on('Start', function (data) { //data contains the variables that we passed through in the html file
                        var Name = data['Name'];
                        Files[Name] = {  //Create a new Entry in The Files Variable
                                FileSize : data['Size'],
                                Data     : "",
                                Downloaded : 0
                        }
                        var Place = 0;
                        try{
                                var Stat = fs.statSync('static/tmp/' +  Name);
                                if(Stat.isFile())
                                {
                                        Files[Name]['Downloaded'] = Stat.size;
                                        Place = Stat.size / 524288;
                                }
                        }
                        catch(er){} //It's a New File
                        fs.open("static/tmp/" + Name, 'a', 0755, function(err, fd){
                                if(err)
                                {
                                        console.log(err);
                                }
                                else
                                {
                                        Files[Name]['Handler'] = fd; //We store the file handler so we can write to it later
                                        socket.emit('MoreData', { 'Place' : Place, Percent : 0 });
                                }
                        });
        });

Lines 45 through 74 of sockets/connection.js of the API shows the handling of events from the client in context with the io object.  On line 46 we see that we are binding to the ‘Start’ event emitted by the client and retrieve our information from the supplied data object.

On the client side we can understand how these events are signaled to the API.

                        function StartUpload(){
                                if(document.getElementById('FileBox').value != "")
                                {
                                        FReader = new FileReader();
                                        Name = document.getElementById('NameBox').value;
                                        var Content = "<span id='NameArea'>Uploading " + SelectedFile.name + " as " + Name + "</span>";
                                        Content += '<div id="ProgressContainer"><div id="ProgressBar"></div></div><span id="percent">50%</span>';
                                        Content += "<span id='Uploaded'> - <span id='MB'>0</span>/" + Math.round(SelectedFile.size / 1048576) + "MB</span>";
                                        document.getElementById('UploadArea').innerHTML = Content;
                                        FReader.onload = function(evnt){
                                                socket.emit('Upload', { 'Name' : Name, Data : evnt.target.result });
                                        }
                                        socket.emit('Start', { 'Name' : Name, 'Size' : SelectedFile.size });
                                }
                                else
                                {
                                        alert("Please Select A File");
                                }
                        }

Lines 57 through 75 of static/index.html shows the ‘StartUpload’ function that is bound to the onClick event of the upload button.  Other than some basic functions to create and establish the file reader the essential trigger lies on line 69 showing that the socket.io client ’emits’ an event to the API.

On the API side the API signals the client when it wants more data and is done after the data has been streamed to the disk.  Lines 75 through 110 of sockets/connection.js show the ‘Upload’ event.

       socket.on('Upload', function (data){
                        var Name = data['Name'];
                        Files[Name]['Downloaded'] += data['Data'].length;
                        Files[Name]['Data'] += data['Data'];
                        if(Files[Name]['Downloaded'] == Files[Name]['FileSize']) //If File is Fully Uploaded
                        {
                                fs.write(Files[Name]['Handler'], Files[Name]['Data'], null, 'Binary', function(err, Writen){
                                        var inp = fs.createReadStream("static/tmp/" + Name);
                                        var options = {ffmpegPath:'./ffmpeg'};
                                                waveform.getWaveForm( __dirname + '/../static/tmp/' + Name, options, function(error, peaks){
                                                  if(error){
                                                    console.log(error);
                                                  }
                                                  
                                                  // Emit Peaks
                                                  socket.emit('Done', {'peaks' : peaks, 'file': Name});
                                                });
                                         });
                                });
                        }
                        else if(Files[Name]['Data'].length > 10485760){ //If the Data Buffer reaches 10MB
                                fs.write(Files[Name]['Handler'], Files[Name]['Data'], null, 'Binary', function(err, Writen){
                                        Files[Name]['Data'] = ""; //Reset The Buffer
                                        var Place = Files[Name]['Downloaded'] / 524288;
                                        var Percent = (Files[Name]['Downloaded'] / Files[Name]['FileSize']) * 100;
                                        socket.emit('MoreData', { 'Place' : Place, 'Percent' :  Percent});
                                });
                        }
                        else
                        {
                                var Place = Files[Name]['Downloaded'] / 524288;
                                var Percent = (Files[Name]['Downloaded'] / Files[Name]['FileSize']) * 100;
                                socket.emit('MoreData', { 'Place' : Place, 'Percent' :  Percent});
                        }
                });

Pay special attention to lines 101 and 108 which shows how the API emits an event to the client to request more data and also supplies the progress indication.

Media Analyzer On the client side the progress received from the API is drawn in real time on a progress bar indicator as well as the actual percentage and data chunk progress.

Lines 83 – 94 of sockets/connection.js contains interesting logic executed when the file has been received and verified to be complete.  A special library makes use of the ‘ffmpeg’ binary to plot a series of peak values which are transmitted to the client on line 91 as a JSON array.

Media Analyzer Once the event is received by the client the peaks are rendered via the wavesurfer.js library and displayed on the page.

Websockets give an API a powerful mechanism of real time bidirectional communications and open the possibilities to the next level of service provision and client interactivity.

Feel free to explore the code in the repository and try it our or modify it for yourself.

Was This Helpful?
All of the content on this site is presented without advertiser support and is produced exclusively by me. If you find any of this information useful please consider it against the cost of a course or book.

I gladly accept donations of any amount which goes directly towards producing more quality content and videos for this site.

[wpedon id=119]

Pages: 1 2 3 4

Written by YourAPIExpert