WhatIs ReaItime? Since the explosion 0f the Web, developers have been inclined t0 think in terms 0f building websites. Even in this b00k, l've spent a good deal 0f time writing about building websites. But make no mistake about it, a realtime user experience does not exist entirely inside a web browser. The original web browsers were designed t0 load and display web pages. The idea 0f a web page iS quite similar tO the printed page. lt iS a static document' stored in a computer, but a static document nonetheless. The interface Of web browsers evolved to work within this paradigm. There is a Next button and a Back button designed t0 take you from the current page t0 the page that you viewed previously. That makes perfect sense when you re working with documents. However, the Web is quite quickly shifting away from a document-based paradigm t0 a web-based form 0f commumcatlon. lt used t0 be that a web page was more or less published t0 the Web. A page was created and given a specific URI, and when the user wentto that page, it was pretty clear what t0 expect. Now we have sites like Faceb00k ( 中 : 〃ルルル扣記わ 00 たェ 0 川 ) where the same URL is not only different for each of the hundreds of millions of different users, but it changes moments after a user loads the page. ChangingInteractions ln the past, the interaction between a user and a web application was very simple. When a user wanted content, she would load up her browser, point it at a URL, and get the content (see Figure 1 ー 1 ). If she wanted t0 write a blog post, she'd load up her browser, fill out a form, and press submit. When she wanted [ 0 see comments, it was much the same. This has changed. NO longer can a website wait for users t0 navigate t0 the right URL; the website must contact the user wherever that user may be. The paradigm has shifted from a website-centric 1 れ Odel , where the website was at the center Of the interaction, tO a user-centric model. NOW all interactions start and end at the user (see Fig- ure 1-2 ) , whether she is visiting the website or sending in Short Message Service (SMS) updates. A truly realtime experience exiStS anywhere the user iS at a given moment. If the user is interacting with the web browser, then that's the place tO contact her. If she's got her instant messenger program open, she'd better be able tO interact with your app from that window. ・ when she's Offline and your application has an important message for her, send it via SMS. Naturally, you'll need t0 ask the user's permission before you d0 some Of these things, but your application needs tO Offer them. 2 ー Chapter 1 : lntroduction

ln this function, we use the base dOjO framework tO create some DOM elements tO display the posted content. After creating a couple of P elements and a STRONG tag tO shOW the author name, we use dOjO animation tO make the HTML fade in. lt's just an extensible way 0f printing the HTML contentto the page. Since this file gets included by both ⅵ ver. html and river-post. html , this function may execute two different actions depending on which page is loaded. When a user is 100k - ing at the ⅵ ver-post. html file, we'll be able t0 access the "POSt" form button via Java- Script. We simply connect that button's onclick event tO the submitPost function we created earlier. When we don't have that form element, we assume that we re going tO view the liveblog feed and subscribe to the /river/flow channel. FinaIIy, we need to add the code that gets this whole process up and running. Add the following code t0 the b0 は om of your ⅵて . js file: google. set0nLoadCa11back( function() { d0j0. require("dojox. cometd"); d0j0. add0nLoad(setupRiver); d0j0. add0nUn10ad(dojox. cometd, "disconnect"); Because we're loading the dOjO files frOI れ remote servers, we need [ 0 wait tO run our setup functions until all of the code has been loaded. Luckily, the Google Ajax libraries API provides the callback function google. set0n LoadCa11back t0 let us know when that has happened. ln that callback function, we tell D 可 0 which features we're going [ 0 need, which in this case is only the cometd extension. Then, we instruct d 可 0 [ 0 call our setupRiver function when it iS ready tO continue. The very last step is tO instruct the cometd library t0 disconnect when we unload ()r navigate away from) this page ・ At this point we have all the code in place for a fully functionalliveblog, so I [ it out. Running the app is simple. From a terminal window, navigate t0 the 叩 ps directory and enter the following command: ~ cometd-java/apps $ mvn jetty:run Now the server should be up and running. To fully test this out, you'll wantto open one browser, say Safari or lnternet Explorer, and point it to 印 : 〃 127.0.0.1 : 8080 〃ⅳ . 川 I. Then, open another browser, say Firefox, and point it [ 0 中 : 〃 127.0.0.1 : 8080 / r ⅳげ甲 0 立 . 川 l. Figure 4-4 shows a liveblog session in action (see Figure 4-4 ). Once you've loaded up the browsers, you should see the two web pages we created. As you start posting content, you should notice hOW quickly the content shOWS up in the Other browser window. Keep in mind that this involves a trip through the server, and it's not just drawing content from one window tO the Other dynamically; all Of the content is posted t0 the Web and loaded back in the browser in realtime. If you opened up another browser, say Opera or Chrome, and pointed it at 中 : 〃 127.0.0.1 : 8080 〃ⅳげ . わ [ 川な you'd see the content refresh in tWO browsers just as quickly as the one. A ReaItime Live 引 og ー 69

The Two-Connection Limit There iS big a reason you need tO open these files up in separate browsers and not Just different tabs in the same browser when testing. 、åOSt modern browsers limit the amount Of concurrent connections tO Jt1St tWO connections per server. ThiS means that if you have tWO connections open tO the server and try [ 0 open another connection in the same browser, even if it's in a new tab, the browser will wait until one Of the original connections disconnects. When we're dOing long polling, that means we'll be waiting a long time between connections. Cometd actually helps with this issue by using the advice part of the protocolto instruct the clientto fall back to regular polling atstandard intervals. Although this helps keep connections alive, it means we re not getting truly realtime content because Of the length Of time between requests. ln practice for normal users, this isn't as much Of an issue, but when building sites, it poses a bit Of a problem. The solution is simple: use totally different browsers. You can easily check to see if the cometd libraries have fallen back to standard polling at long intervals by examining the transfers with the Firefox extension Firebug ( : 〃 ge 尾わ g. co 川 ). Firebug has many features that make debugging web applications much easier, such as the ability tO examine the network activity, including connections that are still active. When you load up and enable Firebug, the console tab will show you the different POST requests currently active (see Figure 4-7 ). If one of them is constantly active, long polling is working. If the connection returns immediately, it's fallen back. To fix this, navigate away from the page for a minute; it should go right back [ 0 long polling when you return. POST h p : 〃朝 me 」 0 ( host な om 秋 d ト POST h p : / / 爬 a me 」 0 ( host / ( om d POST http://realtime.localhost.com/td POST h p : / / 爬朝 me 」 0 ( host な om d ト POST h れ p : 〃代 altime 」 0 ( ho 靆な om d ト POST http://realtime.localhost.com/td ト POST h p : 〃爬ⅱ me 」 0 ( host / ( om に td POST h れ p : 〃代朝 me 」 0 ( host な om 秋 d ト POST h p : / なに altime 」 0 ( host な om に td 200 OK 6m5 200 OK 7m5 2000K 2m 05 2000K 2m 05 2000K 2m05 2000K 2m 05 2000K 2m 05 ~ 000K 2m 05 dojoxd.js 0 ⅲ・ 16 ) dojoxd.js 0 ⅲに 16 ) dojo.xd.js 朝 16 ) dojoxd.js 価 n に 16 ) dojo.xd.js ()i 飛 16 ) dojo.xd.js (line 16 ) d 可 0 ょ djs ()i 16 ) dojo.xd.js (line 16 ) d 可 0 ょ d.js 0 ⅲに 16 ) ん g Ⅲ℃ 4-7. 尾わ″ g dut ・ ing 4 long 〃 0 ″ⅲ g s お s れ 72 ー Chapter4: River ofContent incomplete, for all intents and purposes, everything is ready [ 0 go. will keep waiting until it does. Although the status bar may claim the page load is still loading an asset. lt is! lt's waiting on your long polling operation [ 0 finish, and it status bar say something [ 0 the effect Of "completed 5 Of 6 items," as if the browser is WhiIe the browser is looking at a page in a long polling section, you may notice the

嶬で b Design & Development Building the Realtime User Experience The 嶬で b is increasingly happening in realtime. 嶬宝 h websites such as Facebook and Twitter leading the way' users ℃ coming tO expect that Ⅱ sites should serve content as it occurs¯on smartphones as well as computers. This bOOk shows you h ( ) 从 to build realtime user experiences by adding chat' streaming content' and including at 紅 without making big changes tO the existing infrastructure. You'll alSO learn hOW tO serve realtime content beyond the browser. Throughout the b00k are many practical JavaScript and Python examples that you can use on your site 〃 0 . And in the final chapter, you'll build a location-aware game that combines all of the technologies discussed. Use the latest realtime syndication technologY' including PubSubHubbub BuiId dynamicwidgets on your homepageto show realtime updates from sources ■ Learn h0Wt0 use long polling t0 push" contentfrom your server tO browsers create an application using theTornado web serverthat makes sense Of massive amounts Of streaming content Understand the unique requirements for setting up a basic Chat service Use IM and SMS tO enable usersto interactwith your site outside Ofa web browser lmplementcustom analyticsto measure engagement in realtime ■ ■ 第 ■ Previous experience developing web iS recommended. 。 7 〃な 0 〃召グ〃 川尸阨 / ん〃 0 〃召 〃〃肥わ . 爿叩ド C() 〃 7 わ加 es 4 〃 S 2 / 72 / 〃 鰓〃 / ⅳ〃 . 鹿召リ功 / 〃 g 花ノ es な / 加″ g カ川を昭 / 0 励 0 間″側励 c 砠ツ / 娵ル召 ア / / / ツ翫励 / 加雇 —MarshaIl Kirkpatrick 朝一加ら ReadWriteWeb Ted Roden is a member 0f the Research and Development group at 7 カ e Ne 肥 7 カ〃い , wher ℃ he has extensively 1 ℃ searched topics related tO realtime user experience. 0 REILLY 0 0 「 e Ⅲ y. ( om US $ 34.99 CAN $ 43.99 I S B N : 9 7 8 ー 0 ー 5 9 6 ー 8 0 6 1 5 ー 6 5 3 4 9 9 川ⅧⅢはⅢⅧ馴 II 9 / 8 0 5 9 6 8 0 6 1 5 6 Free online edition fo 「 45 days with purchase of this bOOk. Details on last page. Safarl BooksOnline

I mentioned SMS, but the mobile experience does not end there. These days, users have phones in their pockets with full-fledged web browsers that in some cases offer more functionality than their desktop-based brethren. Among other things, mobile browsers can handle Offline data storage, GPS sensors, and touch-based interfaces. Their im- pressive featureset, coupled with nearly ubiquitous wireless broadband, means they cannot be treated as second class lnternet citizens. AppIications and user experiences simply must be built with mobile devices in mind. Push Versus PuII For about as long as the Web has been around, there have been tWO main ways Of getting content tO a user: 〃レ s 〃 and 〃″ . Pull is the methOd in which most interactions have worked—the user clicks a link and the browser pulls the content down from the server. If the server wants [ 0 send additional messages tO the user after the data has been pulled down, it JIlSt waits and queues them up until the client makes another request. The idea behind push technology iS that as soon as the server has a new message for the user, it sends it tO him immediately. A connection is maintained between the server and the client and new data iS sent as needed. ln the scheme of the lnternet, push technology is not a new development. Throughout the years there have been different standards dictating how it should work. Each pro- posed standard has had varying levels ofsupport amongst browser makers and different requirements on the server Side. The differing behaviors and requirements of the two technologies have led many de- velopers tO use one or the Other. This has meant that many sites wanting tO Offer dy- namic updates tO their users had tO resort tO AJax timers polling the site everyX seconds tO check for new content. ThiS increased amount Of requests is taxing on the server and provides a far less graceful user experience than it should have. pushing content out tO the user as it happens gives the user a much more engaging experience and uses far less resources on the server. Fewer requests means less band- width is used and less CPU consumed, because the server is not constantly checking and responding to update requests (see Figure 1-3 ). Prerequisites This bOOk assumes the reader is comfortable with most aspects Of web development. The example code in this text uses Java, JavaScript, PHP, and Python. You are encour- aged [ 0 use the technologies that make the most sense tO you. If you're more comfort- able with PostgreSQL than MySQL, please use what you're more familiar with. Many of the command-line examples assume you re using something Unix-like (Mac OS X, Linux, etc. ) , but most Of this software runs on Windows, and l'm confident that you can translate any commands listed SO that they work in your environment. 4 ー Ch 叩 ter 1 : lntroduction

Adding this instant messaging functionality—、 vhich opens an instantaneous C01 れ 1 れ u ー nication channel with a user, regardless Of whether they're using your site in the web browser—opens up a 10t Of doors. Part Of building a truly realtime experience iS allowing users tO get data in and take it out from whatever method iS most readily available. lnstant messaglng provides a paradigm that is familiar tO most users and allOWS for communication in places not available via the standard web browser. Setting Upan API ー 151

Twitter updates, from firehose tO web browser, 91 ー 100 web framework and nonblocking web server, 79 web server, 88 tornado. web. Application function, 83 tornado. web. Application module, 104 tornado. web. RequestHandler module, 104 touch-based interfaces, 4 Tweet class (example), 85 TweetFirehose class (Tornad0 example), 88 TweetProcessor class (Tornad0 example), 89 Twitpic, thumbnail URL, 53 Twitter account for this book, 7 realtime application, asynchronous Tornado, 84 ー 87 realtime streaming APls, 87 ー 91 realtime view Of images streaming intO, 54 trending widget (example), 39 ー 48 TypingHandler class (chat example) , 123 51 ー Unix-like operating systems, sending SMS via command line, 154 updated—time field (SUP file), 12 UpdateHandler class (example) chat application, 111 , 115 Tornad0 application, 86 updates field (SUP file), 13 , 19 urlfetch module (Google) , 147 urllib2, 89 URLs generating for SUP feeds, 26 image service, 53 routing for Tornad0 requests, 83 SUP feeds, 17 thumbnails for image services, 53 topic URL for PubSubHubbub feed, 33 user involvement, measuring (see analytics on realtime 嶬 b ) user-centric 1 れ Odel , 2 UserInf0 class (example), 222 ー 224 UserThreats class (example), 232 ー 234 uuid (Python), 108 validators for Atom/RSS feeds and SUP headers, 26 FriendFeed validator, 25 weather command, 148 weather information via instant messaging, 146 ー 149 web application development, 4 ℃ browsers lndex command for weather information, 147 checking for authenticated users, 145 XMPPHandler class, 139 xmpp module, 140 user identifier in sender field, 145 264 receiving messages in iPandemic game, 263 ー receiving instant messages via, 139 (example) , 223 allowing for users of iPandemic game XMPP, 129 X-SUP-ID headers, 17 , 25 X-Hub-Subscribers headers, 29 Woopra (analytics service), 187 messages, 154 wireless carriers, email address for SMS setting up JavaScript, 41 ー 48 HTML, 40 trending on Twitter (example), 39 ー 48 live images streaming intO Twitter, 51 ー 54 FriendFeed in realtime, 48 ー 51 widgets, 39 ー 56 website-centric model, 2 website for this book, xi, 7 Weblogs.com Pings, IO applications, 64 parameters Of interest [ 0 cometd creating, 63 ー 65 section, 75 adding filters parameter tO <servlet> web. xml file serving, 56 Web, move from document-serving [ 0 message- two-connection limit, 72 realtime updates ⅲ live blog feed, 69

contain the map Object, SO that it can be easily referenced in the different member functlons. The first of those member functions is getLocation, which is a wrapper call for the browsers, own navigator. geolocation. getCurrentPosition meth0d. This method is available on most recent 1 れ Obile browsers and iS even available desktop browsers with increasing frequency. Firefox on the desktop already includes this functionality natively within the browser. Rather than checking tO see whether each browser sup- ports this function every time we want tO call it, we simply call this ipGeo. get Loca tion method, which wraps up that logic and makes the proper success and failure callbacks depending on what happens. Because locating the exact GPS coordinates 0f the browser can take some time, navigator. geolocation. getCurrentPosition runs asynchronously and executes the proper callback when data is available. Aside from the success and fail callback functions, we also pass along a small object with a single key/value pair for maximumAge. Making a getCurrentLocation request is expensive, bOth in terms Of the amount Of time it takes tO return and the drain on batteries it takes tO use a GPS sensor in a mobile device. Setting the maximumAge param- eter tO 3()0 OOO milliseconds allOWS us [ 0 cache the results for five minutes. This amount Of cache time will ensure that any geolocation-based action that takes place in a stand- ard session will require only one actual external request. This alSO stops the result from being cached indefinitely, which is actually the default behavior on some devices. The next method is ipGeo. initialize, which takes in a parameter that is just the ID Of the DIV we're going to use to display the map. This method then checks t0 see whether Google thinks this browser is compatible with their maps API and creates the map object. There are a number of different options ( 印 : / ん 0 . goog . し 0 川ん〃な / a 内 / doc 川 e 〃 0 〃 / 尾を〃記 . 川 / # GMa 〃〇〃行 0 ) that can be used when working with these maps, but we're just going t0 use the default options and tellthe map t0 use the default user interface. Next, we simply call the get Location methOd and, upon success, we center the map on the coordinates it returned and add a marker at the current location. NOW that we've built S01 れ e basic JavaScript functionality intO this application, we need to add it t0 the HTML views. 从々 don't need t0 add this t0 the logged-out page,Just the page where the user has already been authenticated. Open up 川〃 s / 〃ⅲ . 川 / and make the following adjustments: く tit1e>iPandemic く /title> - f0 て jQuery - く script type="text/javascript" src="http://www ・ google ・ com/jsapi"> く /script> - for the maps API - く script src="http://maps ・ google ・ com/maps?file=api&v=2&sensor=true&key= YOUR-GOOGLE-MAPS-API-KEY" type="text/javascript"> く /script> 242 ー ChapterIO: Putting lt 則 Together

module, re, has a method called findall. This method returns an array ofany matches found in the string we provide. This way, on the client side we can just check the size Of the array tO determine hOW many matches we've made, if any. Next, we build the message object that we'll be sending ou い 0 the browser. This object contains a unique field called id, which we'll be using as our cursor. lt alSO contains basic information such as the tweet itselfand WhO sent it. The StatS Object is alSO passed through. TO save time, we even build the HTML string that will be displayed by the browser. When the browser gets the data, it can simply display this html value to the client. TO send it [ 0 the browser, we callthe Tweet ( ). new tweets method that we built earlier. As you'll remember, that method loops through all of the connected clients and outputs the Python object encoded as a JSON string. The whole workflow of this method is pictured in Figure 5 ー 5. This processing workflow ends up calling the new tweets method. As a quick refresher, that method then 100PS through the list of connected clients to run the callback method associated with each request. That callback method then outputs the data in JSON format and finishes the connection. The server side of this script is totally complete, though if you run the server from the command line, you won't see much Of a change. At this point we're accepting web requests, collecting tons Of data from Twitter, parsing it up, and sending it back tO any client who cares t0 connect. There's a big missing piece, though: the web page. So let's build that next. From the Firehose to the Web Browser We're already collecting the data we need, and Tornado is ready to respond to requests for updates. Basically, all we have left to do is build the web page and write the JavaScript. TempIates in Tornad0 The template language in Tornado is based on the templates in Django ( ん印 : 〃ルルル 4 〃 go 〃 ro. は工 0 川 . B0th 0f these projects contain rich templating languages where standard HTML files get parsed based on logic written in a subset of Python. However, a big difference between the templating language that inspired Tornado and Tornado itself is the ability to include arbitrary Python code. ln Django, you're restricted to variables, filters, and standard flOW control statements. TO go beyond the 1 れ OSt basic logic, you must create new template filters and run those. With Tornado, you're al- lowed tO use any valid Python expression, including functions and list comprehensions. As long as it's a valid Python expression, it will work in these templates. From the Firehose t0 the Web B 「 ow 史「一 91

Customized AnaIytics The paid realtime analytics services are useful tO get an overview Of what is happening on a website in realtime. However, the applications that we've been building SO far have not been limited tO the web browser. ln order tO properly figure out the current status Of our applications, we're going tO need [ 0 build a custom solution that can handle tracking the different types Of actions our users take. ln this section we'll build a simple analytics application that can track users as they navigate a website, but alSO tracks any Other arbitrary action that happens outside Of a web browser. Sending Tracking Pings with JavaScript For the web-browser-based part Of collecting these analytics, we re going [ 0 use a simple JavaScript-based implementation. The JavaScript file that we build can be hosted either on the same machine as the main web application or somewhere else entirely. Once this JavaScript is ready, we'll include it from each page of the site and run a function tO start tracking user actions. TO get started, create a file called rea ~ 襯 e - a れ a いい . and insert the following code: var Ana1ytics uid: 0 , start time: 0 , last activity: 0 , " { { ping-url } } " ping url: start: function() { Ana1ytics. start time = new Date() . getTime(); Analytics. last activity = Analytics. start time; / / ensure we have a user lD Ana1ytics. setupUID(); / / setup the event listeners var updateLastActivity = function() { Ana1ytics. last activity = new Date(). getTime(); window. addEventListener('click ・ window. addEventListener(' s ( r011 ・ WindOW. addEventListener('mousemove ・ window. addEventListener(' keypress ・ / / when we' re unloading the page = function() { var beforeUn10ad updateLastActivity, false); updateLastActivity, false); , updateLastActivity, false) ; updateLastActivity, false); / / ping again, but note that we' re exiting Ana1ytics. ping('exit ・ ) ; / / ensure that we get one last ping in before we unload the page window. addEventListener(' beforeunload ・ beforeUnIoad, false); Customized AnaIytics ー 189