The Command Center The application that we've built at this point runs on Google's App Engine platform and exists almost entirely on the mobile phone. lt has realtime components that allow the application tO reach users when they're not currently lOOking at the site, and even via SMS, when they're not actively doing anything on the phone or the computer. However, now we that we have realtime access tO the germs as they are built and destroyed, we can build a full-featured front-end [ 0 view everything while not actively participating in the game. This part of the game will be built using Tornado so that we can take advantage of its support for long polling. This will also require us to run the game from a separate server, which will enable us to use PubSubHubbub in the same way that any other subscriber would use it. The basic site TO get started, let's create a file in the ゆ a 〃川ん 0 川 ado folder called げ . . start if off with the following code: import os, logging import tornado. httpserver import tornado. i0100P import tornado. web import tornado. auth import simplejson as j son import time, re, uuid from tornado. options import define, options from tornado. escape import url escape import urllib, ur11ib2 from xml. dom import minidom # Define options that can be changed as we run this via the command line define("port" defauIt=8088, heIp="Run server on a specific port", type=int) # a basic controller to load the default template class HomeHand1er(tornado. web. RequestHand1er) : def get(self) : # see if we can load the user infO from a cookie self. get secure cookie(' user ・ ) user_json if user_json: user = json. loads(user_json) user = None self. render("command-center. html" class App1ication(tornad0. web. App1ication) : def init_(self) : handlers else: user=user) Building the Game ltself ー 275

statuses/retweet The retweet feed returns all of the updates that contains all of the retweets on the public timeline. Like several Other methods, access tO this one alSO iS restricted and requires special access. statuses/sample This is a public feed that contains a small sample of the main firehose feed. De- velopers whO are granted higher access levels can get more data out Of this feed making it a more statistically significant sample and appropriate for data mining and research applications. For this application, we'll be using the statuses/sample streaming API method. AI- though this is just a small (and therefore statistically insignificant) subset of the main firehose feed, it's a lot of data, and certainly enough for our purposes. lt also requires no special permission from Twitter, aside from a standard user account. This example will require an active TWitter account. If you don't have an account, head over tO the Twitter website ( わな〃 : 〃ール r. CO 川 ) tO sign 4 、、 up. Once you've done that, feel free t0 follow the Twitter account of this very bOOk at @therealtimebook. The Tornad0 web server runs as a single process on a single thread. lts nonblocking architecture ensures that it can quickly respond tO certain requests even while many Other connections are open and waiting. However, even though Tornado runs in a single thread, we're going tO need tO create some Other threads tO handle the work from Twitter. The Twitter streaming API feeds will essentially stay open forever, as long as we can keep up with them. TO ensure that we dO keep up, we re going tO separate the work intO separate threads. The first thread just catches the data as it arrives and queues it up. Any processing work tO be done on the tweets should be in a separate thread, tO ensure that we're able tO keep up with Twitter. Our example doesn't dO any compli- cated processing Of the data, but l've separated the work out anyway, tO ensure we can easily modify it in the future. ln its simplest form, creating new threads in Python iS as simple as creating a class and defining a run method. When you ask Python to startthe thread, it will automatically callthat method and you're on your way. Python has a bit of a reputation for poor handling of multithreaded applications. However, the threaded aspects Of our application aren't very complicated, so it should be fine. The first thread we're going tO create iS the thread that handles the communication with Twitter. ln your nne て . py file, add the following code after the UpdateHand1er class.• 88 ー Chapter 5 : Taming the Firehose with Tornado

g ア川〃 . / 山 ) t0 generate a new API key for this application. When the form asks for your URL, enter the App Engine URL created earlier. The next page will display your unique Google Maps API key; be sure t0 COPY that URL and save it locally on your computer. we'll be using that key ⅲ this application, and there is no obvious way t0 get Google Maps t0 display the same key for you again ・ EC2 0 「 0ther Hosted Server part 0f this application runs on Google's App Engine infrastructure' but another part is hOSted on a standard web server. ThiS server needs [ 0 have python installed and you must have shell access, as we'll be running this application with the Tornado which must be started from the command line. Any public-facing server will work for this task as long as you have shell access and python is either installed or installable. If you re using this server tO hOSt another application, you can easily hOSt this right alongside it using a different port. If you're using Amazon EC2 t0 hostthe part 0f the application that does not run on App Engine, you're going t0 want t0 open a new port for this application. TO d0 that from your EC2 instance, you'll wantto open the port with the following command: $ ec2-authorize default -C ~ / ノ 0 イ t - le. pem -K *. pem \ -p 8088 Many of the community AMls (Amazon Machine lnstance) already have the ec2- authorize command installed. If you don't have it installed already, your package manager may have it under ec2-api-tools; otherwise, it can be downloaded directly from the Amazon 从 b services developer website ( わな〃 : 〃 de リ elO 〃夜・ . a 川 a て 0 〃ル e わ覊なお . co 川ん 0 〃〃厩リ〃川 a ″ D = 357 c ego リ ID = 88 ). GeoModel This application deals with geolocation in a number 0f different ways. As users cough and germs spread, going tO need tO determine hOW close one germ iS tO another. There are a couple Of ways tO dO this, but a convenient methOd would be tO ask our datastore [ 0 return a list Of germs within a specified distance Of a certain germ. While App Engine's data store does understand the concept 0f latitude and longitude coor- dinates through its Geopt datatype, it has no built-in ability tO return a list based on proximity. TO get this functionality, we can use an open source third-party library called GeoModeI ( 印 : / ん 0 . goog . co 川ゆ / geo 襯 ode の . The easiest way [ 0 incorporate the source iS tO check out the latest version from Subversion directly intO our directory. $ svn ( 0 http://geomodel.googlecode.com/svn/trunk/geo geo/geocell test. py A geo/geocell. py A geo/geomodel. py A 220 ー Ch 叩 ter 10 : Putting lt 則 Together

iPandemic Con セて 01 Center S.ay 50m2 び気 00 ~ ・一 1 を ' 3 90 ミ 0 セ 0 be は 10 れ 9 ni h セ in manhattan. 鵞の 30 erai 90 ー ms are & bO セ セ 0 90 セセ . 製・一 1 : 工 wish 1 sa 響セ h & を一 Poor Ma れ ha しセ日 100k3 、、ご芋・・ミ Ck. Nei1 joined the chat 、当↑・臧 : Xyie, 工 ju 島セ wa セ 0h04 第 your 90 て m die ・ Satellite , Fiatiton 1 け 下 e Ne 第、 e 引 V ⅱは 0 総 / を イ P_S ⅸ Cgty SQ いれ、 こ : QV を Park S 曲績い S を碑 ) 0 聳材いま ー曰會日な、卩 - ーⅱ V 第 ! もを . ニ。 Figure 10-15. Viewing ビ co 川〃に d co 〃な司記厩夜・ Adding a chat based on long polling can be done from almost any server setup; if it's not directly supported, firing up a simple Tornado web server on a separate port will certainly dO the trick. The SMS integration on this application is a small extra feature— the application would certainly work without it—but having it there enhances the user's experience. Likewise for lnstant Messaging, this addition can easily be done by firing up an App Engine instance and responding to a few API calls. Having these technologies ⅲ your application is pretty easy—small steps, but the end result is that users can interact with your application in drastically different ways. AS the web increasingly moves away from the desktop computer, it'S important that your application can respond in every way your users expect. Depending on your applica- tion, there is a good chance that users would start interacting with your application more frequently, and hang around longer, if you gave them the means to do it on their own terms. Ge 謝 S 第部 JO 爲 Qn 、 に 9 卩・ Elmer m を第 ′ Bo し 0 00502 0m0 区Ⅲ 5 0 し をを stR を朝 East 望亜 Park 1W0 Bridges 。 ' ' ふ怺し第 - Ⅲ技新朝い 478 296 ー Chapter 10 : Putting lt 則 Together

as long as the ap 〃 . ッ a 襯 i に is updated as well. Applications are not limited to one Python file; this is justthe single file created by the launcher application. The application created by thelauncher program is actually complete and ready to run. From the launcher interface, click the button called "Run," which will start up the development server on your local machine. Assuming your project was also set up to run on port 8080, you can point your browser at 印 : / 刀 oca 0 止 8080 to see something like Figure 7 ー 4. ⅸ tp : / / lo ( a 旧 05t : 8080 / 、 - 直ー 1 十 6 h 【 tp = / / loca 旧 0 , t = 8080 / H 0 world! 日 g ″尾 7-4. "HelIo WorId" ル卍 10 乢ん〃 g a line 可 co Taking Advantage 0fGoog 厄 Q• Goog 厄 Writing applications on App Engine gives the developer access to Google's web-serving infrastructure as well as some Other useful services. One Of the most immediately useful features is the ability tO get some user information for free through Google Accounts. lt takes only one line of code to force a user to log in, and knowing they're logged in, you can easily make use Of some Of their user information. The Python part of a Google App Engine application, as specified ⅲ川 a ⅲ . , is very similar tO the way Tornado works. ln main ( ) , the actual application object is created by defining which classes respond to which URL requests. If a request comes into a specified variable via an HTTP get request, the get method of the class is called; the same logic applies for post requests. Starting with the shell of the application created by the launcher application, let's modify it slightly [ 0 force users to 10g in and prepare for the instant messaging part of our application. When you open the 川ⅲ〃 . file, it should look like the following code below. Expand it by adding the highlighted: Getting Started with 600g 厄 A 叩 Engine

N ew messages? MyWeb ( 弱 myPostHandIerFu nction() Tornad0 Async Code Waiting … calIbackResponse Function() N0tify via callback New messages g レ尾 5-2. TO 川 4 hanging 尾甲イ es なⅵ 4 ca ″黻ルれ 0 〃 s BuiIding an AppIication SO before we can build any realtime features with Tornado, we need tO put together an app that responds t0 basic requests in using this framework. Create a file called runner ・ and add the following code. import logging import tornado. httpserver import tornado. i0100P import tornado. web from tornado. options import define, options import OS # Define options that can be changed as we run this via the command line define("port", defau1t=8888, he1p="Run server on a specific port", type=int) class MainHand1er(tornado. web. RequestHand1er) : def get(self) : logging. inf0("Request t0 MainHandler! " ) self. write("Great, now let's make this app speak in realtime. " ) 10Ca1 static path = OS. path. join(os. path. dirname( file ) , application = tornado. web. App1ication([ (r"/", MainHand1er), ] , static_path=local static path) "static") name ma1n http server = tornado. httpserver.HTTPServer(appIication) tornado. options. parse command line() http server. listen(options. port) tornado. i0100P. IOLOOP. instance(). start() Like all Python scripts, this script first imports the libraries it needs tO run and does a bit of housekeeping. ln this case we define the port on which this server should run. Using Tornad0's options library, we set it t0 port 8888 , but that can be changed at 82 ー Chapter 5 : Taming the Firehose with Tornad0

SUP The Simple Update Protocol (SUP) is a simple and compact poll-based protocol that can be used tO monitor thousands Of feeds in one ShOt. lt's not a push format like S01 れ e Of the Others, but it can save countless amounts Of server resources by eliminating the need for frequent polling t0 many separate feeds, and it allows for much quicker updates 0f the new data. This protocol was developed by FriendFeed and is supported by a number 0f sites around the Web, including YouTube. SUP is remarkably easy to implement for b0 市 subscribers and publishers. The biggest downside t0 this protocol is thatit's still based on polling. SO it's not strictly real- time, but it's darn close. P ″わ立わ H レ励″わ PubSubHubbub is a publish/subscribe protocol based on web hooks or callbacks. This protocol describes an entirely push-based system designed by a group of de- velopers at Google. lt is a totally open standard with a decentralized and open method of providing updates. When new content is posted, the publisher notifies a hub, which then sends out the new updates to each of the subscribers. Subscribers don't have to ping for new content, and the hub sends only the differences in the feed each time, significantly cutting down on the bandwidth transfer after each update. lt's a fairly easy protocol and can be added intO most existing systems without much effort. The most complicated parts, by design, are contained within the hub. rssCloud This protocol was actually developed as part of the RSS 2. O specification. lt works very similar t0 the PubSubHubbub protocol with very slightly different implemen- tation details. The cloud part of rssCloud, which is very much like the hub from PubSubHubbub, receives updates as they're posted. The cloud then pings each subscriber to letthem know that the content has been updated. The problem here is that once a feed has been updated and all the subscribers have been notified, each subscriber will have [ 0 request the feed from the server. Depending on how many subscribers there are, this could mean a 10t Of requests and a ton Of traffic on a big feed. Some clouds support hosting the RSS feed directly on the cloud, which relieves some load from the individual server, but the subscriber has tO download the entire feed either way. rssCloud isn't covered in detail in this book, but more information on it can be found at 〃な〃 : 〃る scloud. 0 鵬 . WebIogs. C0川 Pings Many blogging platforms support' pinging the blogosphere. ' ' These work by ping- ing known URLs as things are published. After being pinged, these services can then download the new feed. However, the basic method 0f pinging doesn't supply link [ 0 the actual feed URL, so the server must parse the site to find the usable RSS/ Atom feed. This protocol alSO doesn't allOW for arbitrary subscribers [ 0 receive pings or getthe data any faster than they would with standard poll requests. More information on this can be found at 印 : 〃ル訪 logs. co 襯ん襯 1. 10 ー Chapter 2 : ReaItime Syndication

・ http://cdc.ipandemic.com/%d ・ % options ・ port app host 'http://www.ipandemic.com/feed' topic url "http://pubsubhubbub.appspot.com/ hub url = = app host + "/catch/germs" callback url # set up the different urls we need to know logging. inf0("Subscribing t0 the HUB") data = urllib. urlencode({ ・ hub. topic' : topic url, ・ hub. callback ・ : callback url, ' hub. verify ・ 'hub. mode' async ' ・ subscribe' } ) try: = ur11ib2. urlopen(hub url, data) response except (IOError, urIIib2. HTTPError), e: if hasattr(), 'code ・ ) and e. code = logging. inf0("204: NO Content") error if hasattr(e, ・ read'): error = e. read() logging. info('%s, Response: "%s" ・ % (), e て ro て ) ) This should 100k vaguely familiar from when we subscribed to a hub way back in Chapter 3. Before we can contact the server and request the subscription, we have t0 define some variables to explain where everything is located. The first variable, app host, is the endpoint ofthis Tornado-based command center application. This host must be publicly accessible because the hub is going to make HTTP requests [ 0 it. For this variable, enter in the final URL that you'll be using. The next variable, topic url, is the URL of the Atom feed [ 0 which we want to sub- scribe. This would be the Atom feed built previously in this chapter. After that, you can safely leave the variables as they are in the text. We're just defining the URL of the hub that is shared by multiple applications. FinaIIy, we define the callback url, which is the endpoint URL used when the hub has new data. Subscribing [ 0 a hub iS a two-step process. The first part is requesting the subscription, which we've just defined. The second part of the subscription process is responding to the challenge requestsent by the hub. This challenge request is used by the hub to ensure that the URL we provide is actually expecting t0 receive updates. To define this, we need [ 0 add a new URL route and a controller class. Let's define that now: class App1ication(tornado. web. App1ication) : def init_(self) : handlers , HomeHand1er), (r"/login/?", GoogleLoginHand1er), (r"/logout/?" LogoutHand1er), (r"/catch/germs/?", CatchGermsHand1er) , = 204 : ( 1a55 CatchGermsHand1er(tornado. web. RequestHand1er) : def get(self) : Building theGame ltself ー 281

(r"/", HomeHand1er), settings "static path" : os. path. join(os. path. dirname( file_), "template path" : 05. path ・ join(os. path. dirname( file ) , "cookie secret" you-cookie-secret ・ "/login" "login url" tornado. web. Application. "static"), "templates"), **settings) init_(self, handlers, if name ma1n tornado. options. parse command line() http server = tornado. httpserver.HTTPServer(App1ication()) http server. listen(options. port) tornado. i0100P. IOLOOP. instance(). start() ThiS is essentially the same base Tornado application that we've used in previous chap- ters. ・み e import what we need, build an application class, and start the application. There are only a couple Of things tO notice here. ln this particular instance, we specify a default port 0f 8088. 嶬市 ile it's not strictly necessary tO use this port, this application must run on a port that can be accessed from App Engine, which has restrictions on the ports it can access. We're using the reference PubSubHubbub hub located at pubsubhubbub. appspot ・ com, which runs on App Engine. AISO, we specify a cookie secret in this code. This will allow Tornado tO secure the cookies against forgery. HomeHand1er, the controller class that handles the root URL 0f the domain, just does tWO quick things. First, HomeHandler tries tO load the user information out Of the secure COOkie. Then, it simply renders a template called CO 川川 4 れ d イ e れ r. わ t 川 l. Let's define that file now. Create a file in your tornado directory called 襯お / co 川襯 d - 記厩既 / 山司 and add the following code ・ く ! DOCTYPE html> く html> く head> く meta http-equiv="content-type" content="text/html; charset=utf-8"/> く tit1e>iPandemic - Command Center く /title> - for jQuery - - > く script type="text/javascript" src="http://www ・ google ・ com/jsapi"> く /scrip わ - fo て the maps API - く script src="http://maps ・ google ・ com/maps?fi1e=api&v=2&sensor=fa1se&key= YOUR-GOOGLE-MAPS-API-KEY" type="text/javascript"> く /script> our 10Ca1 JS く script type="text/javascript" src="{{ static url(' cc. js ' ) }}"> く /script> く /head> く body onload="cc. initialize('map' , false); " onunload="GUn10ad()" class="control-center"> 276 ー Chapter 10 : Putting lt 則 Together

Both of the controllers that allow for us to serve the JavaScript and accept pings from the server are ready tO go. The next thing tO dO is set up the Tornado application Object and start the server. Add the following code to your び . file: class App1ication(tornad0. web. App1ication) : def init_(self) : # create an application-wide version Of the analytics object self. analytics = Ana1ytics() # map the two controller classes tO the URLS handlers (r"/ping/?", PingHandler), # define static file path that we' 11 be using settings = dict( static path=os. path. join(os. path. dirname( file ) , , JSHandIer), "static-new"), # start the application! tornado. web. App1ication. # keep this at the bottom 0f the file if name ma1n tornado. options. parse command line() http server = tornado. httpserver.HTTPServer(App1ication()) http server. listen(options. port) tornado. i0100P. IOLOOP. instance(). start() Once again, this code should 100k fairly familiar if you've followed along with any of the other Tornado-based examples. The firstthing we do is define an App1ication object that extends thetornado. web. App1ication class. ln ⅱ , we instantiate an AnaIytics object that is used by the controller classes. Next, we define a couple of URL mappings from the actual URL to the controller class that handles the work. The rest of the code is essentially Tornado boilerplate COde that gets the server up and running on the correct port. Testing the pings ln order tO test this out, the first thing you need tO dO is decide which server which iS going t0 run it. This code was designed t0 be flexible in terms 0f where and how you run it. AS such, you have a number Of options with regard tO running the server COde. You can put it on a public-facing site, either on itS own port or on the standard port 80, ifthat's available. lfyoujust want to test it out yourself, you can easily start itlocally and limit access tO only your browser. Either way, once you ve copied the COde onto the correct server, launch it fror れ the command line: $ python server ・ py ー po て t = 8888 init_(self, handlers, **settings) 196 ー Ch 叩 ter 9 : Measuring U 史「 Engagement: Analytics on the Realtime Web