From everything that I have read and experienced, web hooks are awesome! They let developers easily extend and integrate web applications and allow users to receive events and data in real-time. Yep, real-time. No polling here folks. So what are web hooks? Lets start with examples, followed by theory, and then cap it off with code.
Examples
The best example of web hooks, that you would most likely be familiar with, is GitHub. GitHub has a services tab in the admin of each repository that allows you to send post-commit hooks to URLs which you specify. They even have a handful of pre-rolled hooks, such as Basecamp, Campfire, Email, FogBugz, IRC, Jabber, Lighthouse and Twitter, and have even open sourced the code.
Another example you may be familiar with, that has been around a little longer, is PayPal's Instant Payment Notifications (IPN). IPN "is PayPal's interface for handling real-time purchase confirmation and server-to-server communications" according to PayPal's website. Translated to English, PayPal sends a request to a URL you specify whenever someone completes a transaction. Ryan Bates has a few fantastic screencasts on PayPal's IPN on Railscasts.com.
GitHub and PayPal are two great examples, but what about a well-known application that does not use web hooks and could benefit? The first that comes to mind for me is Feedburner, a service that provides statistics of subscriber counts and more for feeds. It updates these numbers once daily, yet there are probably thousands of desktop and web applications that repeatedly poll the Feedburner Awareness API throughout the day.
Imagine if, instead of polling Feedburner's API, developers could just provide Feedburner with a URL of their choosing. Once a day, when Feedburner finished creating the daily summaries, they could post real-time requests to the developer's URL with information about the summary. No more cron job for the developer and no more polling the API for updates. Feedburner simply says, “Hey, we just updated the statistics for this feed, here you go.” Just like that, the developer can easily sync up Feedburner data in their application or send real-time notifications to those that are stat addicts like me.
Tired of writing the code and setting up infrastructure to receive email in your web apps? Rick Olson (a.k.a. technoweenie) sure was. That is why he created Astrotrain, a micro-app that sends hooks by way of an HTTP Post request or a Jabber message whenever an email is received. An instance of Astrotrain actually powers the email functionality of Lighthouse and Tender, two popular apps that Rick has worked on.
The possibilities are pretty much endless with web hooks and these examples are just the tip of an iceberg. Now, we have proof that “real” companies are using web hooks. I don't know about you, but I'm relieved it isn't just a crazy fad. How about we dive in a bit more with some theory?
Theory
A while back, when I first started doing research about web hooks, there was one graphic that really made things click for me. It compares a unix program to a web application. To give credit where credit is due, this graphic's original form was from a presentation by Jeff Lindsay, who, as far as I can tell, coined the term “web hooks”.
Program vs Webapp
As the figure above illustrates, an API for a web application is much like STDIN for a unix program. They both help get data in, but what about getting it out? Sure, you can get data out with an API, but only through polling, which is not real-time. I'll explain this more in a sec, but trust me when I say that polling sucks.
Unix programs, on the other hand, have STDOUT, which allows piping commands together (ie: gem list | grep rails). Each unix program completes a specific, simple task that alone is not overly special, but the sum of them working together is greater than the parts. What do web apps have that allow this kind of real-time chaining?
Nowadays, most web apps have feeds and/or some sort of API that allow you to poll for updates, but this is not the answer. Why should we have to ask? The app knows when it has new data, why can't we just tell it to give us the data right away? I am a visual learner, so lets take a look quick at the difference between polling and pushing.
Polling vs. Pushing
By pushing new data when an event happens, the web application no longer has to act like a father during a long car trip, constantly telling us, the developers, "Not Yet!". Also, we can drop the annoying kid act, continually asking, "Are we there yet?". Pretty cool. Push is better than pull. Get it, got it, good.
How Can We Push Data?
Now we know that push is better than pull, but how can we put this into practice? One popular way to push real-time data around is XMPP (Extensible Messaging and Presence Protocol, ie: Jabber). XMPP is great, but it is a heavyweight. You will need another server and to learn another protocol.
Wouldn't it be nice if we could use a protocol that we already knew? Enter web hooks. Web hooks, in their most simple form, are push over http. Below is an example of the most simple web hook you can create using Ruby.
require 'net/http'
Net::HTTP.post_form(URI.parse(url), {
'from' => message.from,
'to' => message.to,
'subject' => message.subject,
'body' => message.body
})
If you can add something like the code above into your application, you can implement web hooks. Enough with the examples and theory, lets do some coding!
Code
For the code portion of this article, we are going to build an app that sends web hooks and a second app that receives and processes those hooks. To help understand the process, lets take a look at one more visual.
Now that we understand what we are going to create, lets generate the two apps and some scaffolding to help get the point across quickly.
rails webhook_sender
cd webhook_sender
ruby script/generate scaffold Widget name:string
rake db:migrate
cd ..
rails webhook_receiver
cd webhook_receiver
ruby script/generate scaffold Message body:text
rake db:migrate
cd ..
At this point, we have two apps. webhook_sender has widgets, which have just a name, and webhook_receiver has messages with a body to store the message contents. Lets start with the webhook_sender app, implementing the functionality to send an HTTP request (web hook) whenever a widget gets modified.
Sending Hooks
We could use ActiveRecord callbacks for this functionality, but Rails actually has a built-in mechanism for this type of thing known as observers. If you aren't familiar with observers, they are classes that respond to lifecycle callbacks, create, update and destroy to name a few, to implement trigger-like behavior outside of the original class (the Widget model).
cd webhook_sender
ruby script/generate observer Widget
Now start up your favorite editor and open up the file app/models/widget_observer.rb. All we have to do is create method names that are similar to the ActiveRecord callbacks we want to hook into.
class WidgetObserver < ActiveRecord::Observer
def after_create(record)
send_hook(record, :create)
end
def after_update(record)
send_hook(record, :update)
end
def after_destroy(record)
send_hook(record, :destroy)
end
private
def send_hook(record, operation)
uri = URI.parse('http://localhost:3001/hooks/create')
Net::HTTP.post_form(uri, {
'id' => record.id,
'name' => record.name,
'operation' => operation
})
end
end
The final thing before we start up our sending app is that we need to tell Rails that our WidgetObserver should always be running. Open up config/environment.rb and add the following line:
config.active_record.observers = :widget_observer
Our app is now aware of our WidgetObserver at all times and ready to start sending web hooks. Lets start this bad boy up!
ruby script/server
Receiving Hooks
Now that our sender app is up and running, lets get our receiving app whipped into shape. First, we are going to need a controller to receive the sent hook. Open up a new terminal tab (or window) and run the following commands.
cd webhook_receiver
ruby script/generate controller Hooks
Next, we need an action in that controller to receive the hook message and process it. Open up app/controllers/hooks_controller.rb and change it to the following.
class HooksController < ApplicationController
protect_from_forgery :except => :create
def create
body = params.except(:action, :controller)
Message.create(:body => body)
render :nothing => true
end
end
Because Rails comes with Cross-Site Request Forgery (CSRF) protection, we need to tell this controller to skip that, otherwise we'll get invalid authenticity token errors when receiving the hook.
In the create action, we create the message and then render nothing. Remember that our sending application doesn't care if we receive this message or not, it just sends the message, therefore :nothing is a perfectly appropriate response. Note also that we excluded the param keys :action and :controller as those do not matter at all for the hook.
We have now created widgets, set the widgets to send hooks when they are modified, and created an application to receive and process those hooks. Lets start up our webhooks_receiver app as well, but on port 3001, so that it does not conflict with our currently running webhook_sender app.
ruby script/server -p 3001
Everything should be up and running and hooked together correctly, but lets check it in a browser just to be sure. Open up two tabs (or windows) in your browser of choice, the first to http://localhost:3000/widgets and the second to
http://localhost:3001/messages.
Create a new widget in the first tab, using the "New widget" link and then refresh the messages tab. You should see the new message in the list. Congratulations! You just created and received your first web hook.
Beyond The Basics
This is a very simple example so that it can be understood by a broader audience, but I hope that you get the idea of how powerful and flexible web hooks can be. That said, if you are going to start implementing hooks into your apps, you will want to consider the following.
Interface for Managing
Typically, you should offer an interface for users to define their own web hook URLs instead of hard coding the url in the WidgetObserver. When I say interface, I am not necessarily referring to a web interface. An API would be a perfectly acceptable way of creating, updating and deleting web hooks. You will also want to support multiple hooks per user and multiple users. The example I provided does not.
Queue the Sending
A good principle to live by in application development is if something can be moved out of the request/response cycle, do it! If I were going to implement hooks like this in a production application, instead of sending the net/http request in the WidgetObserver, I would queue it, using delayed_job or some other mechanism.
The benefit of queueing these hooks is two fold. First, you move the possibly slow process of sending the hooks outside of the request/response cycle. This means the users creating the data using the web interface do not have to wait for hooks to be sent before going about their business. This is particularly important if the subscribing server is out of commission or running slow.
Second, what if one of the subscribing servers is down? If you are doing a once or nothing request, the subscriber will miss out and have to poll for the missed data. On the other hand, if you queue the hook, you can leave it in your queue and keep trying until the subscriber responds with success. Shopify's wiki page on web hooks explains how they leave the request in the queue until the subscribing server responds with a successful response, and even provide a simple example.
Thoughts on Security
The other thing you may have noticed in the example code is that it would be really easy to spoof a hook to the receiving app. Shopify takes the simple route by providing you with a key, which you can check before processing the hook, thus weeding out possible attacks and spiders. PayPal goes to the extreme, allowing you to set up SSL certificates and encrypt the data being sent. For most apps, a simple key or basic authentication should be fine. You will have to decide what level of security is best for you and your app's users.
Conclusion
Web hooks are the most simple way to add real-time notifications to your web applications and they are not just for HTTP. As GitHub's services and Astrotrain show, you can send web hooks using a variety of formats (HTTP, Email, Jabber, IRC).
There are also some very interesting applications popping up, Switchub and AppJet for starters, that deal solely with receiving input, processing that input and then generating output. Imagine a world where web applications can be piped together in the same fashion as Unix programs. That gets me excited about the future!
I leave you, not with a powerful statement that shocks your soul, but rather a list of related links in hopes that this article has left you wanting.
Published in Issue #1: The Beginning
Back