Ian Barber: How Google Builds Web Services

Orde Saunders' avatarPublished: by Orde Saunders

Ian Barber (@ianbarber) was talking at Whisky Web II about how Google builds web services, these are my notes from his talk.

  1. What do Google web services look like?
  2. Why do Google do the things they do?

Why webservices?

We're all building them, is this interesting? Making a API you can call, the main consumers of our data is code - not HTML being sent out. Exposing data to 3rd parties - if your API is good and your client is bad you lose your client to others that build better clients.

GData

Last big iteration of Google's data services was GData.

  • RESTful
  • Atom format

Billions of requests per day going through these APIs. Around 25 data sources, not all of them have docs or client libraries which indicates they are hard to use or not maintained. People got upset with GData as there was a high knowledge barrier to entry and it wasn't working well internally.

Who are APIs for?

  • Internal developers - build the API
  • API consumers - use the API
  • Service users - create the data (e.g. Gmail users)

Don't expose an API unless you know that all three will work together to create a better product.

Cost benefit analysis

If the API is internal then you can make changes to the API and get internal consumers to change to match that. If it's external then you have a commitment to not change it and support it.

Benefit is that the service becomes more available and more successful. People will use your service because it integrates with their service. People can also extend your service with additional functionality they need. When you build services you can change what people can do with your service and you can exploit more areas of your service.

APIs for APIs

At Google they have put in place a central place for all API services (Discovery) that takes care of shared problems to all APIs:

  • auth
  • data format conversion
  • abuse management
  • billing
  • versioning

Now they have more APIs and they are richer. More people are using them so more data for Google.

Client libraries now only have to be written once and they work for all APIs rather than different libs for each API. Core to this is a JSON discovery document that describes the API and everything you need to interact with that API such as URL, methods, query strings, schemas &c. This discovery document can even be used to generate code to query the API.

As different products are all using the same vocabulary to define their APIs it means that the code to consume these different APIs is consistent (even though the data structures are different).

Teams don't have to worry about creating an API, they can just define their data and the API is defined for them with all the functionality out of the box.

This is the approach that Google has adopted as it works for them, others may want to adopt it but it's not intended to be a standard.

There is a version of this available in Google App Engine that will allow you to define your own API using this same infrastructure and will work with the standard Google API client libraries.

As the APIs are all standard it enables Google to have an API explorer web interface that allows people to play around with the API and see what it can do without having to write code.

Core considerations for building an API

  • Who is it for?
  • What are the benefits?
  • How much is it going to cost?
  • Define your basic components - think generic rather than specific.
  • Have a standard idom for all your APIs if at all possible.

Resources