Ryan Richards: Front-to-back: Architecting your data reporting and visualisation stack

Orde Saunders' avatarPublished: by Orde Saunders

Ryan Richards (@rsandor) was talking about his experiences working on fastly.com at Scotland.js, these are my notes from his talk.

What is Fastly?

  • CDN
  • Based on Varnish
  • Datacentres around the world
  • 10^10 requests per day

Architecture

  • Back end: collection and storage of performance data.
  • Middle end: querying and publishing
  • Front end: Visualization and reporting

Fixing latency

  • You can't - it's a fact.
  • Queueing and waiting before publishing
  • Backfill the visualisations (doesn't look good)

Lessons

  • Use what you know...
    • Perl over Node
    • HTTP over 0MQ
    • Canvass over d3
  • If you don't know...
    • Find someone that is already doing it
  • Ask good questions up front
  • Come up with a plan
  • Conduct interviews with everyone asking what they need.

API

  • Sinatra
  • Independent from the core API
  • Highly optimised MySQL queries
  • Easy to use time parameters (from=2 weeks ago&to=now&by=how)

Interface

  • Uses d3
  • Neat, but not perfect

The future

  • "Done" is a myth
  • Data evolves
  • Working on "by second" historical stats
  • Better interface