Ryan Richards: Front-to-back: Architecting your data reporting and visualisation stack

Ryan Richards (@rsandor) was talking about his experiences working on fastly.com at Scotland.js, these are my notes from his talk.

What is Fastly?

• CDN
• Based on Varnish
• Datacentres around the world
• 10^10 requests per day

Architecture

• Back end: collection and storage of performance data.
• Middle end: querying and publishing
• Front end: Visualization and reporting

Fixing latency

• You can't - it's a fact.
• Queueing and waiting before publishing
• Backfill the visualisations (doesn't look good)

Lessons

• Use what you know...
  • Perl over Node
  • HTTP over 0MQ
  • Canvass over d3
• If you don't know...
  • Find someone that is already doing it
• Ask good questions up front
• Come up with a plan
• Conduct interviews with everyone asking what they need.

API

• Sinatra
• Independent from the core API
• Highly optimised MySQL queries
• Easy to use time parameters (from=2 weeks ago&to=now&by=how)

Interface

• Uses d3
• Neat, but not perfect

The future

• "Done" is a myth
• Data evolves
• Working on "by second" historical stats
• Better interface