Ryan Richards: Front-to-back: Architecting your data reporting and visualisation stack
Ryan Richards (@rsandor) was talking about his experiences working on fastly.com at Scotland.js, these are my notes from his talk.
What is Fastly?
- Based on Varnish
- Datacentres around the world
- 10^10 requests per day
- Back end: collection and storage of performance data.
- Middle end: querying and publishing
- Front end: Visualization and reporting
- You can't - it's a fact.
- Queueing and waiting before publishing
- Backfill the visualisations (doesn't look good)
- Use what you know...
- Perl over Node
- HTTP over 0MQ
- Canvass over d3
- If you don't know...
- Find someone that is already doing it
- Ask good questions up front
- Come up with a plan
- Conduct interviews with everyone asking what they need.
- Independent from the core API
- Highly optimised MySQL queries
- Easy to use time parameters (
from=2 weeks ago&to=now&by=how)
- Uses d3
- Neat, but not perfect
- "Done" is a myth
- Data evolves
- Working on "by second" historical stats
- Better interface