Anton Whalley: Realtime replication in the realworld with node.js

Orde Saunders' avatarPublished: by Orde Saunders

Anton Whalley (@antonwhalley) was talking about realtime replication at Scotland.js, these are my notes from his talk.

Ownership of 'your' data has the potential to be a serious issue.

Peer to peer networks in production

Cinema content distribution

  • Content = adverts and films
  • Traditionally couriers and DVDs
  • 12k+ machines
  • File size is large (GB range)

  • BitTorrent - parses file and seeds network, not good for discovery.

  • Content meta data (discovery) provided by CRDT. Distributes meta data between machines.
  • UI built in Node.js, express seneca and angular.
  • Auth and management.
  • No tests but lots of live system monitoring built in from the start. System monitoring provides confidence to customers.
  • Token based system that partitioned clients to restrict what parts of the distributed network can be seen.

Lessons

  • Unix philosophy of small components.
  • Use LevelDB rather than Mongo due to stability issues.
  • HTTP is not a socket.
  • No continuous deployment - communication issues with ops team working for client.
  • Performance driven development.
  • Central authority and non-repudiation - not everyone has a certificate authority in place.