Anton Whalley: Realtime replication in the realworld with node.js
Anton Whalley (@antonwhalley) was talking about realtime replication at Scotland.js, these are my notes from his talk.
Ownership of 'your' data has the potential to be a serious issue.
Peer to peer networks in production
Cinema content distribution
- Content = adverts and films
- Traditionally couriers and DVDs
- 12k+ machines
File size is large (GB range)
BitTorrent - parses file and seeds network, not good for discovery.
- Content meta data (discovery) provided by CRDT. Distributes meta data between machines.
- UI built in Node.js, express seneca and angular.
- Auth and management.
- No tests but lots of live system monitoring built in from the start. System monitoring provides confidence to customers.
- Token based system that partitioned clients to restrict what parts of the distributed network can be seen.
Lessons
- Unix philosophy of small components.
- Use LevelDB rather than Mongo due to stability issues.
- HTTP is not a socket.
- No continuous deployment - communication issues with ops team working for client.
- Performance driven development.
- Central authority and non-repudiation - not everyone has a certificate authority in place.