Alvaro Videla: Scaling Applications with RabbitMQ
Alvaro Videla (@old_sound) was talking about scaling applications using RabbitMQ at Whisky Web II, these are my notes from his talk.
Why do you need messaging?
Classic web app to implement a photo gallery and we need to upload and display images. Can we now notify friends of the user when they upload new images? We also need to give badges for each upload. And post them to twitter. And the bandwidth bill is too high. And we need to call PHP from Python (and Java). And the users don't want to wait until you've finished doing all that before they see their image.
Implementation
- Image handler did the upload and placed it in a folder.
- Resize the image
- Notify friends
- Award badges
- Post to twitter
- Return response to browser
Use messaging
Publish/Subscribe Pattern. When an event is fired several subscribers can pick it up and run their tasks. The controller is now decoupled from the tasks which can run in separate processes.
- Handler does the upload
- Send a message that an image has been uploaded
- Return response to browser
Now when you need to add a new task you just subscribe it to the message.
Messaging and RabbitMQ
- Multi protocol Messaging Server
- Open Source
- Polyglot (PHP, node.js, Erlang, Java, Ruby, .Net, Haskell and more)
- Written in Erlang/OTP (Erlang has lots of concurrency libraries)
- Multi protocol (AMQP, MQTT,STOMP)
Some users of RabbitMQ
- Indeed.com
- MailboxApp
Messaging topology
To get message from from producer to consumer you need an exchange to receive messages. From the exchange they go to the queue where they are held until the consumer subscribes to the queue and receives the messages. Multiple queues can subscribe to an exchange. Direct exchanges have binding keys that if the message key matches they go to the queue that matches the binding key. Topic exchanges are like direct exchanges but accept patterns as message keys.
RabbitMQ simulator is an good resource for building and testing topologies.
To scale up you can add more consumers to a queue to handle more messages in less time. The consumers can be decoupled from the queues if not needed. Consumers can be written in any language and run on any machine and can be swapped out freely. You can run a new consumer on a queue that's a new version and when it's consuming the messages you can then decommission the original consumer.