Messaging Bus

From Laen

Contents 1 Vision 2 Reason For Being 3 Requirements 4 Possible Models 4.1 Publish Subscribe 5 Queues, Recipients, Channels, or Tags 5.1 Tags 6 Possible Transports 6.1 lpr/ipp 6.2 IRC 6.3 syslog 6.4 smtp 6.5 xmlrpc/SOAP over HTTP 6.6 SSHRPC 6.7 Jabber/XMLRPC 6.8 AMQP 6.9 Spread 6.10 Memcached? 7 AMPQ 7.1 Implementations 7.2 More Info 8 Spread

Vision

I need a reliable message-passing interface between machines.

Sending:

1. Queue message to local disk.
2. Connect to messaging server (or use existing connection)
3. Send message, await confirmation
4. Remove message from local disk.

Reason For Being

I want certain things to happen automatically on my network..

• The first time my host comes up on the network I want it to:
  • Register its public SSH host key with the KeyMaster.
  • Register with the backup server to ensure it gets backed up.
• Tell the Nagios server what should be monitored
  • Let Nagios know the state of programs and processes on it.
• Report statistics to the Cacti server

Requirements

Scalable to 100,000 event generating nodes, each pushing 10 events per second and 10,000 event receiving nodes. One million events per second. How many things to watch?

This means that it needs to scale arbitrarily sideways. Add more servers to add capacity. DNS load balacing? Events come in, get processed, get... sent? How? Event routers don't make outbound connections.. Must every receiver connect to every router? No.. It connects to one router, that router updates its routing table and advertises that any event headed to that server must go

Possible Models

Publish Subscribe

"NodeNagios" is a Nagios monitoring server. It wants to know a lot of things, like when a machine's disk fails so it can alert the appropriate people.

• NodeNagios contacts the messaging server and subscribes to: "All new events matching the pattern /RAID STATUS CHANGE/".

(The definition of a "new event" is flexible.. It could mean "since I last checked", or "in the past 5 minutes")

"NodeA" is a machine with a mirrored disk. One of its disks is about to fail.

• NodeA's disk fails.
• NodeA's RAID subsystem generates a disk failure event:

RAID STATUS CHANGE for VolumeID 2 -- PhysDisk 1 has failed. Volume is now degraded

• NodeA pushes this out onto the messaging bus.
• NodeNagios, having subscribed to this type of event, receives it and passes it to the program(s) that are subscribed to this particular event.

Let's think of the implications here...

• The messaging bus is handled at a system-level and not a program level.
• The events are timestamped and given a unique QueueID.. Sequential? UUID? Is UUID sequential, because that could be fun.. Maybe it's just a sequence number given to the client by The Bus?
• Either the Bus or NagiosNode is keeping track of the time of the last event.
• What the heck _is_ The Bus?
• Does The Bus do any message persistance, or is it just a message router?

Queues, Recipients, Channels, or Tags

(Here's where I get extremely juvinile)

• Barfer sends data to multiple IRC-like channels
• Barfer tags data with certain keywords
• Barfer sends data to specific recipients

• Munchers tell Barfers which messages to send to them.
• Munchers

Tags

• Barfer sends data to Bus with certain tags attached.
  • FQDN and arbitrary keys ("Cluster=Compute1").
  • These keys are unauthenticated.
• Bus is configured to recognize certain patters and add more tags.
• Munchers subscribe to sets of tags (ala del.icio.us)

Possible Transports

lpr/ipp

• Excellent for one-way message passing.
• Well... IPP is just HTTP anyway...

IRC

syslog

• Pros: Fire and Forget
• Cons: Fire and forget.. No two-way communication

smtp

xmlrpc/SOAP over HTTP

• Pros: Two way communication
• Pros: HTTP/HTTPS

SSHRPC

• Pros: Two way communication
• Pros: Multiple two-way sockets over the same network connection
• Pros: Tunneling!

Jabber/XMLRPC

AMQP

Spread

Memcached?

It's a key/value store, isn't it?

AMPQ

Advanced Message Queuing Protocol - ..An open protocol for messaging systems.

Implementations

• Apache QPID
• RabbitMQ
• OpenAMQ

More Info

• AMQP

Spread

Spread website

Really simple to configure. All members share the same configuration file. You just specify the multicast address to join, and a list of the other member machines in the groups.

Messages are sent via broadcast, multicast, or unicast. It's a group-based publish/subscribe model.

A daemon runs on each machine, and clients contact it to join channels. Once you join a channel, you poll to receive the message.

There doesn't appear to be any authorization or authentication support, so you can't have a high degree of trust that the messages are authentic.