Need to move messages from your AMQP broker into a data warehouse or analytics platform? The new Airbyte source connector for AMQP makes it simple to stream data from LavinMQ or RabbitMQ directly into any Airbyte destination.
What is Airbyte?
Airbyte is an open-source data integration platform that moves data between sources and destinations. It supports hundreds of connectors out of the box, letting you extract data from APIs, databases, and message queues, then load it into data warehouses like PostgreSQL, Snowflake, or BigQuery.
The platform handles the complexity of data pipelines: scheduling, monitoring, schema detection, and incremental syncs. You configure a source, pick a destination, and Airbyte takes care of the rest.
Why stream AMQP messages to a data warehouse?
Message brokers like LavinMQ and RabbitMQ excel at real-time communication between services. But sometimes you need to analyze that message data over time. Common use cases include:
- Tracking order events for business intelligence dashboards
- Storing IoT sensor readings for historical analysis
- Archiving audit logs for compliance requirements
- Building analytics pipelines from event-driven architectures
The Airbyte source connector bridges these two worlds. It consumes messages from your broker and delivers them to any destination Airbyte supports.
How the connector works
The connector uses the AMQP 0-9-1 protocol, which means it works with both LavinMQ and RabbitMQ. When you configure a sync, the connector creates a queue, binds it to your specified exchange, and starts consuming messages.
Each message becomes a record in your destination. The connector preserves the message payload as JSON and adds a timestamp for when Airbyte processed it.
Configuration options
The connector requires a few pieces of information to connect:
- AMQP URL: Your connection string, including credentials and vhost
- Exchange: The exchange to consume messages from
- Queue name: The queue the connector creates and binds to the exchange
- Binding key: Optional routing key for filtering messages
- Stream name: The name of the Airbyte stream
Getting started
The connector is available on Docker Hub as
cloudamqp/airbyte-source-amqp
and the source code is available on
GitHub.
Note: Custom Docker connectors can only be added to self-hosted Airbyte or Airbyte Enterprise. Airbyte Cloud does not support adding custom connector images.
To add it to your Airbyte instance:
- Go to Settings, then Sources, then click New connector
-
Enter the Docker repository:
cloudamqp/airbyte-source-amqp -
Enter the Docker image tag:
latest - Save and create a new source using your AMQP connection details
Once configured, you can set up a connection to any Airbyte destination. PostgreSQL, BigQuery, S3, or any of the 300+ destinations Airbyte supports will work with messages from your broker.
Example: streaming messages to PostgreSQL
A typical setup might look like this: your application publishes order events to an exchange in LavinMQ. The Airbyte connector consumes those events and writes them to a PostgreSQL table. Your analytics team can then query that table to build dashboards and reports.
The configuration would include your CloudAMQP connection URL, the exchange name where orders are published, and a queue name for Airbyte to use. On the destination side, you configure your PostgreSQL connection and choose which schema and table to write to.
Summary
The Airbyte source connector for AMQP opens up new possibilities for working with data. You can now stream messages from LavinMQ or RabbitMQ into data warehouses, lakes, or any other Airbyte destination without writing custom code. The connector handles connection management, message consumption, and data formatting automatically.
Check out the GitHub repository for documentation and examples, or pull the image from Docker Hub to get started.
