Rembus

The zero-infrastructure messaging middleware

The goal of Rembus is to be a messaging middleware for systems where the mains building blocks are scientific computing and components decoupling.

It is a fresh approach to messaging: instead of requiring buffers or streams of bytes and loading the application with encoding/decoding tasks it accepts primitive types, dictionaries, lists and dataframes types through a thin API.

Data at Rest

Pub/Sub messages can be automatically materialized into tables by declaring a schema that maps topics to columns.

{
  "tables": [
    {
      "table": "telemetry",
      "topic": ":sensor/telemetry",
      "columns": [
        {"col": "sensor", "type": "TEXT"},
        {"col": "temperature", "type": "DOUBLE"},
        {"col": "pressure", "type": "DOUBLE"},
        {"col": "timestamp", "type": "UBIGINT"}
      ]
    }
  ]
}

Once materialized, the data becomes queryable. How to query depends on the catalog backend.

If the catalog is a traditional SQL database such as SQLite or PostgreSQL, multiple connections are permitted and standard SQL clients can access the database concurrently while Rembus is running:

SELECT * FROM telemetry WHERE temperature > 15.0

When using an embedded DuckDB catalog only the Rembus process has direct access to the database and queries should be executed via Rembus RPC using topic query_<topic>:

# Return a DataFrame.
df = rpc(
    cli,
    "query_telemetry",
    {"where": "temperature > 15.0"}
)

RPC

Rembus provides the request/reply pattern where a component request a service and another component implements and exposes such service.

There may be multiple components exposing the same service and different logics for routing the request to a specific server, implementing fail-over or balancing features.

Pub/Sub

One component produce a message and one or many interested subscribers receive it.

When a component goes online again if the retroactive feature is enabled the component receives the messages published when it was offline.

Brokerless

In a simple scenario or where a Broker may be redundant or tought as a single point of failure a brokerless architecture consents peer to peer comunication between components.

DataFrames support

Rembus uses the Arrow columnar format to transport dataframes messages between components.

Julia and Python components may exchange dataframes betweeen DataFrames.jl, Pandas and Polars libraries.