Skip to main content

Architecture Overview

Starbloom is designed as a modular, scalable data infrastructure platform for web3 applications. This document explains the key architectural components and how they work together.

System Architecture​

Key Components​

1. Data Ingestion Layer​

The ingestion layer consists of specialized connectors that handle data acquisition from various sources:

  • Source Connectors: Modular components that:
    • Connect to specific data sources
    • Transform raw data into standardized formats
    • Ensure data consistency and reliability
    • Handle source-specific error cases
    • Manage reconnection logic

2. Message Queue​

The message queue serves as the central nervous system of Starbloom:

  • Properties:
    • Distributed messaging system
    • Guaranteed message delivery
    • Topic-based routing
    • Horizontal scalability
    • Message persistence
    • Fault tolerance

3. Stream Processing Layer​

Stream processors handle real-time data transformation and enrichment:

  • Capabilities:
    • Data filtering
    • Aggregation
    • Transformation
    • Enrichment
    • Window operations
    • State management

4. Storage Layer​

Multi-tiered storage system optimized for different access patterns:

  • Time Series Database:

    • Optimized for temporal data
    • High write throughput
    • Efficient range queries
    • Data retention policies

  • Key-Value Store:

    • Fast random access
    • Flexible schema
    • Atomic operations
    • Cache-friendly design

  • Cache Layer:

    • In-memory caching
    • Distributed caching
    • Cache invalidation
    • Cache coherence

5. API Layer​

Multiple interfaces for data access:

  • REST API:

    • Historical data queries
    • CRUD operations
    • Pagination
    • Filtering

  • WebSocket API:

    • Real-time updates
    • Bi-directional communication
    • Subscription management

  • Server-Sent Events (SSE):

    • One-way real-time updates
    • HTTP-based streaming
    • Auto-reconnection

Data Flow​

  1. Ingestion:

  2. Processing:

  3. Storage & Serving:

Scalability & Reliability​

Horizontal Scaling​

  • Each component can scale independently
  • Stateless design where possible
  • Load balancing across components

Fault Tolerance​

  • Automatic failover
  • Message persistence
  • Error recovery
  • Data replication

Monitoring & Alerting​

  • Component health checks
  • Performance metrics
  • Error tracking
  • Resource utilization

Security Considerations​

  1. Authentication & Authorization:

    • API key management
    • Role-based access control
    • Request signing
    • Rate limiting

  2. Data Security:

    • Encryption at rest
    • Encryption in transit
    • Secure key storage
    • Audit logging

Next Steps​

Additional Resources​