Data Models

Starbloom offers standardized data models to ensure consistency and interoperability across different data sources and consumers. These models are maintained in our starbloom-proto repository. We encourage you to use these standard data models when possible so you get the most compatibility with existing data producers and consumers in the same category, however you are also free to create and use custom data models when necessary.

Core Data Types

Our standard schemas cover common blockchain and DeFi data structures:

Examples of Standard Messages

message Block {
    google.protobuf.Timestamp timestamp = 1;
    string hash = 2;
    uint64 number = 3;
    string parent_hash = 4;
    uint64 gas_used = 5;
    uint64 gas_limit = 6;
}

message Transaction {
    google.protobuf.Timestamp timestamp = 1;
    bytes from_address = 2 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
    bytes to_address = 3 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
    string value = 4 [(starbloom.numeric) = {}];
    uint64 gas_used = 5;
    string gas_price = 6 [(starbloom.numeric) = {}];
}

message AMMTrade {
    google.protobuf.Timestamp timestamp = 1;
    bytes trader_address = 2 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
    string trade_id = 3;
    bytes input_token_address = 4 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
    string input_amount = 5 [(starbloom.numeric) = {}];
    bytes output_token_address = 6 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
    string output_amount = 7 [(starbloom.numeric) = {}];
}

Field Options

Starbloom provides several options to enhance protocol buffer messages and fields:

Bytes Formatting

The bytes option specifies how binary data should be encoded/decoded:

enum FormatType {
    FORMAT_TYPE_BASE64 = 0;  // Standard base64 encoding
    FORMAT_TYPE_HEX = 1;     // Hexadecimal encoding
}

// Example usage
message Transaction {
    bytes address = 1 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
}

Numeric Handling

The numeric option defines precision and scale for numeric values:

message NumericOption {
    optional uint32 precision = 1;  // Total significant digits
    optional uint32 scale = 2;      // Decimal places
}

// Example usage
message Trade {
    string amount = 1 [(starbloom.numeric) = {precision: 18, scale: 8}];
}

Indexing

Fields can be indexed for faster querying:

message IndexOption {
    IndexType type = 1;   // ORDERED or UNORDERED
    bool unique = 2;      // Unique constraint
}

// Example usage
message Transaction {
    string hash = 1 [(starbloom.index) = {
        type: INDEX_TYPE_ORDERED,
        unique: true
    }];
}

Schema Evolution

Version Control

All data models follow semantic versioning:

{author}.{connector}.{version}.{event_name}
example: starbloom.ethereum.1_2_0.Block

Backward Compatibility Guidelines

1. Field Numbers

   • Never reuse field numbers
   • Keep deprecated fields reserved
   • Add new fields with new numbers

2. Field Types

   • Maintain wire format compatibility
   • Use optional fields for new additions
   • Consider default values carefully

Evolution Example

// Version 1.0.0
message Trade {
    google.protobuf.Timestamp timestamp = 1;
    bytes trader = 2 [(starbloom.bytes).format = FORMAT_TYPE_HEX];
    string amount = 3 [(starbloom.numeric) = {}];
    
    // Deprecated in 1.1.0
    reserved 4;
    reserved "old_field";
    
    // New in 1.1.0
    string trade_id = 5;
}

Best Practices

1. Schema Design

   • Use clear, descriptive field names
   • Include comments for complex fields
   • Group related fields together
   • Use appropriate data types
   • Consider query patterns when designing indexes

2. Field Options

   • Use bytes formatting for blockchain addresses
   • Apply numeric options for financial values
   • Add indexes for frequently queried fields
   • Use composite indexes for common query combinations

3. Versioning

   • Maintain backward compatibility
   • Document breaking changes
   • Use semantic versioning
   • Plan for future extensibility

Contributing

We welcome community contributions to improve our standard schemas. To contribute:

1. Fork the starbloom-proto repository
2. Make your changes
3. Submit a pull request with:
   • Clear description of changes
   • Use cases and rationale
   • Backward compatibility considerations