Delta Sharing Protocol — Cross-Platform Data Exchange with Databricks, Spark, and Pandas

What Delta Sharing Solves

Sharing datasets across organizational boundaries has historically required copying data. You extract a snapshot, compress it, upload it to an SFTP server or S3 bucket with temporary credentials, and your counterpart downloads the whole thing — even if they only need the last 30 days, or a single partition, or three columns out of fifty. Every subsequent update repeats the cycle. The receiver has no reliable way to know when the data changed, and the provider has no visibility into who accessed what.

Delta Sharing is an open protocol developed by Databricks and donated to the Linux Foundation that eliminates data copying. The sharing server never moves data — instead, it returns presigned cloud storage URLs (S3, ADLS, GCS) that the recipient uses to read files directly from the provider's object store. Only the files relevant to the query — based on partition pruning and predicate pushdown evaluated by the server — are returned. The recipient reads Delta Lake Parquet files directly using standard clients: the Python delta-sharing library, PySpark with the Delta Sharing connector, Power BI, or Tableau. No egress copies, no stale snapshots, no credential sharing.

This is distinct from Unity Catalog's internal governance model, which controls access within a single Databricks workspace or metastore. Delta Sharing extends that governance boundary outward — to partners running different clouds, different Spark distributions, or no Spark at all. A recipient with nothing more than Python and a bearer token can read a live table from your Delta Lake.

Protocol Architecture

The Delta Sharing protocol defines three components: a sharing server, a profile file, and a set of REST API endpoints. The server sits in front of a Delta Lake — it knows which tables are shared, who is allowed to read them, and how to generate short-lived presigned URLs for the underlying Parquet files. The client presents a bearer token, queries the REST API to discover available shares and tables, requests file metadata, and then reads files directly from cloud storage using the presigned URLs returned by the server. The server never streams data — it only orchestrates access.

REST API Endpoints

The Delta Sharing REST protocol is straightforward. Clients call four categories of endpoints: listing shares, listing schemas and tables within a share, reading table metadata (version, schema, protocol), and querying table data (with optional predicate hints). All endpoints use bearer token authentication via the Authorization: Bearer header.

# List all shares accessible with this token
GET /shares

# List schemas within a share
GET /shares/{share}/schemas

# List tables within a schema
GET /shares/{share}/schemas/{schema}/tables

# Get table metadata (version, schema, protocol)
GET /shares/{share}/schemas/{schema}/tables/{table}/metadata

# Query table data — returns a stream of JSON lines
# Each line is either a metadata action or a file action with presigned URL
POST /shares/{share}/schemas/{schema}/tables/{table}/query
Content-Type: application/json

{
  "predicateHints": ["date >= '2026-01-01'"],
  "limitHint": 10000,
  "version": 42
}

The query response is a newline-delimited JSON stream. The first line contains a protocol object describing the minimum reader version required, followed by a metaData line with the full table schema, followed by one file line per Parquet file the client should read. Each file line includes the presigned URL, file size, row count, and column statistics — the same statistics stored in the Delta Lake transaction log that enable predicate evaluation before reading.

Running the Open-Source Sharing Server

The delta-io/delta-sharing repository contains a reference server implementation in Scala that you can run against any Delta Lake on S3, ADLS, or GCS. You configure which tables to expose, create bearer tokens per recipient, and deploy it as a standard JVM service. This is the path for organizations that want to share data without Databricks or that run Delta Lake on open-source infrastructure like open-source Delta Lake alongside Iceberg and Hudi.

# server.yaml — reference server configuration
version: 1
shares:
  - name: "ecommerce_share"
    schemas:
      - name: "sales"
        tables:
          - name: "orders"
            location: "s3a://my-data-lake/delta/ecommerce/orders"
          - name: "products"
            location: "s3a://my-data-lake/delta/ecommerce/products"
  - name: "analytics_share"
    schemas:
      - name: "metrics"
        tables:
          - name: "daily_revenue"
            location: "s3a://my-data-lake/delta/analytics/daily_revenue"

# Authorization: bearer tokens per recipient
authorization:
  bearerTokens:
    - token: "a8f3d2c1..."   # partner-a-token
      expirationTime: "2027-01-01T00:00:00Z"
    - token: "b7e4c9d0..."   # partner-b-token
      expirationTime: "2027-01-01T00:00:00Z"

# Docker Compose — run the reference server locally
version: "3.8"
services:
  delta-sharing-server:
    image: deltaio/delta-sharing-server:latest
    ports:
      - "8080:8080"
    volumes:
      - ./server.yaml:/config/server.yaml
      - ~/.aws:/root/.aws:ro   # AWS credentials for S3 access
    environment:
      AWS_DEFAULT_REGION: us-east-1
    command: ["--config", "/config/server.yaml"]

Creating Shares in Databricks Unity Catalog

In Databricks, Delta Sharing is managed through Unity Catalog using SQL DDL statements or the Databricks Terraform provider. A share is a named container for tables, partitions, and views that you expose to specific recipients. Shares are metastore-level objects — one metastore can have many shares, and a table can appear in multiple shares with different access policies.

-- Create a share
CREATE SHARE ecommerce_external_share
  COMMENT 'Daily sales data for partner analytics teams';

-- Add a table to the share
ALTER SHARE ecommerce_external_share
  ADD TABLE catalog.sales_schema.orders;

-- Add a table with partition filtering — only share data from 2025 onward
ALTER SHARE ecommerce_external_share
  ADD TABLE catalog.sales_schema.orders
  PARTITION (year >= '2025');

-- Add a table with column masking — exclude PII columns
ALTER SHARE ecommerce_external_share
  ADD TABLE catalog.sales_schema.customers
  (customer_id, region, signup_date, tier);   -- explicit column list, no email/phone

-- Share a view (recipients see computed results, not raw tables)
ALTER SHARE ecommerce_external_share
  ADD VIEW catalog.reporting_schema.partner_summary;

-- Inspect what's in the share
SHOW ALL IN SHARE ecommerce_external_share;

-- Create a recipient (represents an external organization)
CREATE RECIPIENT partner_acme_analytics
  COMMENT 'ACME Corp analytics team — quarterly review access';

-- List recipients
SHOW RECIPIENTS;

-- Get the activation link for a recipient (one-time URL to download the profile file)
DESCRIBE RECIPIENT partner_acme_analytics;

-- Grant a share to a recipient
GRANT SELECT ON SHARE ecommerce_external_share TO RECIPIENT partner_acme_analytics;

-- Revoke access (token is immediately invalidated server-side)
REVOKE SELECT ON SHARE ecommerce_external_share FROM RECIPIENT partner_acme_analytics;

Reading Shared Data with Python

The delta-sharing Python libraryprovides a client that reads the profile file, queries the sharing server's REST API, and returns data as a Pandas DataFrame or PyArrow Table. No Spark cluster is required — the client downloads Parquet files directly using presigned URLs and reads them with PyArrow.

pip install delta-sharing

# profile.json — distributed by the data provider
# Download from the activation link in Databricks or generate from the reference server
{
  "shareCredentialsVersion": 1,
  "endpoint": "https://adb-1234567890.12.azuredatabricks.net/api/2.0/unity-catalog/delta-sharing",
  "bearerToken": "a8f3d2c1b5e7f9...",
  "expirationTime": "2027-01-01T00:00:00.000Z"
}

import delta_sharing

# Load the profile
profile_file = "/path/to/profile.json"

# List all shares accessible with this token
client = delta_sharing.SharingClient(profile_file)
shares = client.list_shares()
for share in shares:
    print(share.name)

# List tables in a share
tables = client.list_all_tables()
for table in tables:
    print(f"{table.share}.{table.schema}.{table.name}")

# Load an entire table as a Pandas DataFrame
df = delta_sharing.load_as_pandas(
    f"{profile_file}#ecommerce_external_share.sales.orders"
)
print(df.shape)
print(df.dtypes)

# Load with predicate hint — server evaluates partition filters before returning file URLs
# Only files matching the predicate are returned; non-partition predicates are still
# applied client-side after download
df_filtered = delta_sharing.load_as_pandas(
    f"{profile_file}#ecommerce_external_share.sales.orders",
    predicateHints=["year = '2026'", "month = '01'"],
    limitHint=500000,
)

# Load as PyArrow Table for downstream Arrow-native processing
import pyarrow
table = delta_sharing.load_as_arrow(
    f"{profile_file}#ecommerce_external_share.sales.orders",
    predicateHints=["region = 'EU'"],
)
print(table.schema)

Reading Shared Data with PySpark

For large tables where Pandas would exhaust memory, the Delta Sharing Spark connector reads shared tables as a Spark DataFrame. The connector handles authentication, pagination of file listings, and parallel Parquet reads across executors. Recipients can run any Spark distribution — EMR, Dataproc, Azure HDInsight, or open-source Spark — without being Databricks customers.

# Install the connector — match the version to your Spark version
# For Spark 3.5:
pip install delta-sharing

# In spark-submit or notebook:
# --packages io.delta:delta-sharing-spark_2.12:3.3.0

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("DeltaSharingReader")
    .config("spark.jars.packages", "io.delta:delta-sharing-spark_2.12:3.3.0")
    .getOrCreate()
)

profile_file = "/dbfs/mnt/profiles/partner-profile.json"

# Load a shared table as a Spark DataFrame
df = (
    spark.read
    .format("deltaSharing")
    .option("responseFormat", "delta")   # use delta format for CDF and versioning support
    .load(f"{profile_file}#ecommerce_external_share.sales.orders")
)

# Standard Spark operations — filter pushdown happens at the connector level
df_eu = df.filter("region = 'EU' AND year = 2026")
df_eu.groupBy("product_category").sum("revenue").show()

# Read a specific version (time travel)
df_v42 = (
    spark.read
    .format("deltaSharing")
    .option("versionAsOf", "42")
    .load(f"{profile_file}#ecommerce_external_share.sales.orders")
)

# Read as of a timestamp
df_ts = (
    spark.read
    .format("deltaSharing")
    .option("timestampAsOf", "2026-06-01T00:00:00Z")
    .load(f"{profile_file}#ecommerce_external_share.sales.orders")
)

Incremental Consumption with Change Data Feed

Delta Sharing supports sharing tables with Change Data Feed (CDF) enabled, allowing recipients to consume incremental changes — inserts, updates, and deletes — rather than re-reading the entire table on each run. This is particularly valuable for synchronizing a shared table into a recipient's own data warehouse or operational database without full snapshot loads.

The provider must have CDF enabled on the source table. Recipients query the table's change stream starting from a version or timestamp. Each change record includes a _change_type column (insert, update_preimage, update_postimage, delete) and a _commit_version column for ordering.

-- Provider: enable CDF on the source table before sharing
ALTER TABLE catalog.sales_schema.orders
  SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

-- Then add to share (CDF capability propagates automatically)
ALTER SHARE ecommerce_external_share
  ADD TABLE catalog.sales_schema.orders
  WITH CHANGE DATA FEED;

# Recipient: read CDF changes starting from version 100
changes_df = (
    spark.read
    .format("deltaSharing")
    .option("readChangeFeed", "true")
    .option("startingVersion", "100")
    .load(f"{profile_file}#ecommerce_external_share.sales.orders")
)

# Filter to only inserts and updates (ignore deletes for an append-only sink)
from pyspark.sql.functions import col
inserts_updates = changes_df.filter(
    col("_change_type").isin(["insert", "update_postimage"])
)

# Incremental pipeline: track last processed version in a checkpoint table
last_version = spark.sql(
    "SELECT COALESCE(MAX(processed_version), 0) FROM my_catalog.checkpoints.orders_cdf"
).collect()[0][0]

new_changes = (
    spark.read
    .format("deltaSharing")
    .option("readChangeFeed", "true")
    .option("startingVersion", str(last_version + 1))
    .load(f"{profile_file}#ecommerce_external_share.sales.orders")
)

# Process and update checkpoint
new_changes.write.format("delta").mode("append").saveAsTable("my_catalog.sink.orders")
max_version = new_changes.agg({"_commit_version": "max"}).collect()[0][0]
spark.sql(f"INSERT INTO my_catalog.checkpoints.orders_cdf VALUES ({max_version})")

Row and Column Filters for Fine-Grained Access

Unity Catalog shares support row filters and column masks — the same mechanisms used for internal workspace access control — applied to data leaving the organization via Delta Sharing. This allows you to share a single physical table with multiple recipients, each seeing a different slice of the data, without maintaining separate materialized views per recipient.

-- Create a row filter function: each recipient sees only their region's data
CREATE FUNCTION catalog.security.region_filter(region_col STRING)
  RETURN region_col = current_user_region();   -- hypothetical session variable

-- Apply row filter to the shared table
ALTER TABLE catalog.sales_schema.orders
  SET ROW FILTER catalog.security.region_filter ON (region);

-- Column masking: recipients see masked PII
CREATE FUNCTION catalog.security.mask_email(email STRING)
  RETURN CASE
    WHEN is_account_group_member('full_data_group') THEN email
    ELSE CONCAT(LEFT(email, 2), '***@', SPLIT(email, '@')[1])
  END;

ALTER TABLE catalog.sales_schema.customers
  ALTER COLUMN email SET MASK catalog.security.mask_email;

Row filters and column masks are enforced server-side by the Delta Sharing endpoint — the server evaluates them before generating presigned URLs, so recipients only receive presigned URLs for files containing rows that pass the filter. For column masks, the server returns Parquet files with the masked column values, not the originals. Recipients cannot bypass these controls by querying the underlying storage directly because they have no storage credentials.

Recipient Management and Token Security

Each recipient has exactly one bearer token at a time. The Databricks activation flow generates a one-time activation link — the recipient visits the URL once to download their profile file containing the token. After activation, the link expires. If a token is compromised, you delete the recipient and recreate it; the new activation link generates a new token, and the old token is immediately invalidated at the server.

-- Create a recipient with IP allowlisting
CREATE RECIPIENT partner_acme_analytics
  IP FILTER ("203.0.113.0/24", "198.51.100.50/32")
  COMMENT 'ACME Corp — only from their office and VPN egress IPs';

-- Rotate a recipient token: delete and recreate
-- This immediately invalidates the old token
DROP RECIPIENT partner_acme_analytics;
CREATE RECIPIENT partner_acme_analytics
  IP FILTER ("203.0.113.0/24", "198.51.100.50/32")
  COMMENT 'ACME Corp — rotated 2026-07-07';

-- List which shares a recipient has access to
SHOW GRANTS TO RECIPIENT partner_acme_analytics;

-- Audit: see all recipients and their grant counts
SELECT recipient_name, created_at, updated_at
FROM system.information_schema.recipients
ORDER BY created_at DESC;

Audit Logging and Usage Monitoring

Databricks logs all Delta Sharing access events to Unity Catalog system tables under system.access.audit. Every recipient query — list shares, list tables, query table data — generates an audit event with the recipient identity, table queried, timestamp, number of files returned, and client IP. This gives data providers visibility into exactly who accessed which tables and when.

-- Query Delta Sharing audit events
SELECT
  event_time,
  user_identity.email AS recipient,
  action_name,
  request_params.share AS share_name,
  request_params."table" AS table_name,
  response.result AS result,
  source_ip_address
FROM system.access.audit
WHERE service_name = 'deltaSharingDataService'
  AND event_time > CURRENT_TIMESTAMP - INTERVAL 7 DAYS
ORDER BY event_time DESC;

-- Volume by recipient — detect unexpected access patterns
SELECT
  user_identity.email AS recipient,
  DATE(event_time) AS access_date,
  COUNT(*) AS query_count,
  COUNT(DISTINCT request_params."table") AS distinct_tables
FROM system.access.audit
WHERE service_name = 'deltaSharingDataService'
  AND event_time > CURRENT_TIMESTAMP - INTERVAL 30 DAYS
GROUP BY 1, 2
ORDER BY 1, 2;

-- Alert: any access outside business hours (UTC)
SELECT *
FROM system.access.audit
WHERE service_name = 'deltaSharingDataService'
  AND HOUR(event_time) NOT BETWEEN 6 AND 22
  AND event_time > CURRENT_TIMESTAMP - INTERVAL 24 HOURS;

Delta Sharing vs. Alternative Data Sharing Approaches

Organizations considering cross-boundary data sharing typically evaluate three approaches. Delta Sharing occupies a specific position: it requires Delta Lake on the provider side, but the recipient can use any tool that speaks the protocol. It is the right choice when you own a Delta Lake and need to expose live data to partners without copying. For organizations without Delta Lake, or sharing across heterogeneous table formats, the tradeoffs shift considerably — as explored in our analysis of AWS Lake Formation's cross-account sharing model, which takes a different approach using IAM-controlled access to S3 without the Delta protocol.

Production Checklist