Feast Feature Store — Real-Time and Batch Feature Serving for Production ML Systems

The Training-Serving Skew Problem — Why Feature Stores Exist

Machine learning models fail in production for many reasons, but the most insidious and hardest to diagnose is training-serving skew: the features computed during training differ — subtly or dramatically — from the features computed at inference time. An e-commerce recommendation model trained on "user's purchase count in the last 30 days" that uses a different time window, different aggregation logic, or a stale cache at serving time will quietly degrade in accuracy without raising an obvious error.

A feature store solves this by providing a single source of truth for features: the same feature definitions, transformations, and storage layer are used for both training data generation and online inference. Feast (Feature Store) is the leading open-source feature store, originally developed at Gojek and now a CNCF sandbox project. It provides a registry for feature definitions, connectors for offline stores (S3, BigQuery, Redshift, Snowflake), and a serving layer for online stores (Redis, DynamoDB, Cassandra) — all driven from a version-controlled Python feature repository. For production ML pipeline workflows, Feast integrates naturally with MLflow for experiment tracking and model registry.

Feast Architecture — Registry, Offline Store, Online Store, and Feature Server

Feast has four core components that work together across the ML lifecycle:

• Feature Registry — a metadata store (SQLite for local development, SQL database for production) that tracks entity definitions, feature views, feature services, and data sources. The registry is the authoritative source for feature schemas and is updated by running feast apply.
• Offline Store — a columnar data warehouse or data lake (Parquet files on S3, BigQuery, Redshift, Snowflake, DuckDB) that holds historical feature values. Used to generate point-in-time correct training datasets with get_historical_features().
• Online Store — a low-latency key-value store (Redis, DynamoDB, Bigtable, SQLite) populated by materialization jobs. Serves the latest feature values in single-digit milliseconds via get_online_features().
• Feature Server — an optional HTTP/gRPC service that exposes the online store via a REST API, so non-Python inference services (Java, Go, Node.js) can retrieve features without the Python SDK.

Installation and Feature Repository Structure

Feast is a Python package. Install it with the extras for your chosen online and offline stores. The feature repository is a plain directory of Python files — no database servers required for local development (Feast uses SQLite and Parquet by default).

# Install Feast with Redis (online) and S3/Parquet (offline) support
pip install feast[redis]

# For BigQuery offline store
pip install feast[gcp]

# For AWS Redshift + DynamoDB
pip install feast[aws]

# Initialize a new feature repository
feast init my_feature_repo
cd my_feature_repo

# Directory structure after init:
# my_feature_repo/
# ├── feature_store.yaml   ← store configuration
# ├── example_repo.py      ← sample feature definitions
# └── data/
#     └── driver_stats.parquet  ← sample offline data

# feature_store.yaml — production configuration with Redis online store
project: my_ml_platform
registry: s3://my-feast-bucket/registry/registry.db
provider: aws          # cloud provider hint for default connectors

offline_store:
  type: file            # Parquet files; swap for bigquery, redshift, snowflake
  # For BigQuery:
  # type: bigquery
  # project: my-gcp-project
  # dataset: feast_offline

online_store:
  type: redis
  connection_string: "redis://redis-host:6379"
  # For Redis Cluster:
  # connection_string: "redis+cluster://node1:6379,node2:6379,node3:6379"
  # key_ttl_seconds: 86400   # optional TTL to auto-expire stale features

entity_key_serialization_version: 2

Defining Entities, Data Sources, and Feature Views

The three building blocks of every Feast feature repository are entities (the primary keys your features describe — a user, a product, a driver), data sources (where the raw feature data lives), and feature views (named groups of features computed from a data source and keyed by an entity).

# features/user_features.py — complete feature view definition

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Float64, Int64, String, Bool
import pandas as pd

# ── Entity: the primary key ───────────────────────────────────────────────────
user = Entity(
    name="user_id",
    description="Unique user identifier",
    tags={"team": "ml-platform", "domain": "users"},
)

# ── Data Source: where historical feature data lives ─────────────────────────
# Local Parquet file (dev) — swap for BigQuerySource or RedshiftSource in prod
user_stats_source = FileSource(
    name="user_stats_source",
    path="data/user_stats.parquet",
    timestamp_field="event_timestamp",   # required for point-in-time joins
    created_timestamp_column="created",  # dedup: latest row wins per timestamp
)

# ── BigQuery source (production) ──────────────────────────────────────────────
# from feast import BigQuerySource
# user_stats_source = BigQuerySource(
#     name="user_stats_bq",
#     table="my-project.feast_offline.user_stats",
#     timestamp_field="event_timestamp",
# )

# ── Feature View: named feature group with TTL ────────────────────────────────
user_stats_fv = FeatureView(
    name="user_stats",
    entities=[user],
    ttl=timedelta(days=7),       # features older than 7d are excluded from online store
    schema=[
        Field(name="purchase_count_7d",    dtype=Int64,   description="Purchases in last 7 days"),
        Field(name="purchase_count_30d",   dtype=Int64,   description="Purchases in last 30 days"),
        Field(name="avg_order_value_30d",  dtype=Float64, description="Avg order value, 30 days"),
        Field(name="days_since_last_order",dtype=Float32, description="Days since most recent order"),
        Field(name="preferred_category",   dtype=String,  description="Most purchased category"),
        Field(name="is_premium",           dtype=Bool,    description="Premium subscription flag"),
        Field(name="lifetime_value",       dtype=Float64, description="Total LTV"),
    ],
    source=user_stats_source,
    tags={"owner": "ml-team", "version": "v2"},
)

# ── Feature Service: a named bundle for a specific model ─────────────────────
# Groups features used by a particular model version — versioned independently
from feast import FeatureService

recommendation_features = FeatureService(
    name="recommendation_model_v3",
    features=[
        user_stats_fv[["purchase_count_7d", "purchase_count_30d", "avg_order_value_30d",
                        "days_since_last_order", "preferred_category", "is_premium"]],
    ],
    tags={"model": "recommendation", "version": "v3"},
)

# ── On-Demand Feature View: derived features computed at retrieval time ───────
# Useful for features that combine retrieved values with request-time context
from feast import RequestSource, on_demand_feature_view

# Input from the request (passed at retrieval time, not stored)
request_source = RequestSource(
    name="request_context",
    schema=[
        Field(name="current_cart_value", dtype=Float64),
    ],
)

@on_demand_feature_view(
    sources=[user_stats_fv, request_source],
    schema=[
        Field(name="cart_to_avg_ratio", dtype=Float64),
        Field(name="is_high_value_session", dtype=Bool),
    ],
)
def user_session_features(inputs: pd.DataFrame) -> pd.DataFrame:
    df = pd.DataFrame()
    df["cart_to_avg_ratio"] = (
        inputs["current_cart_value"] / inputs["avg_order_value_30d"].clip(lower=1.0)
    )
    df["is_high_value_session"] = inputs["current_cart_value"] > inputs["lifetime_value"] * 0.3
    return df

feast apply — Registering Features and Planning Changes

feast apply reads all Python files in the repository, validates them, and updates the registry. It creates or updates tables in the online store (Redis key prefixes, DynamoDB tables) and registers all entities, feature views, and feature services. This is your infrastructure-as-code step — run it in CI before any materialization job. Feast's provider model also handles infrastructure provisioning for cloud stores.

# ── feast plan: dry-run, shows what will change ──────────────────────────────
feast plan

# Output (example):
# Created feature views: [user_stats]
# Created entities:      [user_id]
# Created feature services: [recommendation_model_v3]

# ── feast apply: apply changes to the registry and online store ───────────────
feast apply

# ── Verify what was registered ────────────────────────────────────────────────
feast feature-views list
# NAME         ENTITIES    TTL
# user_stats   [user_id]   7 days

feast entities list
# NAME       DESCRIPTION            JOIN KEYS
# user_id    Unique user identifier  [user_id]

feast feature-services list
# NAME                      FEATURES
# recommendation_model_v3   user_stats:purchase_count_7d, ...

# ── Programmatic registry access ─────────────────────────────────────────────
from feast import FeatureStore

store = FeatureStore(repo_path=".")

# List all feature views
for fv in store.list_feature_views():
    print(f"{fv.name}: {[f.name for f in fv.features]}")

# Get a specific feature view
fv = store.get_feature_view("user_stats")
print(f"TTL: {fv.ttl}, Source: {fv.source.name}")

Point-in-Time Correct Training Datasets with get_historical_features()

The most critical correctness guarantee a feature store provides is point-in-time joins. When generating a training dataset, each training example should use the feature values that were available at the time the label was observed— not values computed later. Without this guarantee, a model trained on "days_since_last_order" might accidentally use a value computed weeks after the label event, introducing future leakage that makes offline metrics look better than they are. Feast's get_historical_features() performs a point-in-time correct join using the event_timestamp column in your entity DataFrame.

import pandas as pd
from datetime import datetime, timezone
from feast import FeatureStore

store = FeatureStore(repo_path=".")

# ── Entity DataFrame: who + when to retrieve features for ────────────────────
# This comes from your labels dataset — one row per training example
entity_df = pd.DataFrame({
    "user_id": [1001, 1002, 1003, 1004, 1001],
    # Timestamps tell Feast: "give me the feature values as of THIS moment"
    # Point-in-time join uses the latest feature row at or before this timestamp
    "event_timestamp": [
        datetime(2026, 5, 1, 10, 0, tzinfo=timezone.utc),
        datetime(2026, 5, 3, 14, 0, tzinfo=timezone.utc),
        datetime(2026, 5, 5, 9,  0, tzinfo=timezone.utc),
        datetime(2026, 5, 7, 16, 0, tzinfo=timezone.utc),
        datetime(2026, 5, 10, 8, 0, tzinfo=timezone.utc),  # same user, different time
    ],
    "label_converted": [1, 0, 1, 1, 0],   # your ML target
})

# ── Retrieve point-in-time correct features ───────────────────────────────────
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "user_stats:purchase_count_7d",
        "user_stats:purchase_count_30d",
        "user_stats:avg_order_value_30d",
        "user_stats:days_since_last_order",
        "user_stats:preferred_category",
        "user_stats:is_premium",
    ],
).to_df()

# Or use a FeatureService (recommended for production — versioned bundle)
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=store.get_feature_service("recommendation_model_v3"),
).to_df()

print(training_df.columns.tolist())
# ['user_id', 'event_timestamp', 'label_converted',
#  'purchase_count_7d', 'purchase_count_30d', 'avg_order_value_30d',
#  'days_since_last_order', 'preferred_category', 'is_premium']

# ── Convert to Arrow for large-scale training datasets ────────────────────────
# Avoids loading everything into Pandas memory for 100M+ row datasets
job = store.get_historical_features(
    entity_df=entity_df,
    features=store.get_feature_service("recommendation_model_v3"),
)
arrow_table = job.to_arrow()   # PyArrow Table — stream to S3, GCS, or local

# ── Save to Parquet for model training ────────────────────────────────────────
training_df.to_parquet("training_data_2026_05.parquet", index=False)

# ── Train a model using the Feast-generated dataset ──────────────────────────
import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

df = pd.read_parquet("training_data_2026_05.parquet")

# Encode categorical feature
le = LabelEncoder()
df["preferred_category_enc"] = le.fit_transform(df["preferred_category"].fillna("unknown"))

features = ["purchase_count_7d", "purchase_count_30d", "avg_order_value_30d",
            "days_since_last_order", "preferred_category_enc", "is_premium"]
X = df[features]
y = df["label_converted"]

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

model = lgb.LGBMClassifier(n_estimators=200, learning_rate=0.05)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], callbacks=[lgb.early_stopping(20)])

Feature Materialization — Populating the Online Store for Serving

Materialization reads feature data from the offline store and writes the latest values per entity key into the online store. This is how features become available for low-latency inference. Feast provides two materialization commands: materialize (full range, for backfills) and materialize_incremental (from the last successful materialization timestamp, for scheduled jobs). Both are idempotent — re-running them is safe.

# ── CLI: materialize features into the online store ─────────────────────────

# Full backfill: materialize all features from start_date to end_date
feast materialize 2026-01-01T00:00:00 2026-06-07T00:00:00

# Materialize only a specific feature view
feast materialize-incremental 2026-06-07T00:00:00 --views user_stats

# Incremental: automatically materializes from last_updated_timestamp → end_date
feast materialize-incremental 2026-06-07T12:00:00

# ── Python API: programmatic materialization ──────────────────────────────────
from datetime import datetime, timezone, timedelta
from feast import FeatureStore

store = FeatureStore(repo_path=".")

# Incremental materialization — use in scheduled jobs (Airflow, Dagster, cron)
store.materialize_incremental(
    end_date=datetime.now(tz=timezone.utc),
    feature_views=["user_stats"],        # None to materialize all views
)

# Full range — use for backfills or after data corrections
store.materialize(
    start_date=datetime(2026, 6, 1, tzinfo=timezone.utc),
    end_date=datetime(2026, 6, 7, tzinfo=timezone.utc),
)

# ── Check materialization status ──────────────────────────────────────────────
# feast list of feature views shows last materialization times
feast feature-views list --verbose
# NAME         LAST MATERIALIZED
# user_stats   2026-06-07 10:00:00 UTC

# ── Airflow DAG for scheduled materialization ────────────────────────────────
# dags/feast_materialize.py
from airflow.decorators import dag, task
from datetime import datetime, timezone

@dag(schedule="0 * * * *", start_date=datetime(2026, 6, 1), catchup=False)
def feast_materialize():

    @task
    def materialize_user_features():
        from feast import FeatureStore
        store = FeatureStore(repo_path="/opt/feast/feature_repo")
        store.materialize_incremental(
            end_date=datetime.now(tz=timezone.utc),
            feature_views=["user_stats"],
        )
        return "done"

    materialize_user_features()

dag = feast_materialize()

Online Feature Serving — Python SDK and the Feast REST API Server

At inference time, your model service retrieves the latest materialized features from the online store. get_online_features() hits Redis (or DynamoDB) directly in single-digit milliseconds. The Feast Python feature server exposes this as a REST endpoint, letting non-Python services (Java Spring Boot, Go microservices) consume features over HTTP. For agentic data workflows where AI agents need real-time context, the REST server provides a language-agnostic integration point.

# ── Python SDK: get_online_features() ────────────────────────────────────────
from feast import FeatureStore

store = FeatureStore(repo_path=".")

# Retrieve online features for a batch of users (single Redis round-trip per view)
feature_vector = store.get_online_features(
    features=[
        "user_stats:purchase_count_7d",
        "user_stats:purchase_count_30d",
        "user_stats:avg_order_value_30d",
        "user_stats:days_since_last_order",
        "user_stats:preferred_category",
        "user_stats:is_premium",
    ],
    entity_rows=[
        {"user_id": 1001},
        {"user_id": 1002},
    ],
)

# Convert to dict or DataFrame
features_dict = feature_vector.to_dict()
# {"user_id": [1001, 1002],
#  "purchase_count_7d": [12, 3],
#  "purchase_count_30d": [47, 9], ...}

features_df = feature_vector.to_df()

# Or use a FeatureService (recommended — ensures exact same features as training)
feature_vector = store.get_online_features(
    features=store.get_feature_service("recommendation_model_v3"),
    entity_rows=[{"user_id": user_id} for user_id in user_ids],
)

# ── FastAPI inference service with Feast ──────────────────────────────────────
# inference_server.py
from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np
from feast import FeatureStore

app = FastAPI()
store = FeatureStore(repo_path="/opt/feast/feature_repo")
model = joblib.load("/opt/models/recommendation_v3.pkl")

class PredictRequest(BaseModel):
    user_id: int
    current_cart_value: float

@app.post("/predict")
async def predict(req: PredictRequest):
    # 1. Retrieve pre-computed features from Redis (low-latency)
    features = store.get_online_features(
        features=store.get_feature_service("recommendation_model_v3"),
        entity_rows=[{"user_id": req.user_id}],
    ).to_dict()

    # 2. Compute on-demand features (request-time context)
    avg_order = features["avg_order_value_30d"][0] or 1.0
    cart_ratio = req.current_cart_value / avg_order
    is_high_value = req.current_cart_value > (features.get("lifetime_value", [0])[0] or 0) * 0.3

    # 3. Build feature array and score
    X = np.array([[
        features["purchase_count_7d"][0] or 0,
        features["purchase_count_30d"][0] or 0,
        avg_order,
        features["days_since_last_order"][0] or 999,
        cart_ratio,
        int(features["is_premium"][0] or False),
    ]])

    score = float(model.predict_proba(X)[0][1])
    return {"user_id": req.user_id, "conversion_probability": score}

# ── Feast REST API Server (for non-Python services) ───────────────────────────
# Start the server
feast serve --host 0.0.0.0 --port 6566 &

# Retrieve features via HTTP
curl -X POST http://localhost:6566/get-online-features \
  -H "Content-Type: application/json" \
  -d '{
    "feature_service": "recommendation_model_v3",
    "entities": {
      "user_id": [1001, 1002, 1003]
    }
  }'

Feast on Kubernetes — Feature Server Deployment and Scalability

The Feast feature server is a stateless FastAPI application that reads from the online store. Deploy it on Kubernetes for horizontal scalability and high availability. The registry is read-only at serving time — no write operations hit it on the inference path.

# ── Kubernetes deployment for Feast feature server ───────────────────────────
# feast-server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: feast-feature-server
  namespace: ml-platform
  labels:
    app: feast-feature-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: feast-feature-server
  template:
    metadata:
      labels:
        app: feast-feature-server
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "6566"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: feast-server
          image: my-registry.io/feast-server:0.40.0
          command: ["feast", "serve", "--host", "0.0.0.0", "--port", "6566"]
          ports:
            - containerPort: 6566
          env:
            - name: FEAST_REPO_PATH
              value: /feast/feature_repo
            - name: REDIS_HOST
              valueFrom:
                secretKeyRef:
                  name: feast-secrets
                  key: redis-host
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2"
              memory: "2Gi"
          livenessProbe:
            httpGet:
              path: /health
              port: 6566
            initialDelaySeconds: 15
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health
              port: 6566
            initialDelaySeconds: 5
            periodSeconds: 5
          volumeMounts:
            - name: feature-repo
              mountPath: /feast/feature_repo
              readOnly: true
      volumes:
        - name: feature-repo
          configMap:
            name: feast-feature-repo   # ConfigMap with feature_store.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: feast-feature-server
  namespace: ml-platform
spec:
  selector:
    app: feast-feature-server
  ports:
    - port: 80
      targetPort: 6566
  type: ClusterIP

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: feast-feature-server-hpa
  namespace: ml-platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: feast-feature-server
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

# ── Dockerfile for the feature server ────────────────────────────────────────
FROM python:3.11-slim

RUN pip install feast[redis]==0.40.0 gunicorn

COPY feature_repo/ /feast/feature_repo/

WORKDIR /feast
CMD ["feast", "serve", "--host", "0.0.0.0", "--port", "6566"]

Feature Monitoring — Freshness SLAs, Drift Detection, and CI/CD Workflows

Feature quality issues are a leading cause of silent ML model degradation. The key monitoring dimensions are freshness (is the online store being materialized on schedule?), completeness (are there unexpected null or zero values?), and distribution drift (have feature statistics shifted relative to training?). Feast exposes materialization timestamps that you can track in data quality observability systems like Monte Carlo or custom Prometheus metrics.

# ── Feature freshness monitoring ─────────────────────────────────────────────
# monitoring/feature_freshness.py
from datetime import datetime, timezone, timedelta
from feast import FeatureStore
from prometheus_client import Gauge, start_http_server

FEATURE_FRESHNESS_SECONDS = Gauge(
    "feast_feature_freshness_seconds",
    "Seconds since last materialization",
    ["feature_view"],
)

def check_freshness(store: FeatureStore):
    for fv in store.list_feature_views():
        last_updated = fv.materialization_intervals
        if last_updated:
            age = (datetime.now(tz=timezone.utc) - last_updated[-1][1]).total_seconds()
        else:
            age = float("inf")
        FEATURE_FRESHNESS_SECONDS.labels(feature_view=fv.name).set(age)

# ── Prometheus alert: stale features ─────────────────────────────────────────
# prometheus/alerts.yaml
groups:
  - name: feast.rules
    rules:
      - alert: FeastFeatureStoreFreshnessBreach
        expr: feast_feature_freshness_seconds > 3600   # > 1h stale
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Feature view {{ $labels.feature_view }} not materialized in > 1h"

      - alert: FeastFeatureServerHighLatency
        expr: histogram_quantile(0.99, feast_feature_server_request_duration_seconds_bucket) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Feast feature server P99 latency > 50ms"

# ── CI/CD pipeline for feature repository changes ────────────────────────────
# .github/workflows/feast-ci.yml
name: Feast Feature Repository CI

on:
  pull_request:
    paths: ['feature_repo/**']
  push:
    branches: [main]
    paths: ['feature_repo/**']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install Feast
        run: pip install feast[redis]==0.40.0

      - name: Validate feature repository
        run: |
          cd feature_repo
          feast plan          # dry-run, exits 1 on validation errors

      - name: Run unit tests
        run: pytest tests/test_feature_views.py -v

  apply:
    needs: validate
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Apply to staging registry
        env:
          FEAST_REGISTRY_PATH: ${{ secrets.FEAST_STAGING_REGISTRY }}
          REDIS_HOST: ${{ secrets.STAGING_REDIS_HOST }}
        run: feast apply

# ── Feature view unit test ────────────────────────────────────────────────────
# tests/test_feature_views.py
import pytest
import pandas as pd
from datetime import datetime, timezone
from feast import FeatureStore

def test_user_stats_feature_view():
    store = FeatureStore(repo_path=".")
    fv = store.get_feature_view("user_stats")
    feature_names = [f.name for f in fv.features]
    assert "purchase_count_7d" in feature_names
    assert "avg_order_value_30d" in feature_names
    assert fv.ttl.days == 7

def test_point_in_time_join():
    store = FeatureStore(repo_path=".")
    entity_df = pd.DataFrame({
        "user_id": [1001],
        "event_timestamp": [datetime(2026, 6, 1, tzinfo=timezone.utc)],
    })
    result = store.get_historical_features(
        entity_df=entity_df,
        features=["user_stats:purchase_count_7d"],
    ).to_df()
    assert "purchase_count_7d" in result.columns
    assert len(result) == 1