AWS Lake Formation — Fine-Grained Access Control, Column Masking, and Data Catalog

What is AWS Lake Formation and Why It Exists

Traditional S3-based data lakes rely on IAM bucket policies and S3 prefix grants for access control. This works at the storage level, but it cannot restrict which columns an analyst sees in a Parquet file, which rows a consumer can query, or enforce governance consistently across Athena, Redshift Spectrum, and Glue ETL jobs simultaneously. Each engine has its own access model and there is no central audit trail of who queried which columns.

AWS Lake Formation sits on top of the AWS Glue Data Catalog and intercepts metadata requests made by query engines. When Athena, Redshift Spectrum, or a Glue ETL job queries a Lake Formation-registered table, Lake Formation enforces column-level visibility and injects row-filter WHERE clauses before returning credentials to the query engine. The result is consistent governance across all engines from a single control plane. If your workloads run on Databricks rather than S3 and Athena, Databricks Unity Catalog provides equivalent fine-grained governance with column masking and row filters enforced across Databricks SQL, Spark, and ML — Lake Formation is the native choice when your query layer is Athena and your storage is S3.

Architecture — Catalog, Location Registry, and the Permission Model

Lake Formation builds a governance layer on top of three existing AWS services. Understanding how they interact explains why Lake Formation permissions are independent of — and override — S3 bucket policies for registered locations.

Initial Setup — Enabling Lake Formation and Registering Locations

Before Lake Formation can enforce permissions, you register S3 locations with a dedicated service role and designate Lake Formation admins. The Terraform configuration below establishes the minimum viable governance foundation — service role, admin designation, and S3 location registration.

# terraform/lakeformation-foundation.tf

# Service role that Lake Formation uses to access registered S3 locations
resource "aws_iam_role" "lakeformation_service" {
  name = "LakeFormationServiceRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lakeformation.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "lakeformation_s3_access" {
  name = "LakeFormationS3DataLakeAccess"
  role = aws_iam_role.lakeformation_service.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:ListBucket"
        ]
        Resource = [
          "arn:aws:s3:::my-datalake-prod",
          "arn:aws:s3:::my-datalake-prod/*"
        ]
      }
    ]
  })
}

# Designate Lake Formation admins
# Only these principals can grant permissions to others
resource "aws_lakeformation_data_lake_settings" "main" {
  admins = [
    "arn:aws:iam::123456789012:role/DataPlatformAdminRole",
    "arn:aws:iam::123456789012:user/terraform-deployer",
  ]

  # Allow creating databases without explicit Lake Formation grants
  # Set to false in strict governance mode
  create_database_default_permissions {
    permissions = []
  }
  create_table_default_permissions {
    permissions = []
  }
}

# Register the S3 data lake location with the service role
resource "aws_lakeformation_resource" "datalake_bucket" {
  arn      = "arn:aws:s3:::my-datalake-prod"
  role_arn = aws_iam_role.lakeformation_service.arn
}

# Grant database-level access (required before table-level grants)
resource "aws_lakeformation_permissions" "analyst_describe_sales_db" {
  principal = "arn:aws:iam::123456789012:role/DataAnalystRole"

  database {
    name = "sales_db"
  }

  permissions             = ["DESCRIBE"]
  permissions_with_grant_option = []
}

LF-Tags — Attribute-Based Access Control at Scale

Direct resource grants — granting a role access to a specific table by name — require a new grant every time a table is added. With hundreds of tables across dozens of domains, this becomes unmanageable. LF-Tags solve this with attribute-based access control (ABAC): tag tables with key-value attributes like data_classification=confidential and domain=finance, then grant roles access to all tables matching a tag expression. LF-Tags pair naturally with data mesh domain ownership models — each domain team tags their own tables, and platform-level LF-Tag policies enforce cross-domain access boundaries without central per-table permission management.

# boto3 — Create and assign LF-Tags

import boto3

lf = boto3.client("lakeformation", region_name="us-east-1")

# Create classification tag with allowed values
lf.create_lf_tag(
    TagKey="data_classification",
    TagValues=["public", "internal", "confidential", "restricted"],
)

# Create domain tag
lf.create_lf_tag(
    TagKey="domain",
    TagValues=["sales", "marketing", "finance", "hr", "engineering"],
)

# Create PII indicator tag (used at column level)
lf.create_lf_tag(
    TagKey="contains_pii",
    TagValues=["true", "false"],
)

# Assign tags to a table
lf.add_lf_tags_to_resource(
    Resource={
        "Table": {
            "DatabaseName": "sales_db",
            "Name": "customer_orders",
        }
    },
    LFTags=[
        {"TagKey": "data_classification", "TagValues": ["confidential"]},
        {"TagKey": "domain", "TagValues": ["sales"]},
    ],
)

# Assign column-level PII tag to sensitive columns
lf.add_lf_tags_to_resource(
    Resource={
        "TableWithColumns": {
            "DatabaseName": "sales_db",
            "Name": "customer_orders",
            "ColumnNames": ["customer_email", "customer_phone", "billing_address"],
        }
    },
    LFTags=[
        {"TagKey": "contains_pii", "TagValues": ["true"]},
    ],
)

# Grant access via LF-Tag policy — works for ALL matching tables,
# including tables created in the future
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/DataAnalystRole"
    },
    Resource={
        "LFTagPolicy": {
            "ResourceType": "TABLE",
            "Expression": [
                {"TagKey": "data_classification", "TagValues": ["public", "internal"]},
                {"TagKey": "domain", "TagValues": ["sales", "marketing"]},
            ],
        }
    },
    Permissions=["SELECT", "DESCRIBE"],
    PermissionsWithGrantOption=[],
)

# Senior analysts access confidential finance data
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/SeniorAnalystRole"
    },
    Resource={
        "LFTagPolicy": {
            "ResourceType": "TABLE",
            "Expression": [
                {"TagKey": "data_classification", "TagValues": ["confidential"]},
                {"TagKey": "domain", "TagValues": ["finance"]},
            ],
        }
    },
    Permissions=["SELECT", "DESCRIBE"],
    PermissionsWithGrantOption=[],
)

Column-Level Security — Restricting Which Columns Are Visible

Column-level security lets you grant SELECT on a table while hiding specific columns. This is the standard pattern for PII protection: data engineers and senior analysts get all columns, while BI tool service accounts and external analysts only see non-sensitive fields. Lake Formation enforces this at the metadata layer — excluded columns do not appear in DESCRIBE TABLE output and any Athena query referencing them fails with a column-not-found error.

# Grant SELECT with excluded PII columns — BI tool service account
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/BiToolServiceRole"
    },
    Resource={
        "TableWithColumns": {
            "DatabaseName": "sales_db",
            "Name": "customer_orders",
            "ColumnWildcard": {
                # All columns EXCEPT these are visible to the principal
                "ExcludedColumnNames": [
                    "customer_email",
                    "customer_phone",
                    "billing_address",
                    "credit_card_last4",
                    "ssn_hash",
                ]
            },
        }
    },
    Permissions=["SELECT"],
    PermissionsWithGrantOption=[],
)

# Grant full column access to data engineers
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/DataEngineerRole"
    },
    Resource={
        "TableWithColumns": {
            "DatabaseName": "sales_db",
            "Name": "customer_orders",
            "ColumnWildcard": {},  # All columns — no exclusions
        }
    },
    Permissions=["SELECT", "INSERT", "DELETE", "ALTER"],
    PermissionsWithGrantOption=[],
)

# Alternatively — grant access to a specific list of allowed columns
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/ExternalPartnerRole"
    },
    Resource={
        "TableWithColumns": {
            "DatabaseName": "sales_db",
            "Name": "customer_orders",
            # Explicit allowlist — only these columns are visible
            "ColumnNames": [
                "order_id",
                "order_date",
                "product_id",
                "quantity",
                "amount",
                "region",
            ],
        }
    },
    Permissions=["SELECT"],
    PermissionsWithGrantOption=[],
)

Row-Level Security — Data Filters and SQL Expressions

Data filters apply SQL WHERE expressions transparently to every query a principal runs against a table. The analyst writes a normal SELECT * FROM customer_orders — Lake Formation injects the filter before returning credentials to Athena. The analyst never sees the filter expression and cannot bypass it by reformulating the query. Apache Iceberg tables registered in the Glue Data Catalog work with Lake Formation data filters — row-level filtering operates on the Iceberg metadata layer, not the Parquet files directly, which preserves Iceberg partition pruning for performance.

# Create a row-level data filter — US region only
lf.create_data_cells_filter(
    TableData={
        "TableCatalogId": "123456789012",
        "DatabaseName": "sales_db",
        "TableName": "customer_orders",
        "Name": "us_region_only",
        "RowFilter": {
            # Standard SQL WHERE expression — injected into every query
            "FilterExpression": "region = 'US'",
        },
        # Also restrict to non-PII columns through the same filter
        "ColumnWildcard": {
            "ExcludedColumnNames": ["customer_email", "customer_phone"],
        },
    }
)

# Create filter for active orders — hide cancelled/draft rows
lf.create_data_cells_filter(
    TableData={
        "TableCatalogId": "123456789012",
        "DatabaseName": "sales_db",
        "TableName": "customer_orders",
        "Name": "active_orders_only",
        "RowFilter": {
            "FilterExpression": "status IN ('completed', 'processing', 'shipped')",
        },
        "ColumnWildcard": {},  # All columns through this filter
    }
)

# Multi-tenant filter — each tenant only sees their own data
# Create one filter per tenant (or use dynamic SQL via parameter)
lf.create_data_cells_filter(
    TableData={
        "TableCatalogId": "123456789012",
        "DatabaseName": "saas_db",
        "TableName": "events",
        "Name": "tenant_acme_only",
        "RowFilter": {
            "FilterExpression": "tenant_id = 'acme'",
        },
        "ColumnWildcard": {},
    }
)

# Grant the data filter to a principal
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/USRegionAnalystRole"
    },
    Resource={
        "DataCellsFilter": {
            "TableCatalogId": "123456789012",
            "DatabaseName": "sales_db",
            "TableName": "customer_orders",
            "Name": "us_region_only",
        }
    },
    Permissions=["SELECT"],
    PermissionsWithGrantOption=[],
)

Column Masking — Protecting PII Without Removing Columns

Sometimes downstream consumers need to see that a column exists and has data, but the raw values must be protected. A common example is providing a masked email for display purposes while hiding the actual address, or computing aggregates on a salary column without exposing individual values. Lake Formation supports three masking approaches implemented as data filter patterns combined with Athena views.

-- Pattern 1: Athena view-based masking — hash the email, null the SSN
-- Create a masking view on top of the raw table

CREATE OR REPLACE VIEW sales_db.customer_orders_masked AS
SELECT
    order_id,
    customer_id,
    order_date,
    amount,
    region,
    product_id,
    status,
    -- MASK_HASH: replace with deterministic hash (join-able but not reversible)
    TO_HEX(SHA256(TO_UTF8(customer_email)))       AS customer_email,
    -- MASK_PARTIAL: show last 4 digits, mask the rest
    REGEXP_REPLACE(customer_phone, '\d(?=\d{4})', 'X') AS customer_phone,
    -- MASK_NULL: replace sensitive field with NULL entirely
    NULL                                           AS billing_address,
    NULL                                           AS ssn_hash
FROM sales_db.customer_orders;

-- Grant the view (not the raw table) to external partners
-- In Lake Formation, grant SELECT on the view's underlying table
-- via a column-level grant that excludes raw PII, then let Athena
-- apply the masking expressions through the view definition

# Pattern 2: Data filter with column subset + Athena view for masking logic
# Lake Formation handles row/column restriction; the view handles value masking

# Step 1 — create a filter that exposes only non-PII columns from the raw table
lf.create_data_cells_filter(
    TableData={
        "TableCatalogId": "123456789012",
        "DatabaseName": "sales_db",
        "TableName": "customers",
        "Name": "pii_columns_excluded",
        "RowFilter": {"AllRowsWildcard": {}},
        "ColumnWildcard": {
            "ExcludedColumnNames": ["ssn", "credit_card_number", "date_of_birth"],
        },
    }
)

# Step 2 — grant the filter to analyst role
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/AnalystRole"
    },
    Resource={
        "DataCellsFilter": {
            "TableCatalogId": "123456789012",
            "DatabaseName": "sales_db",
            "TableName": "customers",
            "Name": "pii_columns_excluded",
        }
    },
    Permissions=["SELECT"],
    PermissionsWithGrantOption=[],
)

# Step 3 — for the email column (keep visible but mask value),
# create a separate masked_email Glue column via a Glue catalog update
# and use a Glue ETL job or Athena view to populate it
# The analyst queries the view; raw table access requires DataEngineerRole

Cross-Account Data Sharing — RAM and Lake Formation Combined

Cross-account Lake Formation sharing requires two steps that are easy to miss: sharing the Glue Catalog database via AWS Resource Access Manager (RAM) so the consumer account can see the catalog entry, and granting Lake Formation permissions to the consumer account ID so they can actually query data. Missing either step results in the consumer seeing the database in their catalog but getting AccessDeniedException on any table query.

# boto3 — Producer account: share a database cross-account

import boto3

lf_producer = boto3.client("lakeformation", region_name="us-east-1")
ram_producer = boto3.client("ram", region_name="us-east-1")

CONSUMER_ACCOUNT_ID = "987654321098"
PRODUCER_ACCOUNT_ID = "123456789012"

# Step 1: Grant Lake Formation permissions to the consumer account ID
lf_producer.grant_permissions(
    Principal={"DataLakePrincipalIdentifier": CONSUMER_ACCOUNT_ID},
    Resource={"Database": {"Name": "analytics_db"}},
    Permissions=["DESCRIBE"],
    PermissionsWithGrantOption=["DESCRIBE"],
)

lf_producer.grant_permissions(
    Principal={"DataLakePrincipalIdentifier": CONSUMER_ACCOUNT_ID},
    Resource={
        "Table": {
            "DatabaseName": "analytics_db",
            "TableWildcard": {},  # All current and future tables
        }
    },
    Permissions=["SELECT", "DESCRIBE"],
    PermissionsWithGrantOption=["SELECT", "DESCRIBE"],
)

# Step 2: Share the Glue Catalog database resource via RAM
ram_producer.create_resource_share(
    name="analytics-db-consumer-share",
    resourceArns=[
        f"arn:aws:glue:us-east-1:{PRODUCER_ACCOUNT_ID}:database/analytics_db"
    ],
    principals=[CONSUMER_ACCOUNT_ID],
    allowExternalPrincipals=False,  # Keep within AWS Organization
)

# boto3 — Consumer account: accept invitation and query

import boto3

ram_consumer = boto3.client("ram", region_name="us-east-1")
glue_consumer = boto3.client("glue", region_name="us-east-1")
lf_consumer = boto3.client("lakeformation", region_name="us-east-1")

# Step 3: Accept the RAM resource share invitation
invitations = ram_consumer.get_resource_share_invitations(
    resourceShareArns=[]
)["resourceShareInvitations"]
for invite in invitations:
    if invite["status"] == "PENDING":
        ram_consumer.accept_resource_share_invitation(
            resourceShareInvitationArn=invite["resourceShareInvitationArn"]
        )

# Step 4: Create a local Glue database pointing at the producer catalog
glue_consumer.create_database(
    DatabaseInput={
        "Name": "shared_analytics_db",
        "TargetDatabase": {
            "CatalogId": "123456789012",  # Producer account
            "DatabaseName": "analytics_db",
        },
    }
)

# Step 5: Grant the shared data to local IAM roles in consumer account
lf_consumer.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::987654321098:role/ConsumerAnalystRole"
    },
    Resource={
        "Table": {
            "CatalogId": "123456789012",  # Producer catalog
            "DatabaseName": "analytics_db",
            "TableWildcard": {},
        }
    },
    Permissions=["SELECT", "DESCRIBE"],
    PermissionsWithGrantOption=[],
)

# Consumer analyst can now query in Athena:
# SELECT * FROM shared_analytics_db.revenue_summary WHERE quarter = '2026-Q1'

Integration with Athena, Redshift Spectrum, and Glue ETL

Lake Formation permissions are enforced transparently by all three query engines. Athena requires no special configuration — it calls Lake Formation automatically for any table in a registered location. Glue ETL jobs need an explicit SUPER or table-level grant to the Glue service role. Redshift Spectrum requires an external schema pointing at the Glue catalog with create external schema ... from data catalog and the Redshift IAM role must be granted Lake Formation permissions.

-- Athena — Lake Formation enforcement is automatic
-- Query engines do NOT need special configuration

-- This SELECT respects column exclusions and row filters automatically
SELECT
    order_id,
    customer_id,
    amount,
    region,
    order_date
FROM sales_db.customer_orders
WHERE order_date >= DATE '2026-01-01'
  AND status = 'completed'
ORDER BY amount DESC
LIMIT 1000;

-- Attempting to access excluded columns fails with:
-- COLUMN_NOT_FOUND: Column 'customer_email' cannot be resolved
-- SELECT customer_email FROM sales_db.customer_orders;

-- Verify which columns are visible to the current role
DESCRIBE sales_db.customer_orders;
-- Hidden columns do not appear in DESCRIBE output

# Grant Glue ETL job role access — required separately from Athena access

# The Glue service role needs Lake Formation table permissions
# It does NOT inherit permissions from the account admin

lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/GlueETLRole"
    },
    Resource={
        "Table": {
            "DatabaseName": "sales_db",
            "Name": "customer_orders",
        }
    },
    Permissions=["SELECT", "INSERT", "DELETE"],
    PermissionsWithGrantOption=[],
)

# Also grant the Glue role access to the output database/table
lf.grant_permissions(
    Principal={
        "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/GlueETLRole"
    },
    Resource={
        "Database": {"Name": "curated_db"},
    },
    Permissions=["CREATE_TABLE", "DESCRIBE"],
    PermissionsWithGrantOption=[],
)

-- Redshift Spectrum — create external schema backed by Glue/Lake Formation

-- The Redshift cluster's IAM role must have Lake Formation SELECT grants
-- and AmazonAthenaFullAccess + AWSGlueConsoleFullAccess policies

CREATE EXTERNAL SCHEMA lakeformation_sales
FROM DATA CATALOG
DATABASE 'sales_db'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftSpectrumRole'
REGION 'us-east-1';

-- After creation, query the schema normally
-- Lake Formation enforces column and row filters identically to Athena
SELECT
    product_id,
    SUM(amount) AS total_revenue,
    COUNT(*)    AS order_count
FROM lakeformation_sales.customer_orders
WHERE order_date BETWEEN '2026-01-01' AND '2026-03-31'
GROUP BY product_id
ORDER BY total_revenue DESC;

Audit Logging — CloudTrail and CloudWatch Logs Insights

Lake Formation logs all permission grants, revocations, and GetTemporaryGlueTableCredentials calls (one per Athena or Glue query) to CloudTrail. Enabling CloudTrail data events for the Lake Formation API gives you a per-query audit trail of which principal accessed which table and columns — required for SOC 2 and GDPR compliance audits.

# CloudWatch Logs Insights — Lake Formation access audit queries

# Find all Lake Formation AccessDeniedException errors in the last 7 days
# Log group: CloudTrail log group for the account

fields @timestamp, userIdentity.sessionContext.sessionIssuer.arn,
       requestParameters, errorCode, errorMessage
| filter eventSource = "lakeformation.amazonaws.com"
| filter errorCode = "AccessDeniedException"
| sort @timestamp desc
| limit 200

# Track table access frequency per principal (who queries what)
fields @timestamp, userIdentity.sessionContext.sessionIssuer.arn,
       requestParameters.tableArn
| filter eventSource = "lakeformation.amazonaws.com"
| filter eventName = "GetTemporaryGlueTableCredentials"
| stats count() as access_count by
    userIdentity.sessionContext.sessionIssuer.arn,
    requestParameters.tableArn
| sort access_count desc

# Alert on cross-account permission grants — potential data exfiltration
fields @timestamp, userIdentity.arn, requestParameters
| filter eventSource = "lakeformation.amazonaws.com"
| filter eventName IN ["GrantPermissions", "BatchGrantPermissions"]
| filter requestParameters.principal.dataLakePrincipalIdentifier not like "arn:aws:iam::123456789012"
| sort @timestamp desc

# Monitor LF-Tag assignment changes — catch unauthorized tag modifications
fields @timestamp, userIdentity.arn, eventName,
       requestParameters.lFTags, requestParameters.resource
| filter eventSource = "lakeformation.amazonaws.com"
| filter eventName IN ["AddLFTagsToResource", "RemoveLFTagsFromResource",
                        "CreateLFTag", "DeleteLFTag", "UpdateLFTag"]
| sort @timestamp desc

Production Checklist