Pulumi for Data Infrastructure — Type-Safe IaC, Multi-Cloud, and Developer-First Provisioning

Infrastructure in a Language You Already Know

Most infrastructure-as-code tools invent their own configuration language and then spend years bolting loops, conditionals, and functions back onto it. Data platforms suffer the most: a lakehouse, a warehouse, a streaming cluster, and the IAM around them all share structure that a domain-specific language struggles to express without copy-paste.

Pulumi takes the opposite bet. You describe your cloud in a general-purpose programming language, and Pulumi’s engine turns that program into a desired-state plan it reconciles against the real world. Loops are loops, functions are functions, and a reusable database module is just a class.

This guide walks the model end to end: projects and stacks, the type system that makes it safe, packaging with components, where state lives, how secrets stay encrypted, and how to drive the whole thing from your own code with the Automation API.

Projects, Programs, and Stacks

Three nouns carry the whole mental model. A program describes how your infrastructure should be composed. A project is the directory holding that program plus metadata on how to run it. A stack is an isolated, independently configurable instance of the program — dev, staging, and production are three stacks of one project.

Every project has a Pulumi.yaml file. The Pulumi docs require exactly two keys: name and runtime. The runtime can be nodejs, python, go, dotnet, java, yaml, or bun. Optional keys include description, main for a non-default program location, and a backend block with a url.

# Pulumi.yaml — the project file
name: data-platform
runtime: python
description: Cloud data infrastructure for the analytics team
backend:
  url: s3://acme-pulumi-state

Per-stack configuration lives in sibling files named Pulumi.<stack-name>.yaml — for example Pulumi.dev.yaml. You rarely edit these by hand; the pulumi config command writes them for you, keeping the values that differ between environments out of the program itself.

From Empty Directory to First Deploy

pulumi new scaffolds a project from a template, wiring up the Pulumi.yaml, a language SDK, and a first stack. From there the loop is preview, then up.

# Scaffold a Python + AWS project
pulumi new aws-python

# See what would change, without changing anything
pulumi preview

# Create or update the stack's resources
pulumi up

# Tear it all down when you are done
pulumi destroy

pulumi up shows the same diff as preview, then asks for confirmation before it acts. That preview is the safety rail: it is a plan of creates, updates, replacements, and deletes derived by diffing your program against recorded state — the equivalent of reading a change before you approve it.

The Program Is Real Code

By default Pulumi runs a Python project from __main__.py. You import the core pulumi package plus a provider SDK such as pulumi_aws, instantiate resource classes, and export values with pulumi.export.

# __main__.py
import pulumi
import pulumi_aws as aws

# Config values come from Pulumi.<stack>.yaml
config = pulumi.Config()
bucket_prefix = config.get("bucketPrefix") or "analytics"

# A resource: its properties are the desired state
raw_zone = aws.s3.BucketV2(f"{bucket_prefix}-raw")

# Export a stack output for the CLI and other programs
pulumi.export("raw_bucket", raw_zone.id)

The pulumi.Config object reads per-stack settings, so the same program provisions a small bucket in dev and a locked-down one in production without any branching in the source. Configuration is data; the program is logic.

Inputs, Outputs, and Why apply Exists

The one concept that trips up newcomers is the Output. When you read a property like bucket.id, the value may not exist yet — the bucket has not been created. Pulumi models this as an asynchronous Output, a future value that also carries the dependency graph.

You cannot concatenate an Output like a plain string. To transform one, use .apply, which runs a function once the real value resolves and returns a new Output. This is exactly how Pulumi tracks that resource B depends on resource A and orders creation to maximize safe parallelism.

import pulumi
import pulumi_aws as aws

raw_zone = aws.s3.BucketV2("analytics-raw")

# Wrong: raw_zone.id is an Output, not a str
# url = "s3://" + raw_zone.id  # TypeError-ish behavior

# Right: transform the resolved value inside apply
bucket_url = raw_zone.id.apply(lambda name: f"s3://{name}")

# For multiple Outputs, use pulumi.Output.all
arn_and_id = pulumi.Output.all(raw_zone.arn, raw_zone.id).apply(
    lambda vals: {"arn": vals[0], "id": vals[1]}
)

pulumi.export("bucket_url", bucket_url)

Packaging Patterns with ComponentResource

A data platform is never one resource. A “landing zone” is a bucket, a lifecycle policy, an encryption key, and the IAM to reach it. Pulumi lets you bundle these into a single logical unit by extending ComponentResource.

You call super().__init__ with a type token in package:module:Type form, create child resources with ResourceOptions(parent=self) so they nest under the component in the resource tree, and finish with register_outputs.

import pulumi
from pulumi_aws import s3

class LandingZone(pulumi.ComponentResource):
    def __init__(self, name: str, opts: pulumi.ResourceOptions = None):
        # type token identifies this component in the resource tree
        super().__init__("acme:data:LandingZone", name, None, opts)

        self.bucket = s3.BucketV2(
            f"{name}-bucket",
            opts=pulumi.ResourceOptions(parent=self),
        )
        s3.BucketVersioningV2(
            f"{name}-versioning",
            bucket=self.bucket.id,
            versioning_configuration={"status": "Enabled"},
            opts=pulumi.ResourceOptions(parent=self),
        )

        self.bucket_id = self.bucket.id
        self.register_outputs({"bucketId": self.bucket_id})

Consumers then instantiate LandingZone("bronze") like any other resource. This is where a real language pays off: your platform team ships components as ordinary packages, and product teams compose them without touching raw cloud primitives — the same self-service goal we cover in building internal developer platforms.

Where State Lives

Pulumi records the last-known state of your resources so it can diff future runs against reality. The docs describe two classes of backend for that state.

• Pulumi Cloud. The default managed backend — online or self-hosted — that needs no configuration after install and handles concurrency, history, and auditing for you.
• DIY backends.A “do it yourself” object store you manage: AWS S3, Azure Blob, Google Cloud Storage, an S3-compatible server like MinIO or Ceph, PostgreSQL, or the local filesystem.

You choose with pulumi login. The URL scheme selects the backend, and DIY backends keep checkpoint history in a .pulumi/history/ directory.

# Managed Pulumi Cloud (default)
pulumi login

# Self-hosted Pulumi Cloud
pulumi login https://pulumi.acmecorp.com

# DIY backends — the scheme picks the store
pulumi login s3://acme-pulumi-state
pulumi login gs://acme-pulumi-state
pulumi login azblob://acme-pulumi-state

# Local filesystem (shorthand for file://~)
pulumi login --local

An important reassurance from the docs: Pulumi state does not include your cloud credentials. If you run a DIY backend on the same S3 that already holds your object storage, you own the security and auditing that Pulumi Cloud would otherwise provide.

Secrets That Stay Encrypted

Data infrastructure is full of credentials — warehouse passwords, API tokens, connection strings. Pulumi encrypts these rather than storing plaintext. Passing --secret to pulumi config setencrypts the value with the stack’s encryption key and stores only ciphertext in the stack config file.

# Store an encrypted config value
pulumi config set --secret dbPassword S3cr37

# pulumi config list then shows [secret], never the value
pulumi config

Pulumi tracks the transitive usage of that secret and keeps everything it touches encrypted, and it masks the value if a program tries to print it. The encryption provider is configurable. The docs list default, passphrase, awskms, azurekeyvault, gcpkms, and hashivault.

# Pick the secrets provider when creating a stack
pulumi stack init prod \
  --secrets-provider="awskms://1234abcd-12ab-34cd-56ef?region=us-east-1"

# HashiCorp Vault transit engine
pulumi stack init prod --secrets-provider="hashivault://payroll"

# Change provider on an existing stack
pulumi stack change-secrets-provider "gcpkms://projects/p/locations/l/keyRings/r/cryptoKeys/k"

The Automation API — Pulumi Without the CLI

The Automation API is a programmatic interface for running Pulumi operations without invoking the CLI directly. It is stable in TypeScript/JavaScript, Python, Go, C#, and Java, and it still drives the CLI under the hood — so the CLI must be present at runtime.

In Python it ships as pulumi.automation. With an inline program you pass a function that defines resources — no separate Pulumi.yaml — and drive the full lifecycle with up, preview, refresh, and destroy.

import pulumi.automation as auto
import pulumi_aws as aws

def program():
    bucket = aws.s3.BucketV2("automation-raw")
    pulumi.export("bucket", bucket.id)

stack = auto.create_or_select_stack(
    stack_name="dev",
    project_name="data-platform",
    program=program,
)

stack.set_config("aws:region", auto.ConfigValue(value="us-east-1"))
stack.preview(on_output=print)
up_result = stack.up(on_output=print)
print(up_result.outputs["bucket"].value)

This is what turns Pulumi into a building block. The docs highlight CI/CD pipelines, integration testing, blue-green deployments, database migrations bundled with app code, custom internal CLIs, and exposing provisioning behind a REST or gRPC API. When infrastructure is part of a larger workflow rather than a manual step, the Automation API is the seam.

How It Compares to DSL-Based Tools

The honest trade-off is not “better” — it is different shape. A configuration-language tool constrains what you can express, and that constraint is sometimes a feature: fewer ways to be clever, easier to review at a glance.

Pulumi gives you the full expressiveness of a programming language, which means real abstractions, real testing with your normal unit-test framework, and IDE autocomplete over provider schemas — at the cost of the discipline any codebase demands. If you already know Terraform modules and remote state, the concepts map cleanly; the difference is that loops and helpers are language features rather than DSL constructs.

It is also worth naming the alternative philosophy. Tools like Crossplane model infrastructure as Kubernetes objects reconciled by an in-cluster control loop, correcting drift continuously. Pulumi’s model is imperative-at-authoring, desired-state-at-apply: it converges when you run it, not on a timer. Which fits depends on whether you want a control plane always watching or a deployment you trigger deliberately.

A Production Checklist

• Pin versions.Provider SDKs version independently of the core engine — lock both in your language’s dependency file so an upgrade is a reviewed change.
• Stack per environment. Keep dev, staging, and production as separate stacks of one project, with differences pushed into Pulumi.<stack>.yaml config, not code branches.
• Preview in CI. Run pulumi preview on every pull request so the plan is part of code review, and gate pulumi up behind approval.
• Encrypt every secret. Never commit plaintext credentials — use config set --secret and a KMS-backed provider your security team controls.
• Componentize. Wrap repeated resource groups in a ComponentResource so product teams consume a reviewed interface, not raw cloud primitives.
• Own your state backend. Pick Pulumi Cloud or a DIY backend deliberately, and back up the object store that holds it.

Pulumi’s appeal is that infrastructure stops being a second-class dialect and becomes ordinary software: versioned, tested, refactored, and reviewed the way everything else in your repository already is. For data teams juggling storage, compute, catalogs, and the IAM that ties them together, expressing all of it in one language — with real abstractions instead of copy-paste — is often the difference between a platform that grows and one that ossifies.

For deploying Pulumi stacks via a GitOps pipeline — where a Git commit to the stack config triggers a preview and apply in CI — see Grafana GitOps with ArgoCD for the ArgoCD patterns that integrate with Pulumi's Automation API.