Skip to content

Product engineering case study

Building a Cloud-Native Job Orchestration Platform

How a 4-person engineering team built a serverless system that scaled to hundreds of thousands of executions — and is still running today.

Mark Danielson

The Challenge

Data at scale demands more than spreadsheets and support tickets

Fast-growing health plan

Bright Health expanding nationally across multiple markets

Single point of failure

All data operations funneled through one third-party provider acting as broker

Hundreds of data partners

Vendors, provider groups, government agencies, and HIEs

Multiple formats and locations

APIs, flat files, and encrypted transfers across varied systems

Manual processes at breaking point

Email-based workflows that couldn't scale with the business

Bright Health Health Plan 3rd Party Provider Data Broker Providers Government Vendors HIEs Manual processes Tickets & emails Days of delay

Build vs. Buy

A disciplined path from evaluation to commitment

Evaluate

Assessed off-the-shelf enterprise job schedulers and ETL tools against multi-location, multi-format requirements

Try ADF

Started pragmatically with Azure Data Factory for initial data movement and orchestration needs

Outgrow ADF

Rapidly exceeded ADF capabilities as partner count, format variety, and iteration speed demanded more flexibility

Design Custom

Architected a serverless orchestration platform on Azure Functions with modular capability layers

Microsoft Validation

Validated architecture design with Microsoft's Azure team before writing a line of code

Build

Delivered MVP in under 3 months, then iterated across 125+ sprints to production maturity

This design could be a case study in the proper use of Azure primitives. I have no notes.

— Microsoft Cloud Sales Engineering

The Platform

A serverless orchestration engine built on Azure

Azure Static Web App providing the operations dashboard, job scheduling interface, and self-service file access for business usersStatic Web AppJob Dashboard / Scheduling / File AccessServerless .NET functions serving as the central orchestration layer — scheduling, sequencing, retry logic, and coordination across all downstream servicesAzure FunctionsOrchestration EngineAll application code, Databricks notebooks, and dbt models version-controlled in GitHub with automated deployment pipelinesGitHubSource Control / CI/CDAzure Blob Storage managing encrypted flat files, partner data exchanges, and compressed archives with lifecycle policiesBlob StorageData Files / Flat Files / ArchivesAzure Key Vault securing SFTP credentials, API keys, encryption keys, and partner connection strings with access policiesKey VaultSecrets / Credentials / KeysAzure Databricks running data transformation notebooks, dbt models, and managed Spark clusters with programmatic spin-up and tear-downDatabricksNotebooks / dbtClusters (Compute)Azure Synapse Analytics storing all structured data, job definitions, scheduling configuration, and orchestrator metadata consumed by Azure FunctionsSynapse AnalyticsStructured Data / Job ConfigMedallion architecture data mart progressing raw ingested data through validated, conformed, and business-ready layersData MartBronze / Silver / Gold (Medallion)Hundreds of external data partners exchanging files and API calls — health plans, provider groups, government agencies, and health information exchangesExternal PartnersVendors / Providers / Gov Agencies / HIEs

Hover or focus a component to see details

The Platform

What the platform delivers

Encrypt / Decrypt Data

PGP encryption for partner file exchanges with automated secret management via Key Vault

Send / Receive Flat Files

SFTP and Azure Blob transfers supporting hundreds of partner endpoints with retry, centralized audit logging, and anomaly detection

External APIs

API integrations with configurable authentication, payload transformation, and response validation

Trigger Azure Functions

Orchestrated invocation of serverless function apps for custom processing, validation, and routing logic

Spin Up / Down Databricks Clusters

Programmatic cluster lifecycle management — provision compute on demand, tear down when complete to minimize cost

Execute Databricks Notebooks

Remote notebook execution with parameterized inputs, output capture, and integration into larger job chains

Run dbt Transformations

Orchestrated dbt model runs producing tested, documented data transformations across the medallion layers

Manage Medallion Pipeline

End-to-end Bronze, Silver, and Gold data pipeline with quality gates, lineage tracking, and incremental processing

Version-Controlled Everything

All application code, Databricks notebooks, and dbt models in GitHub with automated deployment pipelines and pull request workflows

.NET Azure Functions Blob Storage Key Vault Static Web Apps Databricks dbt Synapse Analytics GitHub

Built for People, Not Just Processes

A platform shaped by the people who use it

Started with a 4-person Azure engineering team. The broader data engineering and integrations team grew from 4 to 40+ as the platform proved its value and the business expanded.

Operations / Power Users

Full platform access — schedule jobs, manage configurations, monitor executions, troubleshoot failures, and manage partner integrations

~25 users

Business Users

Self-service file access, real-time job status monitoring, and direct download of partner data files without filing IT tickets

~75 users

Business users no longer had to write an email or file a ticket to get copies of files sent to vendors or provider groups. We gave them direct, self-service access.

Before

  1. Business user needs a copy of a file sent to a vendor
  2. Files a support ticket with IT operations
  3. Waits in queue behind other requests
  4. IT manually locates and retrieves the file
  5. File insecurely delivered days later via email

Days of waiting, multiple handoffs

Agile/Scrum from day one. Partnered with PMO for governance and visibility. Used spike stories for research and iterative delivery to build confidence across the organization.

Impact

Measurable outcomes that lasted

0
Jobs Configured
0+
Executions Managed
~0
Active Users
$0/mo
Azure Spend
0+
Sprints Delivered
0.0 months
Design to MVP
0
Databricks Notebooks
0+ years
Still Running

We were good stewards of cloud resources — tearing down compute, stopping idle instances, compressing blobs, and removing anything unnecessary.

What I bring

Engineering leadership grounded in delivery

Platform Thinking

I build foundations that accelerate the entire team. These aren't just features, but the systems that make features possible.

Cross-Functional Partnership

My best work happens at the intersection of engineering, product, and design. The platform succeeded because we built it together.

Cost Discipline

Technology should be an accelerator, not just a cost center. I optimize for sustainability and long-term business impact.

Shipping Culture

A small team that shipped constantly, with over 125 sprints together. I believe in iterative delivery, pragmatic architecture, and earning trust through execution.

Team Building

This greenfield project was only successful because of the team, its collaborative dynamic, and high engagement. I invest in building teams that trust each other and deliver together.

I've seen what happens when engineering teams build the right foundations. You ship faster, you operate leaner, and the business becomes profitable because technology is an accelerator, not a cost center.

Mark Danielson