FPT SoftwareFHNCertificationsReference architectures
Talk to NGT
DataOps · 03 of 6

Data platform engineering

The lakehouse platform, built and run as code on any cloud, ready for AI.

DataOps Lakehouse Platform
01

The data platform NGT builds and runs

Walk it, or select a component for what it does and the practice behind it. Switch clouds to see each provider's services.

DataOps Lakehouse Platform
passfaildeploy · schedulepolicies · lineageaccess rulesSourcesAWS CloudOrchestrate & CI/CD · always onIngestLakehouse · bronze → silver → goldServeGovern & secure · always onConsumersDatabases & ERPApps & SaaSFiles & documentsEvents & IoTGitpipelines as codeAWS CodePipelinetest · promotePlatform as codeTerraform · PulumiDev · QA · Prodsame topologyAmazon MWAAschedule · retryAmazon CloudWatchpipeline SLAsAWS Cost Explorercost per teamAWS GluebatchAmazon MSKstreamingBronzeraw, as landedAmazon EMRSpark · dbt jobsSilvercleaned, conformedData testsquality gateGolddata productsData contractschema · owner · rulesOpen tablesApache IcebergQuarantineowner alertedAmazon RedshiftAmazon QuickSightAmazon SageMaker AIfeatures · registryAmazon OpenSearchServicevector indexBusiness usersML, RAG & agentsAWS Glue DataCatalogcatalog · glossaryLineage & auditFine-grained accessLake FormationIAM Identity CenterAWS KMSAWS Secrets ManagerAWS CloudTrailaudit trail

Point at a component to light its links; select it for what it does here and the practice that makes it right on AWS.

Point at a component to see what it does; select it for the best practice.Open the architecture
02

Six layers, each provisioned as code

Every layer is built from code and promoted Dev → QA → Prod.

  1. 01IngestionBatch and streaming connectors.KafkaAirbyteSpark
  2. 02Lakehouse storageOpen tables on object storage, by environment and domain.Delta LakeApache IcebergS3 · ADLS · GCS
  3. 03Compute & transformationSQL and Spark with autoscaling and cost limits.DatabricksSnowflakeBigQuerySynapsedbt
  4. 04OrchestrationScheduled, retried and monitored as code.AirflowCloud Composer
  5. 05Catalog, lineage & accessMetadata, lineage and fine-grained access in one place.Unity CatalogMicrosoft PurviewAWS Lake FormationDataplexAmazon DataZone
  6. 06Serving & AI-ready accessBI, feature store, model registry and AI gateway on governed data.BI dashboardsFeature storeMLflowAI gateway
IaC · CI/CD · security · observability
03

Where we start on each cloud

The design is cloud-agnostic. These are NGT's reference starting points.

AWSServerless data lake → data mesh with Amazon DataZone
  • S3, Glue and Athena, set up with the Serverless Data Lake Framework
  • Domains publish data products through Amazon DataZone
  • Lake Formation for fine-grained access
Amazon S3AWS GlueAmazon AthenaAWS Lake FormationAmazon DataZoneAmazon MSK
Microsoft AzureAzure Databricks lakehouse
  • Unity Catalog and Delta Lake; jobs as Databricks Asset Bundles
  • Azure DevOps, Key Vault, Microsoft Entra ID
  • MLflow, Azure ML and AI Search on the same data
Azure DatabricksUnity CatalogDelta LakeAzure DevOpsKey VaultEntra IDMicrosoft PurviewSynapse
Google CloudBigQuery-centred analytics
  • Pub/Sub, Dataflow (Apache Beam) and Dataproc at scale
  • BigQuery as warehouse and checkpoint; Looker dashboards
  • Cloud Composer, Dataplex and Vertex AI on the same data
BigQueryDataflowPub/SubDataprocCloud ComposerDataplexLookerVertex AI
04

AI-ready data for ML, RAG and agents

AI stalls when its data sits outside the platform. We serve it governed, under the same access rules and audit.

Governed dataAI and analytics workloads
Governed access layeridentity · policies · audit

Point at a source to see what it feeds, or at a result to see where it comes from.

05

Toolbox

From NGT's DataOps toolbox. We start from what you already run.

Data platforms

DatabricksSnowflakeBigQuerySynapse

Processing & orchestration

sparkKafkaAirflowAirbytedbt

Automation & delivery

githubterraformjenkinsargokubernetes

Quality & observability

pytestDVCgrafanaprometheusdatadogelasticsearch

Governance

Microsoft PurviewAWS Lake FormationDataplexUnity Catalog

ML & AI

mlflowrayhuggingfacelangchain
SourcesNGT DataOps service offering drafts (Sep–Oct 2026)NGT DataOps & Data Platform Ops service catalogue (Sep 2026)NGT DataOps one-page offering and operating model (Sep 2026)NGT AI-First DevOps Transformation service offering (Oct 2026)NGT Google Cloud service offering v0.1