FPT SoftwareFHNCertificationsReference architectures
Talk to NGT
DataOps · 01 of 6

Lifecycle

Seven stages in one loop, with orchestration and governance always on.

githubterraformsparkargografanaprometheus
Every stage as code, on tools you already run
01

The DataOps lifecycle

Seven stages, one continuous loop. Orchestration with CI/CD and governance with security stay on throughout.

DataOpscontinuous loop
Step 1 of 7

Plan

Agree the data product, its users, quality rules and service levels first.

Always onOrchestrate & CI/CD · pipelines as code · promotion Dev → QA → ProdGovern & secure · catalog · lineage · access · PII protection
02

Who does what at each stage

DataOps engineers own the platform and automation; data engineers own the data products; AI drafts, engineers review.

Stage
Data engineering team
DataOps engineers (NGT)
Platform & tools
AI assistance, engineer-reviewed
01Plan
Data engineering team
  • Define the data product
  • Write quality rules with the business
DataOps engineers (NGT)
  • Agree service levels and access model
  • Size environments, set cost guardrails
Platform & tools
  • Contract templates in Git
AI assistance, engineer-reviewed
  • Drafts docs from schemas
02Ingest
Data engineering team
  • Map sources to the target model
DataOps engineers (NGT)
  • Standard connectors and templates
  • Secrets and network access as code
Platform & tools
  • Kafka, Airbyte, Spark
AI assistance, engineer-reviewed
  • Suggests schema mappings
03Store
Data engineering team
  • Model tables and domains
DataOps engineers (NGT)
  • Storage, catalog and environments as code
Platform & tools
  • Delta Lake or Iceberg
  • Unity Catalog, Purview, Lake Formation
AI assistance, engineer-reviewed
    04Transform
    Data engineering team
    • Transformations as code
    DataOps engineers (NGT)
    • CI/CD with review and promotion
    Platform & tools
    • Spark, dbt, Databricks, Snowflake, BigQuery
    AI assistance, engineer-reviewed
    • Assisted coding and review
    05Test
    Data engineering team
    • Business-rule tests
    DataOps engineers (NGT)
    • Test stage and gates in every pipeline
    Platform & tools
    • pytest and SQL tests in CI
    AI assistance, engineer-reviewed
    • Drafts test cases to review
    06Serve
    Data engineering team
    • Publish data products
    DataOps engineers (NGT)
    • Column- and row-level security
    • Governed access for ML, RAG, agents
    Platform & tools
    • BI, feature store, vector index
    AI assistance, engineer-reviewed
      07Operate
      Data engineering team
      • Fix data logic
      DataOps engineers (NGT)
      • Monitoring, runbooks, on-call
      • Cost and capacity
      Platform & tools
      • Grafana, Prometheus, Datadog
      AI assistance, engineer-reviewed
      • Helps triage alerts and logs
      03

      From DevOps practice to DataOps

      Practices teams already know from DevOps, mapped onto the data platform.

      DevOps practiceIn DataOpsWhat changes
      CI/CD for applicationsCI/CD for pipelines and jobsData changes ship through review and tests
      Infrastructure as codePlatform IaC: storage, catalog, permissionsEnvironments rebuilt the same way
      Monitoring and SREData observability: freshness, volume, SLAsProblems found before a report is wrong
      DevSecOpsAccess policies, PII masking, keys, auditCompliance evidence from the platform
      Platform engineeringSelf-service data platform with templatesNew data products start from a template
      SourcesNGT DataOps service offering drafts (Sep–Oct 2026)NGT DataOps & Data Platform Ops service catalogue (Sep 2026)NGT DataOps one-page offering and operating model (Sep 2026)NGT AI-First DevOps Transformation service offering (Oct 2026)NGT Google Cloud service offering v0.1