FPT SoftwareFHNCertificationsReference architectures
Talk to NGT
MLOps · 02 of 5

Self-hosted platform

Open-source MLOps in one private EKS cluster: Kafka, Airflow, MLflow, PyTorch, Triton, Prometheus, Grafana.

Self-Hosted MLOps on Amazon EKS
01

A reference platform on Amazon EKS

One VPC, one EKS cluster in a private subnet, only basic AWS services. Select a component for details.

Self-Hosted MLOps on Amazon EKS
label · run · approveimagesread in placepassedsigned releaseimage in · verdict outverdicts · scorescandidate runOn-premises siteAWS CloudVPCPrivate subnet · one EKS clusterGPU nodes · trainingRegional servicesDomain expertslabel · retrain · approveFile storageNAS · images stay hereImage sourcescameras, scannersWeb appReact · no-codeAPIFastAPI · OpenAPIPostgreSQLmetadataAmazon EC2 GPUinstancestraining onlyAmazon EKSone cluster · TerraformApprovala person signsIngest workerhash de-dupKafkaeventsAirflowpipeline runsPyTorchtraining runsMLflowtracking · registryEvaluation gatefail-closedGrafanadashboardsPrometheusmetricsDrift workerincoming vs trainingTritonverdict APIAmazon ECRcontainer imagesAmazon S3via VPC endpointAWS Lambdaclean-up only

Point at a component to light its links; select it for what it does here and the practice that makes it right on AWS.

Point at a component to see what it does; select it for the best practice.Open the architecture
02

The platform, layer by layer

Open source end to end, with no managed ML, queue or NoSQL service.

  1. 01User interfaceA web app for datasets, labels, runs and approvals, built for non-specialists.Reactgenerated API client
  2. 02APIFastAPI: ports and adapters, contract-first OpenAPI.FastAPIOpenAPI
  3. 03Events and orchestrationKafka carries events; Airflow runs ingest, train, test, deploy and drift workers.KafkaApache Airflowworkers
  4. 04Training and registryPyTorch training; every run and model in the MLflow registry.PyTorchONNXMLflow
  5. 05ServingNVIDIA Triton behind a verdict API: ACCEPT, REJECT or ABSTAIN.NVIDIA Tritonverdict API
  6. 06DataPostgreSQL metadata, S3 through a VPC endpoint, existing file storage mounted, ECR images.PostgreSQLS3NASECR
  7. 07RuntimeOne EKS cluster: CPU nodes for API, workers and serving; GPU nodes for training.Amazon EKSTerraformCPU and GPU node groups
Prometheus & Grafana · access per project and role
03

Self-hosted or managed?

Where self-hosting is required, a CI check can block managed AI services from the build. Many teams choose managed services instead.

Self-hosted on KubernetesManaged ML services
Data locationStays in place; read from existing storageFollows the service's storage and region
PortabilitySame design on any KubernetesTied to one provider's APIs
OperationsYour team or NGT runs cluster, GPUs, upgradesThe provider runs the infrastructure
Time to startLonger: the platform is built firstShorter: services are ready
GPU useGPUs only in the training node groupPaid per use of each service
04

Open-source toolchain

The reference platform's tools, plus others in NGT's MLOps catalogue.

Data and events

KafkaApache Airflow

Training

PyTorchONNX

Tracking and registry

mlflow

Serving

NVIDIA TritonFastAPI

Platform

amazon-ekskubernetesterraformargo

Observability

prometheusgrafana

Also in our catalogue

kubeflowrayFeastEvidentlyhuggingface
SourcesAI Modeling & MLOps case study, NGT (Sep 2026)AI-First DevOps Transformation: Services & Solutions, NGT (Oct 2026)NGT Service Offering v3.2 (Jul 2026)Machine Learning Platform with Kubeflow, NGT reference deckPlatform Engineering v1.2, NGT (Sep 2026)Data Platform Ops & DataOps Service Catalog, NGT (Sep 2026)FPT AI Factory service offering, FPT Smart Cloud (Nov 2025)