DataOps

Serverless IoT Data Lake on AWS

Device data cleansed per event, history reprocessed in batch, all of it queryable in place with SQL.

Device data lands in S3 and every new object is cleansed and normalised by Lambda; Glue reprocesses years of history in batch. Curated data is catalogued for Athena and BI, while an API on API Gateway and Lambda serves datasets and their metadata.

uploadbulk historyreprocessedtablesmetadataOn siteAWS CloudIngest and cleanseStore and catalogAnalyzeOperateServe · data lake APIDevices and sensorsper siteHistory archiveyears of raw dataRaw zoneAmazon S3Amazon SQSobject eventsCleanse, normaliseAWS Lambda per eventAWS Gluebatch ETL for historyCurated zoneS3, partitionedAWS Glue DataCatalogtables, partitionsAmazon AthenaSQL in placeAmazon QuickSightdashboardsAnalysts and appsAmazon API GatewayCognito sign-inData lake servicesAWS LambdaAmazon DynamoDBdataset metadataAmazon OpenSearchServicefind datasetsAmazon CloudWatchlogs, alarmsAWS CloudTrailAPI auditAWS IAMa role per functionAmazon SNSdata-quality alerts

Point at a component to light its links; select it for what it does here and the practice that makes it right on AWS.

When to use it

  • Device fleets whose data has outgrown the system that stores it
  • Years of history that must be cleaned and reprocessed, not just archived
  • Spiky, event-driven loads where idle servers would cost more than the work

How NGT delivers it

Built from code, reviewed at a gate by an NGT architect, and handed over with runbooks your team can run.

DataOps