Overview
The Streaming Data Solution for Amazon Kinesis provides AWS CloudFormation templates where data flows through producers, streaming storage, consumers, and destinations. To support multiple use cases and business needs, the Solution offers four CloudFormation template. Similar to the Streaming Data Solution for Amazon MSK, the templates are configured to apply best practices to monitor functionality using dashboards and alarms and to secure data.
Streaming data must be durably captured by massively scalable storage that is capable of handling high data volume from data producers. A producer can be thousands of data sources, each generating streaming data continuously and which, typically, submit records simultaneously and in small sizes (kilobytes).
Streaming data includes a wide variety of data such as log files generated by customers using mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services and telemetry from connected devices or instrumentation in data centers.
Benefits
Technical details
-
Option 1
-
Option 2
-
Option 3
-
Option 4
-
Option 1
-
AWS CloudFormation template using Amazon API Gateway and AWS Lambda
AWS CloudFormation template using Amazon API Gateway and AWS Lambda
Option 1 – AWS CloudFormation template using Amazon API Gateway, Kinesis Data Streams, and AWS Lambda
Step 1
An Amazon API Gateway REST API acts as a proxy to Amazon Kinesis Data Streams, adding either an individual data record or a list of data records.Step 3
Kinesis Data Streams to store the incoming streaming data.Step 4
An AWS Lambda function processes the records from the data stream.Step 5
Errors and failed records that occur during AWS Lambda processing are annotated, and the events are stored in Amazon Simple Queue Service (Amazon SQS). The queue stores metadata for failed batch records and Lambda errors, allowing customers to retrieve these records and determine the next steps to resolve them.Option 1 – AWS CloudFormation template using Amazon API Gateway, Kinesis Data Streams, and AWS Lambda
Step 1
An Amazon API Gateway REST API acts as a proxy to Amazon Kinesis Data Streams, adding either an individual data record or a list of data records.Step 3
Kinesis Data Streams to store the incoming streaming data.Step 4
An AWS Lambda function processes the records from the data stream.Step 5
Errors and failed records that occur during AWS Lambda processing are annotated, and the events are stored in Amazon Simple Queue Service (Amazon SQS). The queue stores metadata for failed batch records and Lambda errors, allowing customers to retrieve these records and determine the next steps to resolve them.Related content
AWS Architecture BlogReal-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache FlinkRead the full blog postThis post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Kinesis Data Analytics (KDA) with Apache Flink to asynchronously invoke underlying services or databases.
TrainingIntroduction to Amazon Kinesis AnalyticsEnroll nowThis is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
About this deploymentVersion1.7.8Released07/2023AuthorAWSEst. deployment time5-10 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 2
-
AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
Option 2 – AWS CloudFormation template using AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 3
Kinesis Data Analytics Studio processes the incoming records and saves the processed data in an Amazon Simple Storage Service (Amazon S3) bucket.Step 4
An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.
Option 2 – AWS CloudFormation template using AWS CloudFormation template using Amazon EC2, Amazon Kinesis Producer Library, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon CloudWatch
Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 3
Kinesis Data Analytics Studio processes the incoming records and saves the processed data in an Amazon Simple Storage Service (Amazon S3) bucket.Step 4
An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.
Related content
AWS Architecture BlogReal-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache FlinkRead the full blog postThis post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Kinesis Data Analytics (KDA) with Apache Flink to asynchronously invoke underlying services or databases.
TrainingIntroduction to Amazon Kinesis AnalyticsEnroll nowThis is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
About this deploymentVersion1.7.8Released07/2023AuthorAWSEst. deployment time5-10 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 3
-
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
Option 3 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 2
Amazon Kinesis Data Firehose buffers the data before delivering the output to an Amazon S3 bucket. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration.Step 3
An Amazon CloudWatch dashboard monitors the data ingestion and buffering. CloudWatch alarms are set on essential metrics for Kinesis Data Firehose.Option 3 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon S3
Step 1
Amazon Kinesis Data Streams stores the incoming streaming data.Step 2
Amazon Kinesis Data Firehose buffers the data before delivering the output to an Amazon S3 bucket. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration.Step 3
An Amazon CloudWatch dashboard monitors the data ingestion and buffering. CloudWatch alarms are set on essential metrics for Kinesis Data Firehose.Related content
AWS Architecture BlogReal-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache FlinkRead the full blog postThis post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Kinesis Data Analytics (KDA) with Apache Flink to asynchronously invoke underlying services or databases.
TrainingIntroduction to Amazon Kinesis AnalyticsEnroll nowThis is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
About this deploymentVersion1.7.8Released07/2023AuthorAWSEst. deployment time5-10 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console -
Option 4
-
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
Option 4 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance that uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 3
Kinesis Data Analytics processes the incoming records and asynchronously invokes an external endpoint.Step 4
The demo application invokes an AWS Lambda function.Step 5
The external API can be any integration supported by Amazon API Gateway (for example, an Amazon SageMaker endpoint).Step 6
An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.Option 4 – AWS CloudFormation template using Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics, and Amazon API Gateway
Step 1
An Amazon Elastic Compute Cloud (Amazon EC2) instance that uses the Amazon Kinesis Producer Library (KPL) to generate data.Step 3
Kinesis Data Analytics processes the incoming records and asynchronously invokes an external endpoint.Step 4
The demo application invokes an AWS Lambda function.Step 5
The external API can be any integration supported by Amazon API Gateway (for example, an Amazon SageMaker endpoint).Step 6
An Amazon CloudWatch dashboard monitors application health, progress, resource utilization, events, and errors.AWS Architecture BlogReal-Time In-Stream Inference with AWS Kinesis, SageMaker, & Apache FlinkRead the full blog postThis post outlines the architecture to achieve real-time inference on streaming data using various AWS Services. It also covers the integration of Amazon Kinesis Data Analytics (KDA) with Apache Flink to asynchronously invoke underlying services or databases.
TrainingIntroduction to Amazon Kinesis AnalyticsEnroll nowThis is an introductory course on Amazon Kinesis Analytics, which helps you query streaming data or build entire streaming applications using SQL.
TrainingData Analytics FundamentalsEnroll nowIn this self-paced course, you learn about the process for planning data analysis solutions and the various data analytic processes that are involved.
About this deploymentVersion1.7.8Released07/2023AuthorAWSEst. deployment time5-10 minsDownload implementation guide Source code CloudFormation template Subscribe to RSS feedEstimated costDeployment optionsReady to get started?Deploy this solution by launching it in your AWS Console