Create a Data Stream

In this section, you will create a Firehose delivery stream from the AWS Management Console, configure it with a few clicks to store incoming stream data into S3, and start sending data to the stream Kinesis Data Generator (created in the previous section) as the data source.

Amazon Kinesis Data Firehose is a serverless service used to reliably load streaming data into data stores and analytics tools. It can capture, transform, and load streaming data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and 3rd Party services (e.g. Splunk), enabling near real-time analytics.
  1. Login as the lf-admin user and select the N. Virginia Region.
  2. In the AWS Management Console go to Find Services and look for the Kinesis service.
  3. In the Kinesis home page, under the Get Started section, select the Kinesis Data Firehose option and then select Create delivery stream.
  4. Add a new delivery stream and configure as follows:
    • Delivery stream name: lf-kinesis-fh
    • Source: Direct PUT or other sources
    and then select Next.
  5. Select the Disabled option for Data transformation and Record format conversion and then select Next.
  6. In destination, select the Amazon S3.
  7. Configure the destination as follows:
    • S3 bucket: Select the name of the bucket created by CloudFormation lf -data-lake-bucket-youraccountid
    • S3 prefix: dl_tpc_sales_event/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/
    • S3 error prefix: dl_tpc_sales_event_errors/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/!{firehose:error-output-type}
    and then select Next.
  8. Configure the S3 buffer condition as follows:
    • Buffer size: 1
    • Buffer interval: 60
  9. Configure the S3 Compression and encryption condition as follows:
    • S3 compression: Disabled
    • S3 encryption: Disabled
    and then select Next.
  10. For Error Logging, select Disabled.
  11. For Permissions select the Create new or choose button.
  12. Configure the Role Summary section as follows:
    • IAM Role: LF-KinesisServiceRole
    • Policy Name : LF-Stream-Data-Storage-Policy
    and then select Allow.
  13. Select Next.
  14. Select Create delivery stream.
  15. Wait until you see the Successfully created delivery stream lf-kinesis-fh message.
In the next step, you are going to generate real-time data for this delivery stream.