Amazon Kinesis Firehose is a fully managed service that loads streaming data reliably to Amazon Redshift and other AWS services.

How to send data to Kinesis Firehose

You can send data to Firehose delivery stream directly or through other collection systems. Let’s take a look at a few examples.

Amazon Kinesis Agent

You can write to Amazon Kinesis Firehose using Amazon Kinesis Agent. Amazon Kinesis Agent is a stand-alone Java software application that offers a way to collect and send data to Firehose. The agent continuously monitors a set of files and sends new data to your Firehose delivery stream. You can install the agent on Linux-based server environments such as web servers, log servers, and database servers.

Use case

You could analyze a live stream of Apache access log data and aggregate the total request for each HTTP response type every minute.

Firehose API

Writing to a Firehose Delivery Stream using the AWS SDK. You can use the Firehose API to send data to a Firehose delivery stream using the AWS SDK for Java, .NET, Node.js, Python, or Ruby.

Use case

You could move from your internal queue to a Kinesis Firehose service.

Here‘s an example: Streaming Real-time Data into an S3 Data Lake at MeetMe

“While RabbitMQ consumers are implemented elsewhere (such as Flume), we opted to develop our own versions to enable integration with Firehose API”

AWS Lambda

Using AWS Lambda function to forward Stream data to Kinesis Firehose. It’s possible to forward data from Amazon Kinesis Streams or Amazon DynamoDB to Amazon Kinesis Firehose.

Use case

You could create a Lambda function that reads the streams data and writes to the Amazon Kinesis Firehose delivery stream

Fluentd plugin

Fluentd output plugin permits to send events to Amazon Kinesis Firehose (via API).

Use case

You could use the Fluentd’s Amazon Kinesis Output plugin to aggregate semi-structured logs in real-time.

AWS Service Proxy

Using AWS API Gateway via Service proxy. It’s possible to create an API Gateway as a proxy to an AWS service, such as Amazon SNS, Amazon S3, Amazon Kinesis.

Use case

One use case where it’s valuable to have immediate insights is analyzing clickstream data.  In the world of digital advertising, an impression is when an ad is displayed in a browser and a clickthrough represents a user clicking on that ad. All client web requests can be handled by Amazon API Gateway. This handles the web request traffic to get data into your analytics pipeline by being a service proxy to an Amazon Kinesis.

How not to use it

You can’t send data to Firehose delivery stream using the Kinesis Producer Library. The KPL is only for an Amazon Kinesis Streams.