Amazon Kinesis Firehose is a fully managed service that loads streaming data reliably to Amazon Redshift and other AWS services.
How to send data to Kinesis Firehose
You can send data to Firehose delivery stream directly or through other collection systems. Let’s take a look at a few examples.
Amazon Kinesis Agent
You can write to Amazon Kinesis Firehose using Amazon Kinesis Agent. Amazon Kinesis Agent is a stand-alone Java software application that offers a way to collect and send data to Firehose. The agent continuously monitors a set of files and sends new data to your Firehose delivery stream. You can install the agent on Linux-based server environments such as web servers, log servers, and database servers.
Use case
You could analyze a live stream of Apache access log data and aggregate the total request for each HTTP response type every minute.
Firehose API
Writing to a Firehose Delivery Stream using the AWS SDK. You can use the Firehose API to send data to a Firehose delivery stream using the AWS SDK for Java, .NET, Node.js, Python, or Ruby.
Use case
You could move from your internal queue to a Kinesis Firehose service.
Here‘s an example: Streaming Real-time Data into an S3 Data Lake at MeetMe
“While RabbitMQ consumers are implemented elsewhere (such as Flume), we opted to develop our own versions to enable integration with Firehose API”
AWS Lambda
Using AWS Lambda function to forward Stream data to Kinesis Firehose. It’s possible to forward data from Amazon Kinesis Streams or Amazon DynamoDB to Amazon Kinesis Firehose.
Use case
You could create a Lambda function that reads the streams data and writes to the Amazon Kinesis Firehose delivery stream
Fluentd plugin
Fluentd output plugin permits to send events to Amazon Kinesis Firehose (via API).
Use case
You could use the Fluentd’s Amazon Kinesis Output plugin to aggregate semi-structured logs in real-time.
AWS Service Proxy
Using AWS API Gateway via Service proxy. It’s possible to create an API Gateway as a proxy to an AWS service, such as Amazon SNS, Amazon S3, Amazon Kinesis.
Use case
One use case where it’s valuable to have immediate insights is analyzing clickstream data. In the world of digital advertising, an impression is when an ad is displayed in a browser and a clickthrough represents a user clicking on that ad. All client web requests can be handled by Amazon API Gateway. This handles the web request traffic to get data into your analytics pipeline by being a service proxy to an Amazon Kinesis.
How not to use it
You can’t send data to Firehose delivery stream using the Kinesis Producer Library. The KPL is only for an Amazon Kinesis Streams.