Search on S3 Using AWS Lambda
Hi everyone, During the last 3 months, I worked on the Quickwit search engine to adapt it for AWS Lambda runtime. I wrote a blog post to announce the beta release [0] and two tutorials to quickstart: - Indexing and searching 20 million logs entries dataset [1]. - Implementing an E2E use case where an application generates data, uploads it in batches to S3. Users can then search through an HTTP API authenticated with an API Key [2]. On the performance side, I observed on a 20 million log dataset that search is sub-second, and analytics queries take from 1 to 4 second. On the cost side,…
What it does
In the maker’s words, at launch
Hi everyone, During the last 3 months, I worked on the Quickwit search engine to adapt it for AWS Lambda runtime. I wrote a blog post to announce the beta release [0] and two tutorials to quickstart: - Indexing and searching 20 million logs entries dataset [1]. - Implementing an E2E use case where an application generates data, uploads it in batches to S3. Users can then search through an HTTP API authenticated with an API Key [2]. On the performance side, I observed on a 20 million log dataset that search is sub-second, and analytics queries take from 1 to 4 second. On the cost side, obviously the compute using Lambda is fairly expensive, but the huge win is that it scales to 0. Few other solutions do. For instance OpenSearch Serverless has a fixed cost around $700/months. On the search side, thanks to indexing, queries are usually way cheaper than scanning services like CloudWatch (at least 50x cheaper). I would love to get the feedback from the community and see how I can push this further. [0]: Blog post https://quickwit.io/blog/quickwit-lambda-beta [1]: 20 million log dataset search tutorial https://quickwit.io/docs/get-started/tutorials/tutorial-aws-... [2]: E2E use case tutorial https://quickwit.io/docs/guides/e2e-serverless-aws-lambda
Does the same job
all alternatives →- QOQuickwit – OSS Alternative to Datadog, Elasticsearch2024 · github.com · ▲8
Hi HN, Quickwit cofounder here! Quickwit is a cloud-native search engine for observability, and we have just released a significant upgrade with performance improvements, extended Elasticsearch API compatibility, and better integration with OpenTelemetry tools, Grafana, and Jaeger. This release is also a significant milestone as we have been witnessing users of the nightly version of Quickwit deploy clusters with hundreds of nodes, ingest hundreds of terabytes of data daily, and enjoy additional cost savings. We are very excited to put it in the hands of all of our users. One report…
- IBI built a service to help companies save on their AWS bills2020 · ▲73
Hey HN: I'm Kaveh, the founder of Usage (https://www.usage.ai/) We help companies drive down AWS costs. Why? Because the way it's done now is a pain. Stakeholders, especially engineers, are required to spend unnecessary time manually finding underutilized or overly expensive EC2s. We believe the optimization process should be done automatically through a series of sophisticated algorithms. At the moment, there are over 70,000 AWS EC2 prices - doing that manually just won't scale at most organizations. My background is in software engineering. Previous to founding Usage, I…
- BQBucket Query – Automatically index and search your AWS S3 buckets2017 · bucketquery.com · ▲6
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, January 2024
the whole month →
- IM
Life & fun · 2024 · sitinshade.com
