Rust, Apache Arrow, Parquet based cloud native log storage platform

Parseable is an open source, unified infrastructure observability platform built in Rust on a data lake architecture. It tracks logs, metrics, traces, and events across apps, agents, and systems, reducing storage costs by up to 90% through columnar telemetry compression. - parseablehq/parseable

Over the past few months, we've been building Prism; a fast, developer-first observability UI that's deeply integrated with our open-source database, Parseable. Prism is designed for engineers who debug distributed systems, sometimes at odd hours, and need answers right now, without having to deal with five different tools. Backstory: We spoke to dozens of observability practitioners in large enterprises. Across the board, we heard two things: (1)The RoI on observability tools is hard to justify. (2)OSS leads to tool sprawl, metrics in Grafana, logs in ELK, traces… somewhere. We wanted a…

Founder here, Parseable is a lightweight log ingestion and query engine written in Rust. Parseable can ingest data from existing logging agents (FluentBit, LogStash, Vector, syslog-ng and more) using HTTP + JSON output. Ingested logs are stored as semi-indexed Parquet files (on disk or S3). You can query the data with builtin query engine using SQL or use a query engines of choice like Spark, Presto, Trino and so on. We also developed a Grafana data source plugin that lets you visualise log data via Grafana. Sample dashboard link in readme. As log data grew, our industry has responded with…

For write once, read many, immutable data types, Parquet on S3 is one of the most economical and scale approach. Every organisation I have talked to, in the last few years has attempted some form of Parquet on S3 implementation. However, these initiatives face challenges in standardising the approach for both ingesting and querying the data. Data quality, data governance and access control - everything becomes a challenge. We at Parseable are taking the Parquet on S3 approach and making it into an Enterprise grade product. Integrate easily with existing log agents and libraries. Built-in…

A high-performance observability data pipeline. Contribute to vectordotdev/vector development by creating an account on GitHub.

I built a Rust-based CLI/terminal UI for inspecting Parquet files—data, metadata, and row-group-level structure—right from the terminal. If someone sent me a Parquet file, I used to open DuckDB or Polars just to see what was inside. Now I can do it with one command. Repo: https://github.com/kaushiksrini/parqeye