Apache Kafka
Term 34 of 68 in the ERPStack technical glossary
What is Apache Kafka?
Apache Kafka is a distributed event store and stream-processing platform designed to handle high-throughput, real-time data feeds and event messaging streams with high fault tolerance.
Apache Kafka at a glance
- Model
- A durable append-only log per topic, split into partitions that can be read independently
- Ordering
- Guaranteed within 1 partition; across partitions there is 0 ordering guarantee
- Retention
- Time or size based, so a consumer added on day 30 can replay history rather than starting empty
- Delivery
- At-least-once by default — 100% of consumers must be idempotent
- Built with
- Apache Kafka in front of PostgreSQL 18, consumers in TypeScript 5.9 on Docker and Kubernetes, feeding ClickHouse and TimescaleDB, provisioned by Terraform on AWS, monitored through Sentry
- Numbers that matter
- Order guaranteed within 1 partition, 0 across them; a consumer added on day 30 can replay history; 3 concepts to learn — brokers, partitions, groups
- Compare with
- Redis streams, a PostgreSQL Database queue table, or a managed cloud queue
- Commonly paired with
- ClickHouse, TimescaleDB, Redis, Docker and Kubernetes runtimes, Terraform provisioning, and Sentry on every consumer
- Default retention
- retention.ms defaults to 604800000 milliseconds — 7 days — at topic level, with log.retention.ms as the broker-wide equivalent. Retention is the replay budget, not a cleanup setting.
- Consumer arithmetic
- A partition is read by exactly 1 consumer in a group at a time, so 12 partitions cap that group at 12 working consumers and the 13th sits idle.
- Where it fits an ERP
- logistics and manufacturing event flows, with consumers traced through OpenTelemetry Observability and shipped by GitHub Actions.
- Smallest safe cluster
- 3 brokers with replication factor 3 and min.insync.replicas set to 2 is the smallest shape that still satisfies acks=all after losing 1 broker.
How Apache Kafka works in production
The ERPStack approach to Apache Kafka
We integrate Apache Kafka streams for clients processing millions of daily records, such as global supply chains or real-time IoT networks, preventing database lockups.
Frequently asked questions about Apache Kafka
What does Apache Kafka do that a queue does not?
It retains the messages. A traditional queue deletes work once consumed, whereas Apache Kafka keeps an ordered log for a configured period, so a new consumer can replay from the beginning and several independent consumers can read the same topic at their own pace. That is what makes it a backbone for analytics, auditing and integration simultaneously rather than a pipe between 2 services.
How does partitioning affect correctness?
It decides what stays in order. Apache Kafka guarantees ordering within 1 partition only, so all events for a given entity must share a partition key to be processed in sequence. Choosing a key with poor distribution creates a hot partition and limits throughput; choosing one that splits an entity's events across partitions produces updates applied out of order. Both are design decisions, not defaults.
When is Kafka the wrong tool?
When the volume does not justify the operations. Apache Kafka is a distributed system with brokers, partitions, consumer groups and rebalancing to understand, and for modest event volumes a PostgreSQL 18 table used as a queue or a managed message service is simpler and adequate. Adopting it because the architecture diagram looks incomplete without it is how teams acquire an operational burden with no matching benefit.
How does ERPStack use Kafka in business systems?
As the ingestion path for high-volume operational events. Apache Kafka absorbs bursts from warehouse scanners, point-of-sale terminals and IoT sensors so the transactional PostgreSQL 18 database is never the thing absorbing a spike directly. Downstream consumers then update inventory, feed analytics and trigger notifications independently, and each can fall behind and catch up without stopping the others.