Heartbeats: How Synthetic Traffic Keeps Us RunningLet me take you on a journey of how we came to use heartbeats in our application design. It’s a happy story of love and no broken hearts along the way.What are heartbeats?What my teams have called heartbeats are a form of synthetic traffic generated by the application itself. The deployed application periodically generates heartbeats at a defined schedule.Heartbeats provide guaranteed regular traffic. In the cases I’ve used them, they have been low volume. In contrast to application traffic, which could vary massively from zero to huge…
When building systems that process large volumes of messages synchronously, performance bottlenecks can quickly become a challenge specially with single-threaded designs. In this post, we’ll look at how leveraging worker threads in a Clojure-based Kafka consumer can significantly boost throughput & reduce total processing time. Using simple concurrency primitives, it’s possible to achieve parallelism & scale gracefully, all while keeping the codebase clean & maintainable. We’ll start with a baseline, introduce worker threads using Clojure’s core.async & measure the impact.Setup & ContextKafka…
Confluent Kafka is an enterprise-ready distribution of Apache Kafka, providing a reliable and scalable platform for real-time data streaming. It includes additional features such as schema registry, REST proxy, connectors, and advanced monitoring tools.And migrating your existing Kafka cluster can be driven by several factors, including the need to upgrade hardware, move to a cloud-based infrastructure, enhance performance, improve scalability, or optimize resource utilization. This migration can help ensure your data infrastructure remains robust and capable of handling future…
The Jobteaser application contains a lot of different relatively independent modules to help universities provide career guidance to students: a job board, a career event management system, a career advice appointment management system…When we decided to migrate our application’s backend from a monolith to a service-oriented architecture, we strived to keep each module as isolated as possible from the others in the event of an incident. If the career appointment system was down, students should still be able to browse and apply to job ads.That isolation is achieved through what we’ve called…
I’m working with Kafka for more than 2 years and I wasn’t sure if Kafka Consumer eats more RAM memory when it has more partitions. I couldn’t find any useful information on the internet, so I decided to measure everything by myself.
Optional Google Analytics helps us understand visits. Microsoft Clarity records masked interactions to improve the site. Optional tools stay off unless you choose them. Privacy details.