Tags: python architecture oop design-patterns distributed-systems When engineering distributed monitoring agents or designing low-latency health-checking pipelines, separating centralized governance from autonomous edge execution is essential. I designed the Prime-Sentinel Command (PSC) architecture as an object-oriented master-agent pattern to coordinate edge diagnostic nodes (Sentinels) via a centralized orchestrator (Prime). Below is an architectural walkthrough and minimal reference implementation for engineers looking to build similar decoupled telemetry collectors. Many diagnostic…
A few weeks ago, we spoke at OpenSearchCon India about how we make search fast and reliable at scale at Blinkit.I’m Harshit, from the Search Engineering team at Blinkit, and this year we got to speak at OpenSearchCon India at the Jio World Convention Centre in Mumbai. In this post, I’ll walk you through our talk, the conversations that followed, and what our team took away from being part of the OpenSearch community in a more active way this year.First Day : Keynote by BlinkitThe highlight of this year’s conference for our team was the opportunity to deliver a keynote session at…
By Parth Jain, Rakesh Sukumar, Yingwu Zhao, Renzo Sanchez-Silva & Nathan FisherA deep dive into the engineering challenges of building a real-time service dependency map at Netflix scale: from streaming architectures and distributed aggregation pipelines to time-travel queries and the methodology that made it work.IntroductionIn our first post, we introduced the problem: engineers at Netflix needed a unified, real-time view of service dependencies to troubleshoot faster, understand blast radius, and navigate our distributed architecture. We described our multi-source approach, combining eBPF…
How Airbnb built a Kubernetes sidecar to deliver dynamic configuration reliably at scale.By: Bo Teng, Cosmo Qiu, Siyuan Zhou, Ankur Soni, Xin Huang, Willis HarveyIntroductionIn our previous post, we explored Airbnb’s dynamic configuration system, Sitar, with a focus on service architecture and configuration change safety. Now for the harder question: once a config change is committed, which happens several times each minute, how does it actually reach the thousands of Airbnb’s service instances reliably, quickly, and without redeploying the services?This post describes sitar agent: a…
Expedia Group Technology — EngineeringA system that facilitates investigation of service degradations and outages using service telemetry data and AIPhoto by Evangelos Mpikakis on Unsplash.The recent advancements in the artificial intelligence space make us re-evaluate how work is done. From programming, to designing systems, or even operating them in production. While there is considerable focus on automating programming, one area which could undergo transformation is how we monitor and operate our systems and services.A few of us came together and designed Expedia’s® Service Telemetry…
Written by Stefan ZierFor years, Lyft’s localization infrastructure relied exclusively on human translation. While this model usually ensured excellent quality, it was bound by multi-day turnarounds and costs that scaled linearly with every new language. For the few languages Lyft initially supported (Spanish, Portuguese, and French), these limits were acceptable.However, Lyft’s expansion goals quickly outpaced what traditional workflows could support. Lyft’s recent Québec launch required compliance with Bill 96 (legislation mandating French-first user experiences) which demanded faster…
A brutally simple and effective implementation for long-running account move jobs at Zendesk.This article outlines some architectural changes we’ve been able to make to radically simplify the execution model of long-running jobs.By leveraging client behaviour, the resulting system improves overall functionality while removing the many complexities of distributed job execution.Dall-e impression of a server who’s ready to move some data!Background: Account moves at ZendeskBehind the scenes at Zendesk, the data for a given customer account lives in one of our regions across the globe. We don’t…
The Jobteaser application contains a lot of different relatively independent modules to help universities provide career guidance to students: a job board, a career event management system, a career advice appointment management system…When we decided to migrate our application’s backend from a monolith to a service-oriented architecture, we strived to keep each module as isolated as possible from the others in the event of an incident. If the career appointment system was down, students should still be able to browse and apply to job ads.That isolation is achieved through what we’ve called…
Optional Google Analytics helps us understand visits. Microsoft Clarity records masked interactions to improve the site. Optional tools stay off unless you choose them. Privacy details.