Inference
We track 15 posts about Inference from 9 engineering blogs. Most active: Red Hat, Android, Nvidia. Latest post: Oct 13, 2026.
Companies writing about Inference
Recent posts
Red Hat AI Inference 3.5 MaaS governance on AKS (opens on the source site)
Red Hat ·
Running AI models on public cloud Kubernetes clusters offers great flexibility, but it creates an immediate governance challenge: how do you secure endpoints, authenticate developers, and prevent runaway GPU costs across teams? For platform engineers, Red Hat AI Inference addresses this by bringing multi-tenant governance to any Kubernetes distribution. The post Red Hat AI Inference 3.5 MaaS governance on AKS appeared first on Red Hat Developer.
Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations (opens on the source site)
Facebook ·
At Meta, we’ve built a system that enhances ads recommendations by leveraging idle inference capacity in our fleet. Two elasticity levers in the system allow ads recommendation work to expand across opportunistic capacity, without compromising baseline performance. This post describes how we designed and now operate the system at Meta’s scale and how it drives [...] Read More... The post Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations appeared first on Engineering at Meta.
How I massively improved my AI inference performance without buying new hardware (opens on the source site)
Red Hat ·
Let me paint you a picture. You have 16 NVIDIA H200 GPUs spread across 2 nodes. That is, conservatively, several 100,000 dollars of silicon sitting in a data center, connected by RDMA/InfiniBand, running Kubernetes, and serving a large language model. You're living the agentic dream (not really, but it's a great start). Except your time to 1st token (TTFT) is thousands of milliseconds. The post How I massively improved my AI inference performance without buying new hardware appeared first on Red Hat Developer.
NVIDIA Vera Rubin NVL72 Delivers Leading Performance in MLPerf Inference v6.1 Debut (opens on the source site)
Nvidia ·
System performance, efficient infrastructure scaling and continuous software optimization are key levers that determine AI inference economics. Higher system performance means more tokens generated, resulting in higher revenue. Efficient scaling means throughput grows proportionally as hardware gets added, requiring fewer resources to serve users at scale. Continuous optimization means generating more value from infrastructure investments. […]
How to Deploy vLLM NVIDIA Dynamo Inference for High-Throughput Serving (opens on the source site)
Learn how to deploy vLLM NVIDIA Dynamo inference in production with Docker Compose and tuned PagedAttention to eliminate memory fragmentation. Continue reading How to Deploy vLLM NVIDIA Dynamo Inference for High-Throughput Serving on SitePoint.
Speeding up LLM inference with P-EAGLE in vLLM Speculators (opens on the source site)
Red Hat ·
P-EAGLE (Parallel EAGLE), a new speculative decoding algorithm developed by Amazon, brings the next evolution of speculative decoding to Speculators by extending EAGLE-3 with parallel drafting. The post Speeding up LLM inference with P-EAGLE in vLLM Speculators appeared first on Red Hat Developer.
llm-d flow control: Priority queuing for shared GPU inference (opens on the source site)
Red Hat ·
llm-d flow control adds priority-aware admission and tenant fairness scheduling mechanisms for GPUs, giving platform teams a way to prioritize mixed workloads on a shared model pool. This feature is generally available in Red Hat AI Inference 3.5. The post llm-d flow control: Priority queuing for shared GPU inference appeared first on Red Hat Developer.
With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents (opens on the source site)
Nvidia ·
The next era of AI inference won’t be defined by a single breakthrough chip, network or system. It’ll be defined by how every layer of the AI factory works together. That’s why NVIDIA is extending Vera Rubin NVL72 with fast token generation for agentic systems. Announced today, the NVIDIA Vera Rubin rack-scale system NVIDIA Groq […]
Correlation Lied to Us: Rethinking Product Impact with Causal Inference (opens on the source site)
OLX ·
IntroductionAt OLX, professional sellers pay for higher-tier packages because they promise more exposure. More visibility, and, in theory, better results. But when we looked at the data, something unexpected happened. In some cases, ads published with premium packages appeared to perform worse than ads using cheaper packages.That raised an uncomfortable question: If higher-tier packages provide more exposure, shouldn’t they consistently perform better?At first glance, there were several possible explanations. Perhaps the extra visibility weren’t creating as much value as we expected. Perhaps…
Build intelligent Android apps: Cloud and hybrid inference (opens on the source site)
Android ·
Posted by Thomas Ezan, Jolanda Verhoef, Caren Chang, Senior Developer Relations Engineers, Android Developer RelationsWelcome back to the blog post series "Build intelligent Android apps" where we take a basic Android app and transform it into a personalized, intelligent, and agentic experience. In our previous post we explored how to build intelligent on-device features using Gemini Nano through ML Kit's Prompt API. In this post, we will look at how you can leverage Firebase AI Logic to build cloud-hosted and hybrid AI features: Grounding answers in real-world context Routing requests…
Build intelligent Android apps: On-device inference (opens on the source site)
Android ·
Posted by Caren Chang, Developer Relations Engineer, Android Developer RelationsWelcome back to the blog post series "Build intelligent Android apps" where we take a basic Android app and transform it into a personalized, intelligent, and agentic experience. In our previous post we introduced Jetpacker, the demo app we'll use throughout this series. In this blog post, we will share how you can use Gemini Nano through ML Kit’s Prompt API to build intelligent on-device features. Building intelligent on-device features refers to the ability to process prompts and data directly on a device…
Build intelligent Android apps: Cloud and hybrid inference (opens on the source site)
Android ·
Posted by Thomas Ezan, Jolanda Verhoef, Caren Chang, Senior Developer Relations Engineers, Android Developer Relations Welcome back to the blog post series "Build intelligent Android apps" where we take a basic Android app and transform it into a personalized, intelligent, and agentic experience. In our previous post we explored how to build intelligent on-device features using Gemini Nano through ML Kit's Prompt API. In this post, we will look at how you can leverage Firebase AI Logic to build cloud-hosted and hybrid AI features: Grounding answers in real-world context Routing requests…
The future of Siri, or: why private inference isn’t private enough (opens on the source site)
Yesterday Apple announced a big step towards deploying real AI in their Siri ecosystem. In most ways this is good and inevitable: Siri is one of the world’s most widely-used voice agents, and it would be good if it didn’t suck. The idea that Apple would boost its capabilities with frontier models wasn’t so much … Continue reading The future of Siri, or: why private inference isn’t private enough →
Distributing LLM inference in DwarfStar (opens on the source site)
Antirez ·
High end NVIDIA cards, and the server and power needed to run them, cost a lot of money, especially if you plan to reach enough VRAM to run massive models. The alternative, so far, has been Apple hardware, or the DGX Spark that, even if severely limited because of memory bandwidth, still allows to run LLMs prompt processing (prefill) fast enough. The Mac Studio provided up to 512GB unified memory, a solution with modest memory bandwidth (but much better than the Spark) and compute at a price that was, after all, given the current situation, relatively fair. For instance, with DwarfStar the…
Experimenting with local alt text generation in Firefox Nightly (opens on the source site)
Firefox 130 will introduce an experimental new capability to automatically generate alt-text for images using a fully private on-device AI model. The feature will be available as part of Firefox’s built-in PDF editor, and our end goal is to make it available in general browsing for users with screen readers. The post Experimenting with local alt text generation in Firefox Nightly appeared first on Mozilla Hacks - the Web developer blog.
Related topics
This page is generated automatically from the engineering blogs we follow. Every post links to its source, where it was published. See all sources.