Open Source Summit North America 2025

Triton Inference Server has long been a reliable tool for AI model deployment. As Generative AI unfolds its transformative potential, Triton continues to evolve, offering both time-tested features and new capabilities tailored for large language models and more complex agentic workflows.

This session explores how Triton’s core strengths continue to play a crucial role in optimizing generative AI deployments. These include its robust multi-framework support, dynamic batching, concurrent model execution, and the capability to deploy complex inference pipelines through model ensembling and business logic scripting.

We’ll also cover recent enhancements such as OpenAI compatible frontend, allowing easy integration with existing OpenAI-based applications; Python-based backends to standardize the deployment of Python models without writing a custom C++ backend; Triton CLI to simplify model deployment and management; distributed inference enhancements for Data Center scale.

Throughout the presentation, we’ll share practical examples and best practices, equipping our listeners with the knowledge to effectively use Triton Inference Server to optimize AI workloads’ performance and efficiency.