Announcing Together Inference Engine – the fastest inference available

TL;DR

We are thrilled to announce the release of the FASTEST Voice LLM to date! Experience real-time speech streaming from text in 300ms or less. Dive in and test it using our Playground, available SDKs, or these Replit demos for both Nodejs and Python and a chatGPT integration.

Introduction

At PlayHT, our vision revolves around redefining human int... See more

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

Table of Contents

1. Introduction


2. Key LLM Serving Techniques


3. Dynamic SplitFuse: A Novel Prompt and Generation Composition Strategy


4. Performance Evaluation


5. DeepSpeed-FastGen: Implementation and Usage


6. Try out DeepSpeed-FastGen


7. Acknowledgements

1. Introduction

Large langu... See more

Setting up the necessary machine learning infrastructure to run these big models is another challenge. We need a dedicated model server for running model inference (using frameworks like Triton oder vLLM), powerful GPUs to run everything robustly, and configurability in our servers to make sure they're high throughput and low latency. Tuning the in... See more

Deep learning at the speed of light.

Luminal is a deep learning library that uses composable compilers to achieve high performance.

use luminal::prelude::*;

// Setup graph and tensors

let mut cx = Graph::new();

let a = cx.tensor().set([[1.0], [2.0], [3.0]]);

let b = cx.tensor().set([[1.0, 2.0, 3.0, 4.0]]);

// Do math...

let mut c = a.matmul(b).retrieve();

... See more