mirror of
https://github.com/pykeio/ort
synced 2026-04-25 16:34:55 +02:00
78 lines
3.2 KiB
Plaintext
78 lines
3.2 KiB
Plaintext
---
|
|
title: Alternative backends
|
|
---
|
|
|
|
# Alternative backends
|
|
|
|
import { Callout, Steps } from 'nextra/components';
|
|
import Ort from '../../components/Ort';
|
|
|
|
Since [ONNX Runtime](https://onnxruntime.ai/) is written in C++, linking troubles often arise when attempting to use it in a Rust project--especially with WASM. `v2.0.0-rc.12` of <Ort/> introduced support for **alternative backends**: ONNX runtimes that aren't *the* ONNX Runtime.
|
|
|
|
As the Rust ML scene has evolved, many exciting new inference engines supporting ONNX models have popped up, like 🤗 Hugging Face's [`candle`](https://github.com/huggingface/candle), [Burn](https://github.com/tracel-ai/burn), and [`tract`](https://github.com/sonos/tract). These libraries, being written in pure Rust, play much nicer when it comes to linking, and often support any platform Rust's standard library does (which is a lot more than ONNX Runtime!) They're also, of course, memory safe and 🦀blazingly🔥fast🚀
|
|
|
|
<Ort/> alternative backends are simply wrappers that implements the ONNX Runtime C API over another crate. Because they implement the same exact API as ONNX Runtime, using them in <Ort/> is as simple as adding one line of code!
|
|
|
|
## Using an alternative backend
|
|
|
|
<Callout type='warning'>
|
|
Alternative backends are experimental, and are constantly changing and growing -- use them at your own risk!
|
|
|
|
We may not be able to provide the same level of support for different backends as we do with ONNX Runtime.
|
|
</Callout>
|
|
|
|
<Steps>
|
|
|
|
### Install the alternative backend
|
|
We'll use [`ort-tract`](/backends/tract) for this example.
|
|
|
|
```toml filename="Cargo.toml"
|
|
[dependencies]
|
|
ort-tract = "0.3.0+0.22"
|
|
...
|
|
```
|
|
|
|
### Enable the `alternative-backend` feature
|
|
This instructs <Ort/> to not try to download/link to ONNX Runtime.
|
|
|
|
```toml filename="Cargo.toml"
|
|
[dependencies.ort]
|
|
version = "=2.0.0-rc.12"
|
|
default-features = false # Disables the `download-binaries` feature since we don't need it
|
|
features = [
|
|
"std",
|
|
"ndarray",
|
|
"alternative-backend"
|
|
]
|
|
```
|
|
|
|
### Initialize the backend
|
|
Use [`ort::set_api`](https://docs.rs/ort/latest/ort/fn.set_api.html) to use the crate's API implementation (replacing `ort_tract` with whichever backend crate you choose to use):
|
|
|
|
```rs
|
|
fn main() {
|
|
// This should run as early in your application as possible - before you ever use `ort`!
|
|
ort::set_api(ort_tract::api());
|
|
}
|
|
```
|
|
|
|
### Done!
|
|
<Callout type='info'>
|
|
Be sure to check each backend's docs page to see which APIs are and are not supported.
|
|
</Callout>
|
|
|
|
</Steps>
|
|
|
|
## Available backends
|
|
<Ort/> currently has the following backends:
|
|
|
|
- [`ort-candle`](/backends/candle), based on [🤗 Hugging Face `candle`](https://github.com/huggingface/candle)
|
|
- 🔷 **Supports**: CPU, CUDA (though not available via `ort-candle` right now), WebAssembly
|
|
- ⚠️ Limited operator support; though most transformer models have good support.
|
|
- [`ort-tract`](/backends/tract), based on [`tract`](https://github.com/sonos/tract)
|
|
- 🔷 **Supports**: CPU, WebAssembly
|
|
- ✅ [Great operator support](https://github.com/sonos/tract?tab=readme-ov-file#onnx)
|
|
- [`ort-web`](/backends/web) runs ONNX Runtime in the web
|
|
- 🔷 **Supports**: WebAssembly (with WebGL & WebGPU backends!)
|
|
- ✅ Great operator support - *it's the full ONNX Runtime!*
|