Skip to content
banh / docs

Inference providers

Each run sends all decisions to one backend in a single batch, then applies your rules locally. Native Laya is the default. For remote inference, use Jev through the TypeSafe API.

Preset Provider Base URL Default model Token
laya native Runs on your CPU Local bundle Not needed
jev http https://api.typesafe.ai jev-latest Required

Use Jev

Section titled “Use Jev”

Set BANH_INFERENCE_TOKEN to your TypeSafe API key, download the support example, then:

Terminal window
banh run examples/support-triage.yaml --input examples/inputs/support-ticket.json \
  --provider http --model jev --json

Your input and questions are sent to TypeSafe. No model download or local server needed.

Jev model selection

Section titled “Jev model selection”

--model picks the family; --model-id picks the exact model. To pin a version:

Terminal window
banh run examples/support-triage.yaml --input examples/inputs/support-ticket.json \
  --provider http --model jev --model-id jev-1.13.0 --json

Or pin it in YAML: model: { provider: jev, model: jev-1.13.0 }. jev-latest and jev-preview also work. --model-id and BANH_INFERENCE_MODEL_ID override the YAML. See TypeSafe’s models. Predictions differ between versions; re-check thresholds when you switch.

Compatibility presets

Section titled “Compatibility presets”

Kept for existing integrations:

Preset Base URL Default model Token
laya with --provider http Your Laya server Server decides Optional
kev Your Kev server kev-latest Optional
Terminal window
banh run examples/support-triage.yaml --input examples/inputs/support-ticket.json \
  --provider http --model laya --base-url http://127.0.0.1:8000 --json

Configuration

Section titled “Configuration”

Precedence: flags, then environment, then the workflow’s model, then preset defaults. Laya defaults to native; Kev and Jev default to HTTP.

Flag Environment variable Default
--provider BANH_PROVIDER Native for Laya, HTTP for Kev/Jev
--model BANH_MODEL Workflow family, else laya
--base-url BANH_INFERENCE_BASE_URL Preset default
--model-id BANH_INFERENCE_MODEL_ID Workflow model, else preset default
--timeout-ms BANH_INFERENCE_TIMEOUT_MS 60000
(none) BANH_INFERENCE_TOKEN Unset
  • The token is only read from the environment.
  • --model-dir, --cache-dir and --revision are native-only; --model-id is HTTP-only.
  • HTTP runs don’t load native Laya or download weights, though the CLI package still installs it.
  • The server decides which weights run; results can differ between checkpoints.

HTTP behavior

Section titled “HTTP behavior”
  • Posts { state, questions } to <base>/v1/systemone and reads answers and usage. A base URL may include a path prefix or end in /v1.
  • Rejects URLs with query strings, fragments or credentials. HTTP and HTTPS work. Redirects aren’t followed.
  • One attempt by default. Library callers can allow two, retrying connection errors and 502/503/504. The timeout covers all attempts and reading the body.
  • Aborting cancels the request but doesn’t guarantee the remote server stops.

For library use, see TypeScript usage.

Managed inference with Banh Cloud

Section titled “Managed inference with Banh Cloud”

Banh Cloud runs deployed workflows on managed Jev using your plan’s allowance. No TypeSafe key needed. Cloud picks its own models; these settings only affect local banh run.