Skip to main content
Version: Next

remote::hf::serverless

Description​

HuggingFace Inference API serverless provider for on-demand model inference.

Configuration​

FieldTypeRequiredDefaultDescription
huggingface_repo<class 'str'>NoThe model ID of the model on the Hugging Face Hub (e.g. 'meta-llama/Meta-Llama-3.1-70B-Instruct')
api_tokenpydantic.types.SecretStr | NoneNoYour Hugging Face user access token (will default to locally saved token if not provided)

Sample Configuration​

huggingface_repo: ${env.INFERENCE_MODEL}
api_token: ${env.HF_API_TOKEN}