inline::reference
Description​
Reference implementation of batches API with KVStore persistence.
Configuration​
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
kvstore | utils.kvstore.config.RedisKVStoreConfig | utils.kvstore.config.SqliteKVStoreConfig | utils.kvstore.config.PostgresKVStoreConfig | utils.kvstore.config.MongoDBKVStoreConfig | No | sqlite | Configuration for the key-value store backend. |
max_concurrent_batches | <class 'int'> | No | 1 | Maximum number of concurrent batches to process simultaneously. |
max_concurrent_requests_per_batch | <class 'int'> | No | 10 | Maximum number of concurrent requests to process per batch. |
Sample Configuration​
kvstore:
type: sqlite
db_path: ${env.SQLITE_STORE_DIR:=~/.llama/dummy}/batches.db