Skip to main content

OpenAI API Conformance Report

This page provides a detailed breakdown of Llama Stack's conformance to the OpenAI API specification. The conformance score increases as schema issues are fixed and missing properties are implemented.

Auto-generated

This documentation is auto-generated from the OpenAI API specification comparison.

Summary​

MetricValue
Overall Conformance Score84.5%
Endpoints Implemented28/114
Total Properties Checked2598
Schema/Type Issues272
Missing Properties130
Total Issues to Fix402

Category Scores​

Categories are sorted by conformance score (lowest first, needing most attention).

CategoryScorePropertiesIssuesMissing
Moderations17.0%53539
Batch36.3%1686641
Models53.3%1525
Completions54.3%46192
Files54.8%42118
Vector stores65.2%3109414
Embeddings71.4%1440
Responses83.6%2252611
Chat92.8%402236
Conversations98.0%1323224

Missing Endpoints​

The following OpenAI API endpoints are not yet implemented in Llama Stack:

/assistants​

  • /assistants
  • /assistants/{assistant_id}

/audio​

  • /audio/speech
  • /audio/transcriptions
  • /audio/translations

/chat​

  • /chat/completions/{completion_id}/messages

/evals​

  • /evals
  • /evals/{eval_id}
  • /evals/{eval_id}/runs
  • /evals/{eval_id}/runs/{run_id}
  • /evals/{eval_id}/runs/{run_id}/output_items
  • /evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}

/fine_tuning​

  • /fine_tuning/alpha/graders/run
  • /fine_tuning/alpha/graders/validate
  • /fine_tuning/checkpoints/{fine_tuned_model_checkpoint}/permissions
  • /fine_tuning/checkpoints/{fine_tuned_model_checkpoint}/permissions/{permission_id}
  • /fine_tuning/jobs
  • /fine_tuning/jobs/{fine_tuning_job_id}
  • /fine_tuning/jobs/{fine_tuning_job_id}/cancel
  • /fine_tuning/jobs/{fine_tuning_job_id}/checkpoints
  • /fine_tuning/jobs/{fine_tuning_job_id}/events
  • /fine_tuning/jobs/{fine_tuning_job_id}/pause
  • /fine_tuning/jobs/{fine_tuning_job_id}/resume

/images​

  • /images/edits
  • /images/generations
  • /images/variations

/organization​

  • /organization/audit_logs
  • /organization/certificates
  • /organization/certificates/activate
  • /organization/certificates/deactivate
  • /organization/certificates/{certificate_id}
  • /organization/costs
  • /organization/invites
  • /organization/invites/{invite_id}
  • /organization/projects
  • /organization/projects/{project_id}
  • /organization/projects/{project_id}/api_keys
  • /organization/projects/{project_id}/api_keys/{key_id}
  • /organization/projects/{project_id}/archive
  • /organization/projects/{project_id}/certificates
  • /organization/projects/{project_id}/certificates/activate
  • /organization/projects/{project_id}/certificates/deactivate
  • /organization/projects/{project_id}/rate_limits
  • /organization/projects/{project_id}/rate_limits/{rate_limit_id}
  • /organization/projects/{project_id}/service_accounts
  • /organization/projects/{project_id}/service_accounts/{service_account_id}
  • /organization/projects/{project_id}/users
  • /organization/projects/{project_id}/users/{user_id}
  • /organization/usage/audio_speeches
  • /organization/usage/audio_transcriptions
  • /organization/usage/code_interpreter_sessions
  • /organization/usage/completions
  • /organization/usage/embeddings
  • /organization/usage/images
  • /organization/usage/moderations
  • /organization/usage/vector_stores
  • /organization/users
  • /organization/users/{user_id}

/realtime​

  • /realtime/calls
  • /realtime/calls/{call_id}/accept
  • /realtime/calls/{call_id}/hangup
  • /realtime/calls/{call_id}/refer
  • /realtime/calls/{call_id}/reject
  • /realtime/client_secrets
  • /realtime/sessions
  • /realtime/transcription_sessions

/responses​

  • /responses/input_tokens

/threads​

  • /threads
  • /threads/runs
  • /threads/{thread_id}
  • /threads/{thread_id}/messages
  • /threads/{thread_id}/messages/{message_id}
  • /threads/{thread_id}/runs
  • /threads/{thread_id}/runs/{run_id}
  • /threads/{thread_id}/runs/{run_id}/cancel
  • /threads/{thread_id}/runs/{run_id}/steps
  • /threads/{thread_id}/runs/{run_id}/steps/{step_id}
  • /threads/{thread_id}/runs/{run_id}/submit_tool_outputs

/uploads​

  • /uploads
  • /uploads/{upload_id}/cancel
  • /uploads/{upload_id}/complete
  • /uploads/{upload_id}/parts

/videos​

  • /videos
  • /videos/{video_id}
  • /videos/{video_id}/content
  • /videos/{video_id}/remix

Detailed Category Breakdown​

Below is a detailed breakdown of conformance issues and missing properties for each category.

Moderations​

Score: 17.0% · Issues: 5 · Missing: 39

/moderations​

POST

Missing Properties (39)
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.categories.properties.violence/graphic
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.category_applied_input_types.properties.violence/graphic
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.harassment
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.harassment/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.hate
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.hate/threatening
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.illicit
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.illicit/violent
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm/instructions
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.self-harm/intent
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.sexual
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.sexual/minors
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.violence
  • responses.200.content.application/json.properties.results.items.properties.category_scores.properties.violence/graphic
Schema Issues (5)
PropertyIssues
requestBody.content.application/json.properties.inputUnion variants added: 2; Union variants removed: 3
requestBody.content.application/json.properties.modelNullable added (OpenAI non-nullable)
responses.200.content.application/json.properties.results.items.properties.categoriesType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.results.items.properties.category_applied_input_typesType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.results.items.properties.category_scoresType removed: ['object']; Union variants added: 2

Batch​

Score: 36.3% · Issues: 66 · Missing: 41

/batches​

GET

Missing Properties (10)
  • responses.200.content.application/json.properties.data.items.properties.errors.properties.data
  • responses.200.content.application/json.properties.data.items.properties.errors.properties.object
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.data.items.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.data.items.properties.usage.properties.total_tokens
Schema Issues (19)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['batch']
responses.200.content.application/json.properties.data.items.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.usageType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['list']; Default changed: None -> list

POST

Missing Properties (11)
  • requestBody.content.application/json.properties.output_expires_after
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (17)
PropertyIssues
requestBody.content.application/json.properties.completion_windowEnum removed: ['24h']
requestBody.content.application/json.properties.endpointEnum removed: ['/v1/responses', '/v1/chat/completions', '/v1/embeddings', '/v1/completions', '/v1/moderations']
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['batch']
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

/batches/{batch_id}​

GET

Missing Properties (10)
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (15)
PropertyIssues
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['batch']
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

/batches/{batch_id}/cancel​

POST

Missing Properties (10)
  • responses.200.content.application/json.properties.errors.properties.data
  • responses.200.content.application/json.properties.errors.properties.object
  • responses.200.content.application/json.properties.request_counts.properties.completed
  • responses.200.content.application/json.properties.request_counts.properties.failed
  • responses.200.content.application/json.properties.request_counts.properties.total
  • responses.200.content.application/json.properties.usage.properties.input_tokens
  • responses.200.content.application/json.properties.usage.properties.input_tokens_details
  • responses.200.content.application/json.properties.usage.properties.output_tokens
  • responses.200.content.application/json.properties.usage.properties.output_tokens_details
  • responses.200.content.application/json.properties.usage.properties.total_tokens
Schema Issues (15)
PropertyIssues
responses.200.content.application/json.properties.cancelled_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.cancelling_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.completed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.error_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.errorsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.expired_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.failed_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.finalizing_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.in_progress_atType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.modelType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['batch']
responses.200.content.application/json.properties.output_file_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.request_countsType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.usageType removed: ['object']; Union variants added: 2

Models​

Score: 53.3% · Issues: 2 · Missing: 5

/models​

GET

Missing Properties (1)
  • responses.200.content.application/json.properties.object
Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType added: ['object']
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['model']; Default changed: None -> model

/models/{model}​

GET

Missing Properties (4)
  • responses.200.content.application/json.properties.created
  • responses.200.content.application/json.properties.id
  • responses.200.content.application/json.properties.object
  • responses.200.content.application/json.properties.owned_by

Completions​

Score: 54.3% · Issues: 19 · Missing: 2

/completions​

POST

Missing Properties (2)
  • responses.200.content.application/json.properties.system_fingerprint
  • responses.200.content.application/json.properties.usage
Schema Issues (19)
PropertyIssues
requestBody.content.application/json.properties.best_ofType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.echoType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: False -> None
requestBody.content.application/json.properties.frequency_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 0 -> None
requestBody.content.application/json.properties.logit_biasType removed: ['object']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.logprobsType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.max_tokensType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 16 -> None
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 2
requestBody.content.application/json.properties.nType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.presence_penaltyType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 0 -> None
requestBody.content.application/json.properties.promptUnion variants added: 4; Union variants removed: 4
requestBody.content.application/json.properties.seedType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.stopNullable added (OpenAI non-nullable); Union variants added: 3; Union variants removed: 2
requestBody.content.application/json.properties.streamType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: False -> None
requestBody.content.application/json.properties.suffixType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.temperatureType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.top_pType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: 1 -> None
requestBody.content.application/json.properties.userType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.choices.items.properties.logprobsUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['text_completion']; Default changed: None -> text_completion

Files​

Score: 54.8% · Issues: 11 · Missing: 8

/files​

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.data.items.properties.status
  • responses.200.content.application/json.properties.data.items.properties.status_details
Schema Issues (4)
PropertyIssues
responses.200.content.application/json.properties.data.itemsType added: ['object']
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['file']; Default changed: None -> file
responses.200.content.application/json.properties.data.items.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']
responses.200.content.application/json.properties.objectDefault changed: None -> list

POST

Missing Properties (4)
  • requestBody.content.multipart/form-data.properties.expires_after.properties.anchor
  • requestBody.content.multipart/form-data.properties.expires_after.properties.seconds
  • responses.200.content.application/json.properties.status
  • responses.200.content.application/json.properties.status_details
Schema Issues (4)
PropertyIssues
requestBody.content.multipart/form-data.properties.expires_afterType removed: ['object']; Union variants added: 2
requestBody.content.multipart/form-data.properties.purposeEnum values removed: ['fine-tune', 'vision', 'user_data', 'evals']
responses.200.content.application/json.properties.objectEnum removed: ['file']; Default changed: None -> file
responses.200.content.application/json.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

/files/{file_id}​

DELETE

Schema Issues (1)
PropertyIssues
responses.200.content.application/json.properties.objectEnum removed: ['file']; Default changed: None -> file

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.status
  • responses.200.content.application/json.properties.status_details
Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.objectEnum removed: ['file']; Default changed: None -> file
responses.200.content.application/json.properties.purposeEnum values removed: ['assistants_output', 'batch_output', 'fine-tune', 'fine-tune-results', 'vision', 'user_data']

Vector stores​

Score: 65.2% · Issues: 94 · Missing: 14

/vector_stores​

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.data.items.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.data.items.properties.expires_after.properties.days
Schema Issues (10)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.data.items.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectDefault changed: None -> list

POST

Missing Properties (5)
  • requestBody.content.application/json.properties.description
  • requestBody.content.application/json.properties.expires_after.properties.anchor
  • requestBody.content.application/json.properties.expires_after.properties.days
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (10)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2; Union variants removed: 2
requestBody.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.file_idsType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}​

DELETE

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.deletedDefault changed: None -> True
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.deleted']; Default changed: None -> vector_store.deleted

GET

Missing Properties (2)
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (6)
PropertyIssues
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

POST

Missing Properties (2)
  • responses.200.content.application/json.properties.expires_after.properties.anchor
  • responses.200.content.application/json.properties.expires_after.properties.days
Schema Issues (8)
PropertyIssues
requestBody.content.application/json.properties.expires_afterUnion variants added: 2
requestBody.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.expires_afterType removed: ['object']; Union variants added: 2
responses.200.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store']; Default changed: None -> vector_store
responses.200.content.application/json.properties.statusEnum removed: ['expired', 'in_progress', 'completed']; Default changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/file_batches​

POST

Missing Properties (1)
  • requestBody.content.application/json.properties.files
Schema Issues (3)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2; Union variants removed: 2
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.files_batch']; Default changed: None -> vector_store.file_batch
responses.200.content.application/json.properties.statusDefault changed: None -> completed

/vector_stores/{vector_store_id}/file_batches/{batch_id}​

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.files_batch']; Default changed: None -> vector_store.file_batch
responses.200.content.application/json.properties.statusDefault changed: None -> completed

/vector_stores/{vector_store_id}/file_batches/{batch_id}/cancel​

POST

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.files_batch']; Default changed: None -> vector_store.file_batch
responses.200.content.application/json.properties.statusDefault changed: None -> completed

/vector_stores/{vector_store_id}/file_batches/{batch_id}/files​

GET

Schema Issues (10)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.chunking_strategyType removed: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.data.items.properties.statusDefault changed: None -> completed
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectDefault changed: None -> list

/vector_stores/{vector_store_id}/files​

GET

Schema Issues (10)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.chunking_strategyType removed: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.data.items.properties.statusDefault changed: None -> completed
responses.200.content.application/json.properties.data.items.properties.usage_bytesDefault changed: None -> 0
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectDefault changed: None -> list

POST

Schema Issues (7)
PropertyIssues
requestBody.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants added: 2; Union variants removed: 2
responses.200.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.statusDefault changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/files/{file_id}​

DELETE

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.deletedDefault changed: None -> True
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file.deleted']; Default changed: None -> vector_store.file.deleted

GET

Schema Issues (6)
PropertyIssues
responses.200.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.statusDefault changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

POST

Schema Issues (7)
PropertyIssues
requestBody.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.attributesType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.chunking_strategyType removed: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.last_errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file']; Default changed: None -> vector_store.file
responses.200.content.application/json.properties.statusDefault changed: None -> completed
responses.200.content.application/json.properties.usage_bytesDefault changed: None -> 0

/vector_stores/{vector_store_id}/files/{file_id}/content​

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.file_content.page']; Default changed: None -> vector_store.file_content.page

/vector_stores/{vector_store_id}/search​

POST

Missing Properties (2)
  • requestBody.content.application/json.properties.ranking_options.properties.ranker
  • requestBody.content.application/json.properties.ranking_options.properties.score_threshold
Schema Issues (7)
PropertyIssues
requestBody.content.application/json.properties.filtersUnion variants added: 2; Union variants removed: 2
requestBody.content.application/json.properties.max_num_resultsType removed: ['integer']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.queryUnion variants added: 1; Union variants removed: 1
requestBody.content.application/json.properties.ranking_optionsType removed: ['object']; Union variants added: 2
requestBody.content.application/json.properties.rewrite_queryType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.objectEnum removed: ['vector_store.search_results.page']; Default changed: None -> vector_store.search_results.page

Embeddings​

Score: 71.4% · Issues: 4 · Missing: 0

/embeddings​

POST

Schema Issues (4)
PropertyIssues
requestBody.content.application/json.properties.modelType added: ['string']; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.embeddingType removed: ['array']; Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['embedding']; Default changed: None -> embedding
responses.200.content.application/json.properties.objectEnum removed: ['list']; Default changed: None -> list

Responses​

Score: 83.6% · Issues: 26 · Missing: 11

/responses​

POST

Missing Properties (11)
  • requestBody.content.application/json.properties.background
  • requestBody.content.application/json.properties.frequency_penalty
  • requestBody.content.application/json.properties.presence_penalty
  • requestBody.content.application/json.properties.stream_options
  • requestBody.content.application/json.properties.top_logprobs
  • requestBody.content.application/json.properties.top_p
  • requestBody.content.application/x-www-form-urlencoded
  • responses.200.content.application/json.properties.background
  • responses.200.content.application/json.properties.frequency_penalty
  • responses.200.content.application/json.properties.presence_penalty
  • responses.200.content.application/json.properties.top_logprobs
Schema Issues (26)
PropertyIssues
requestBody.content.application/json.properties.includeType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
requestBody.content.application/json.properties.inputUnion variants added: 2; Union variants removed: 2
requestBody.content.application/json.properties.modelType added: ['string']; Nullable removed (OpenAI nullable); Union variants removed: 2
requestBody.content.application/json.properties.parallel_tool_callsDefault changed: None -> True
requestBody.content.application/json.properties.reasoningUnion variants added: 1; Union variants removed: 1
requestBody.content.application/json.properties.service_tierUnion variants added: 2
requestBody.content.application/json.properties.storeType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: None -> True
requestBody.content.application/json.properties.streamType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: None -> False
requestBody.content.application/json.properties.textUnion variants added: 1; Union variants removed: 1
requestBody.content.application/json.properties.tool_choiceUnion variants added: 2; Union variants removed: 1
requestBody.content.application/json.properties.truncationUnion variants added: 2
responses.200.content.application/json.properties.errorUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.incomplete_detailsUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.metadataUnion variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['response']
responses.200.content.application/json.properties.output.itemsUnion variants added: 7; Union variants removed: 4
responses.200.content.application/json.properties.parallel_tool_callsType removed: ['boolean']; Nullable added (OpenAI non-nullable); Union variants added: 2; Default changed: None -> True
responses.200.content.application/json.properties.reasoningUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.service_tierType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.temperatureType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.textType added: ['object']
responses.200.content.application/json.properties.tool_choiceUnion variants added: 3
responses.200.content.application/json.properties.toolsType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.top_pType removed: ['number']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.truncationUnion variants added: 2
responses.200.content.application/json.properties.usageUnion variants added: 1; Union variants removed: 1

Chat​

Score: 92.8% · Issues: 23 · Missing: 6

/chat/completions​

GET

Missing Properties (1)
  • parameters.query.metadata
Schema Issues (7)
PropertyIssues
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.logprobsUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.function_call.properties.argumentsType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.function_call.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.roleEnum removed: ['assistant']; Default changed: None -> assistant
responses.200.content.application/json.properties.data.items.properties.choices.items.properties.message.properties.tool_calls.itemsType added: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.data.items.properties.objectEnum removed: ['chat.completion']; Default changed: None -> chat.completion
responses.200.content.application/json.properties.objectEnum removed: ['list']

POST

Missing Properties (4)
  • responses.200.content.application/json.properties.system_fingerprint
  • responses.200.content.text/event-stream.properties.choices.items.properties.logprobs.properties.content
  • responses.200.content.text/event-stream.properties.choices.items.properties.logprobs.properties.refusal
  • responses.200.content.text/event-stream.properties.system_fingerprint
Schema Issues (10)
PropertyIssues
responses.200.content.application/json.properties.choices.items.properties.logprobsUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.choices.items.properties.message.properties.function_call.properties.argumentsType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.choices.items.properties.message.properties.function_call.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.choices.items.properties.message.properties.roleEnum removed: ['assistant']; Default changed: None -> assistant
responses.200.content.application/json.properties.choices.items.properties.message.properties.tool_calls.itemsType added: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.objectEnum removed: ['chat.completion']; Default changed: None -> chat.completion
responses.200.content.text/event-stream.properties.choices.items.properties.delta.properties.roleEnum removed: ['developer', 'system', 'user', 'assistant', 'tool']; Type removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.text/event-stream.properties.choices.items.properties.delta.properties.tool_callsType removed: ['array']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.text/event-stream.properties.choices.items.properties.logprobsType removed: ['object']; Union variants added: 2
responses.200.content.text/event-stream.properties.objectEnum removed: ['chat.completion.chunk']; Default changed: None -> chat.completion.chunk

/chat/completions/{completion_id}​

GET

Missing Properties (1)
  • responses.200.content.application/json.properties.system_fingerprint
Schema Issues (6)
PropertyIssues
responses.200.content.application/json.properties.choices.items.properties.logprobsUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.choices.items.properties.message.properties.function_call.properties.argumentsType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.choices.items.properties.message.properties.function_call.properties.nameType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.choices.items.properties.message.properties.roleEnum removed: ['assistant']; Default changed: None -> assistant
responses.200.content.application/json.properties.choices.items.properties.message.properties.tool_calls.itemsType added: ['object']; Union variants removed: 2
responses.200.content.application/json.properties.objectEnum removed: ['chat.completion']; Default changed: None -> chat.completion

Conversations​

Score: 98.0% · Issues: 22 · Missing: 4

/conversations​

POST

Schema Issues (3)
PropertyIssues
requestBody.content.application/json.properties.metadataUnion variants added: 1; Union variants removed: 1
responses.200.content.application/json.properties.metadataUnion variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['conversation']

/conversations/{conversation_id}​

DELETE

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.deletedDefault changed: None -> True
responses.200.content.application/json.properties.objectEnum removed: ['conversation.deleted']

GET

Schema Issues (2)
PropertyIssues
responses.200.content.application/json.properties.metadataUnion variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['conversation']

POST

Schema Issues (3)
PropertyIssues
requestBody.content.application/json.properties.metadataType added: ['object']; Nullable removed (OpenAI nullable); Union variants removed: 2
responses.200.content.application/json.properties.metadataUnion variants added: 2
responses.200.content.application/json.properties.objectEnum removed: ['conversation']

/conversations/{conversation_id}/items​

GET

Schema Issues (5)
PropertyIssues
responses.200.content.application/json.properties.data.itemsUnion variants removed: 22
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectType added: ['string']; Default changed: None -> list

POST

Missing Properties (1)
  • parameters.query.include
Schema Issues (6)
PropertyIssues
requestBody.content.application/json.properties.items.itemsUnion variants removed: 3
responses.200.content.application/json.properties.data.itemsUnion variants removed: 22
responses.200.content.application/json.properties.first_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.has_moreDefault changed: None -> False
responses.200.content.application/json.properties.last_idType removed: ['string']; Nullable added (OpenAI non-nullable); Union variants added: 2
responses.200.content.application/json.properties.objectType added: ['string']; Default changed: None -> list

/conversations/{conversation_id}/items/{item_id}​

DELETE

Missing Properties (2)
  • responses.200.content.application/json.properties.created_at
  • responses.200.content.application/json.properties.metadata
Schema Issues (1)
PropertyIssues
responses.200.content.application/json.properties.objectEnum removed: ['conversation']; Default changed: conversation -> conversation.item.deleted

GET

Missing Properties (1)
  • parameters.query.include

How to Improve Conformance​

To improve conformance scores:

  1. Fix Schema Issues: Update Pydantic models in src/llama_stack_api/ to match OpenAI's schema
  2. Add Missing Properties: Implement missing fields in response models
  3. Add Missing Endpoints: Implement endpoints listed in the Missing Endpoints section

Run the coverage analyzer to check your progress:

python scripts/openai_coverage.py --update

Then regenerate this documentation:

python scripts/generate_openai_coverage_docs.py