AI Types
AIChatContent
Defines an interface that represents the contents of an AI chat message.
A AIChatContent can be one of the following values:
AIChatMessage
Defines an interface that represents a single chat message in a conversation with an AI.
Members
author? string
The name of the author of the message.
This is required if the role is
function
.content (string | AIChatContent[])
The contents of the message. This can be a string, an array of objects which represent the contents of the message.
finishReason? string
The reason why the message was finished.
role AIChatMessageRole
The role of the message.
system
means that the message was generated by the system. Useful for telling the AI how to behave while.user
means that the message was generated by the user.assistant
means that the message was generated by the AI assistant.function
means that the message contains the results of a function call.
AIChatMessageRole
The role of a chat message.
system
means that the message was generated by the system. Useful for telling the AI how to behave while.user
means that the message was generated by the user.assistant
means that the message was generated by the AI assistant.function
means that the message contains the results of a function call.
A AIChatMessageRole can be one of the following values:
"system"
"user"
"assistant"
"function"
AIChatOptions
Defines an interface that represents options for
ai.chat(message, options)
.Members
endpoint? string
The HTTP endpoint that the request should interface with.
frequencyPenalty? number
The frequency penalty.
Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.
preferredModel? string
The model that should be used.
If not specified, then a default will be used.
Currently, the following models are supported:
gpt-4
gpt-3.5-turbo
presencePenalty? number
The presence penalty.
Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.
temperature? number
The temperature that should be used.
If not specified, then a default will be used.
topP? number
The nucleus sampling probability.
AICreateOpenAIRealtimeSessionTokenResult
The response to a request to create a realtime session token using the OpenAI interface.
A AICreateOpenAIRealtimeSessionTokenResult can be one of the following values:
AIGenerateImageOptions
Defines an interface that represents options for
ai.generateImage(prompt, negativePrompt, options)
.Members
cfgScale? number
How strictly the diffusion process adheres to the prompt text. Higher values keep the image closer to the prompt.
clipGuidancePreset? string
The clip guidance preset.
height? number
The desired height of the image(s) in pixels.
model? string
The model that should be used to generate the image(s).
negativePrompt? string
The description of what the generated image(s) should not look like.
numberOfImages? number
The number of images that should be generated.
prompt string
The description of what the generated image(s) should look like.
sampler? string
The sampler to use for the diffusion process.
seed? number
The random noise seed that should be used.
steps? number
The number of diffusion steps to run.
stylePreset? string
The style preset that should be used to guide the image model torwards a specific style.
width? number
The desired width of the image(s) in pixels.
AIGenerateImageSuccess
Defines an interface that represents a result from
ai.generateImage(request, options)
.Members
images AIGeneratedImage[]
The list of images that were generated.
success true
AIGenerateSkyboxAction
An event that is used to generate a skybox using AI.
Members
negativePrompt? string
The negative prompt to use for the skybox.
Describes the things that you don't want the skybox to look like.
options AIGenerateSkyboxOptions
The options that should be included in the request.
playerId? string
The ID of the player that created this task. Set by remote action handlers when a task is recieved from a remote player.
prompt string
The prompt to use for the skybox.
Describes things that you want the skybox to look like.
taskId (string | number)
The ID of the async task.
type "ai_generate_skybox"
The type of the event. This helps determine how the event should be applied to the state.
uncopiable? boolean
Whether the action can be structure cloned. If true, then the action should not be passed across message ports without special handling.
AIGenerateSkyboxBlockadeLabsOptions
Options that are specific to Blockade Labs implementations for
ai.generateSkybox(prompt, negativePrompt, options)
.Members
AIGenerateSkyboxOptions
Defines an interface that represents options for
ai.generateSkybox(prompt, negativePrompt, options)
.Members
blockadeLabs? AIGenerateSkyboxBlockadeLabsOptions
Options that are specific to blockade-labs.
endpoint? string
The HTTP endpoint that the request should interface with.
AIGenerateSkyboxRequest
Defines an interface that represents a request for
ai.generateSkybox(request)
.Members
negativePrompt? string
The prompt that that describes what the generated skybox should avoid looking like.
options? AIGenerateSkyboxOptions
The options that should be included in the request.
prompt string
The prompt that describes what the generated skybox should look like.
AIGenerateSkyboxResult
Defines an interface that represents the result from
ai.generateSkybox(request)
.Members
AISloydGenerateModelFailure
A failed response to a request to generate a model using the Sloyd AI interface.
Members
errorCode ("server_error" | "not_supported" | "invalid_record_key" | "unacceptable_request" | "record_not_found" | "not_authorized" | "action_not_supported" | "not_logged_in" | "subscription_limit_reached" | "not_subscribed" | "invalid_subscription_tier")
The error code.
errorMessage string
The error message.
reason? DenialReason
The reason why the request was denied.
success false
AISloydGenerateModelOptions
The options for generating a model using Sloyd AI.
Members
baseModelId? string
The ID of the model that the new model should be based on.
levelOfDetail? number
The level of detail that should be used. Higher values will result in more detailed models. Should be between
0.01
and1
. Defaults to0.5
.outputMimeType? ("model/gltf+json" | "model/gltf-binary")
The MIME type that should be used for the model. If omitted, then "model/gltf+json" will be used.
prompt string
The prompt to use for the model.
recordName? string
The name of the record that should be used. If omitted, then the ID of the user will be used.
thumbnail? object
The options for the thumbnail for the model. If omitted, then no thumbnail will be generated.
AISloydGenerateModelResponse
The response to a request to generate a model using the Sloyd AI interface.
A AISloydGenerateModelResponse can be one of the following values:
AISloydGenerateModelSuccess
A successful response to a request to generate a model using the Sloyd AI interface.
Members
confidence? number
The confidence of the AI in the created model.
mimeType ("model/gltf+json" | "model/gltf-binary")
The MIME type of the model.
modelData string
The data for the model. If the mimeType is "model/gltf+json", then this will be a JSON string. If the mimeType is "model/gltf-binary", then this will be a base64 encoded string.
modelId string
The ID of the model that was created.
name? string
The name of the model.
success true
thumbnailBase64? string
The base64 encoded thumbnail of the model.
AIUrlContent
RealtimeSessionRequest
Defines a request to create a new realtime session.
Members
inputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")
The format of input audio. Options are
pcm16
,g711_ulaw
, org711_alaw
. Forpcm16
, input audio must be 16-bit PCM at a 24kHz sample rate, single channel (mono), and little-endian byte order.inputAudioNoiseReduction? object
Configuration for input audio noise reduction. This can be set to null to turn off. Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.
inputAudioTranscription? object
Configuration for input audio transcription, defaults to off and can be set to
null
to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.instructions? string
The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.
Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
maxResponseOutputTokens? number
Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.
modalities? ("text" | "audio")[]
The set of modalities the model can respond with. To disable audio, set this to ["text"].
model string
The Realtime model used for this session.
outputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")
The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, output audio is sampled at a rate of 24kHz.
temperature? number
Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance.
toolChoice? string
How the model chooses tools. Options are
auto
,none
,required
, or specify a function.tools? object[]
Tools (functions) available to the model.
turnDetection? object
Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to
null
to turn off, in which case the client must manually trigger model response. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. Semantic VAD is more advanced and uses a turn detection model (in conjuction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.voice? string
The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are
alloy
,ash
,ballad
,coral
,echo
,fable
,onyx
,nova
,sage
,shimmer
, andverse
.