AI Types

• AIChatContent

  Defines an interface that represents the contents of an AI chat message.

  A AIChatContent can be one of the following values:

  • AITextContent
  • AIDataContent
  • AIUrlContent

• AIChatMessage

  Defines an interface that represents a single chat message in a conversation with an AI.

  Members

  • Index Signature

    The provider-native response payloads keyed by provider name.

    For example, OpenAI responses are placed in the openai property, Anthropic responses are placed in anthropic, and custom providers use their configured provider name.

    [provider: string]: unknown

  • author? string

    The name of the author of the message.

    This is required if the role is function.

  • content (string | AIChatContent[])

    The contents of the message. This can be a string, an array of objects which represent the contents of the message.

  • finishReason? string

    The reason why the message was finished.

  • role AIChatMessageRole

    The role of the message.

    • system means that the message was generated by the system. Useful for telling the AI how to behave while.
    • user means that the message was generated by the user.
    • assistant means that the message was generated by the AI assistant.
    • function means that the message contains the results of a function call.

• AIChatMessageRole

  The role of a chat message.

  • system means that the message was generated by the system. Useful for telling the AI how to behave while.
  • user means that the message was generated by the user.
  • assistant means that the message was generated by the AI assistant.
  • function means that the message contains the results of a function call.

  A AIChatMessageRole can be one of the following values:

  • "system"
  • "user"
  • "assistant"
  • "function"

• AIChatOptions

  Defines an interface that represents options for ai.chat(message, options).

  Members

  • enableCaching? boolean

    Whether prompt caching should be enabled for providers that support it.

  • endpoint? string

    The HTTP endpoint that the request should interface with.

  • frequencyPenalty? number

    The frequency penalty.

    Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

  • preferredModel? string

    The model that should be used.

    If not specified, then a default will be used.

  • presencePenalty? number

    The presence penalty.

    Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

  • recordName? string

    The name of the record that should be used. If omitted, then the ID of the user will be used.

  • temperature? number

    The temperature that should be used.

    If not specified, then a default will be used.

  • topP? number

    The nucleus sampling probability.

• AICreateOpenAIRealtimeSessionTokenFailure

  A unsuccessful response to a request to create a realtime session token using the OpenAI interface.

  Members

  • errorCode KnownErrorCodes

  • errorMessage string

  • success false

• AICreateOpenAIRealtimeSessionTokenResult

  The response to a request to create a realtime session token using the OpenAI interface.

  A AICreateOpenAIRealtimeSessionTokenResult can be one of the following values:

  • AICreateOpenAIRealtimeSessionTokenSuccess
  • AICreateOpenAIRealtimeSessionTokenFailure

• AICreateOpenAIRealtimeSessionTokenSuccess

  A successful response to a request to create a realtime session token using the OpenAI interface.

  Members

  • clientSecret object

  • sessionId string

  • success true

• AIDataContent

  Defines an interface that represents data that is passed to an AI chat model. This data can be used to represent images, videos, or other types of binary data that the model supports. Some models do not support this type of content.

  Members

  • base64 string

    The base 64 encoded data of the content.

  • mimeType string

    The MIME type of the content.

• AIGenerateImageOptions

  Defines an interface that represents options for ai.generateImage(prompt, negativePrompt, options).

  Members

  • cfgScale? number

    How strictly the diffusion process adheres to the prompt text. Higher values keep the image closer to the prompt.

  • clipGuidancePreset? string

    The clip guidance preset.

  • height? number

    The desired height of the image(s) in pixels.

  • model? string

    The model that should be used to generate the image(s).

  • negativePrompt? string

    The description of what the generated image(s) should not look like.

  • numberOfImages? number

    The number of images that should be generated.

  • prompt string

    The description of what the generated image(s) should look like.

  • recordName? string

    The name of the record that should be used. If omitted, then the ID of the user will be used.

  • sampler? string

    The sampler to use for the diffusion process.

  • seed? number

    The random noise seed that should be used.

  • steps? number

    The number of diffusion steps to run.

  • stylePreset? string

    The style preset that should be used to guide the image model torwards a specific style.

  • width? number

    The desired width of the image(s) in pixels.

• AIGenerateImageSuccess

  Defines an interface that represents a result from ai.generateImage(request, options).

  Members

  • images AIGeneratedImage[]

    The list of images that were generated.

  • success true

• AIGenerateSkyboxAction

  An event that is used to generate a skybox using AI.

  Members

  • negativePrompt? string

    The negative prompt to use for the skybox.

    Describes the things that you don't want the skybox to look like.

  • options AIGenerateSkyboxOptions

    The options that should be included in the request.

  • playerId? string

    The ID of the player that created this task. Set by remote action handlers when a task is recieved from a remote player.

  • prompt string

    The prompt to use for the skybox.

    Describes things that you want the skybox to look like.

  • taskId (string | number)

    The ID of the async task.

  • type "ai_generate_skybox"

    The type of the event. This helps determine how the event should be applied to the state.

  • uncopiable? boolean

    Whether the action can be structure cloned. If true, then the action should not be passed across message ports without special handling.

• AIGenerateSkyboxBlockadeLabsOptions

  Options that are specific to Blockade Labs implementations for ai.generateSkybox(prompt, negativePrompt, options).

  Members

  • remixImagineId? number

    The ID of a previously generated skybox.

  • seed? number

    The random seed to use for generating the skybox.

  • skyboxStyleId? number

    The pre-defined style ID for the skybox.

• AIGenerateSkyboxOptions

  Defines an interface that represents options for ai.generateSkybox(prompt, negativePrompt, options).

  Members

  • blockadeLabs? AIGenerateSkyboxBlockadeLabsOptions

    Options that are specific to blockade-labs.

  • endpoint? string

    The HTTP endpoint that the request should interface with.

  • recordName? string

    The name of the record that should be used. If omitted, then the ID of the user will be used.

• AIGenerateSkyboxRequest

  Defines an interface that represents a request for ai.generateSkybox(request).

  Members

  • negativePrompt? string

    The prompt that that describes what the generated skybox should avoid looking like.

  • options? AIGenerateSkyboxOptions

    The options that should be included in the request.

  • prompt string

    The prompt that describes what the generated skybox should look like.

• AIGenerateSkyboxResult

  Defines an interface that represents the result from ai.generateSkybox(request).

  Members

  • fileUrl string

    The URL that the generated skybox is located at.

  • thumbnailUrl? string

    The URL that the thumbnail for the generated skybox is located at.

• AIGeneratedImage

  Defines an interface that represents an AI generated image.

  Members

  • base64 string

    The base64 encoded image.

  • mimeType string

    The MIME Type of the image data.

  • seed? number

    The seed of the generated image.

  • url string

    The URL that can be used to display the image.

• AIListChatModelsOptions

  Defines options for ai.listChatModels(options).

  Members

  • endpoint? string

    The HTTP endpoint that the request should interface with.

  • recordName? string

    The name of the record that should be used. If omitted, then the user's record is used.

• AISloydGenerateModelFailure

  A failed response to a request to generate a model using the Sloyd AI interface.

  Members

  • errorCode ("not_logged_in" | "not_supported" | "record_not_found" | "invalid_record_key" | "unacceptable_request" | "server_error" | "not_authorized" | "not_subscribed" | "invalid_subscription_tier" | "subscription_limit_reached" | "action_not_supported")

    The error code.

  • errorMessage string

    The error message.

  • reason? DenialReason

    The reason why the request was denied.

  • success false

• AISloydGenerateModelOptions

  The options for generating a model using Sloyd AI.

  Members

  • baseModelId? string

    The ID of the model that the new model should be based on.

  • levelOfDetail? number

    The level of detail that should be used. Higher values will result in more detailed models. Should be between 0.01 and 1. Defaults to 0.5.

  • outputMimeType? ("model/gltf+json" | "model/gltf-binary")

    The MIME type that should be used for the model. If omitted, then "model/gltf+json" will be used.

  • prompt string

    The prompt to use for the model.

  • recordName? string

    The name of the record that should be used. If omitted, then the ID of the user will be used.

  • thumbnail? object

    The options for the thumbnail for the model. If omitted, then no thumbnail will be generated.

• AISloydGenerateModelResponse

  The response to a request to generate a model using the Sloyd AI interface.

  A AISloydGenerateModelResponse can be one of the following values:

  • AISloydGenerateModelSuccess
  • AISloydGenerateModelFailure

• AISloydGenerateModelSuccess

  A successful response to a request to generate a model using the Sloyd AI interface.

  Members

  • confidence? number

    The confidence of the AI in the created model.

  • mimeType ("model/gltf+json" | "model/gltf-binary")

    The MIME type of the model.

  • modelData string

    The data for the model. If the mimeType is "model/gltf+json", then this will be a JSON string. If the mimeType is "model/gltf-binary", then this will be a base64 encoded string.

  • modelId string

    The ID of the model that was created.

  • name? string

    The name of the model.

  • success true

  • thumbnailBase64? string

    The base64 encoded thumbnail of the model.

• AITextContent

  Defines an interface that represents text that is passed to an AI chat model.

  Members

  • text string

    The text of the content.

• AIUrlContent

  Defines an interface that represents a URL that is passed to an AI chat model. This data can be used to represent images, videos, or other types of data that the model supports fetching. Some models do not support this type of content.

  Members

  • url string

    The URL that the content is available at.

• ListedChatModel

  Defines an interface that represents a listed chat model.

  Members

  • isDefault? boolean

    Whether this is the default model.

  • name string

    The name of the model.

  • provider string

    The provider of the model.

• RealtimeSessionRequest

  Defines a request to create a new realtime session.

  Members

  • inputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")

    The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate, single channel (mono), and little-endian byte order.

  • inputAudioNoiseReduction? object

    Configuration for input audio noise reduction. This can be set to null to turn off. Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.

  • inputAudioTranscription? object

    Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.

  • instructions? string

    The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.

    Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.

  • maxResponseOutputTokens? number

    Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

  • modalities? ("text" | "audio")[]

    The set of modalities the model can respond with. To disable audio, set this to ["text"].

  • model string

    The Realtime model used for this session.

  • outputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")

    The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, output audio is sampled at a rate of 24kHz.

  • temperature? number

    Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance.

  • toolChoice? string

    How the model chooses tools. Options are auto, none, required, or specify a function.

  • tools? object[]

    Tools (functions) available to the model.

  • turnDetection? object

    Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. Semantic VAD is more advanced and uses a turn detection model (in conjuction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.

  • voice? string

    The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, and verse.