Skip to main content
Version: Current

AI Types

  • AIChatContent

    Defines an interface that represents the contents of an AI chat message.

    A AIChatContent can be one of the following values:

  • AIChatMessage

    Defines an interface that represents a single chat message in a conversation with an AI.

    Members

    • author? string

      The name of the author of the message.

      This is required if the role is function.

    • content (string | AIChatContent[])

      The contents of the message. This can be a string, an array of objects which represent the contents of the message.

    • finishReason? string

      The reason why the message was finished.

    • role AIChatMessageRole

      The role of the message.

      • system means that the message was generated by the system. Useful for telling the AI how to behave while.
      • user means that the message was generated by the user.
      • assistant means that the message was generated by the AI assistant.
      • function means that the message contains the results of a function call.
  • AIChatMessageRole

    The role of a chat message.

    • system means that the message was generated by the system. Useful for telling the AI how to behave while.
    • user means that the message was generated by the user.
    • assistant means that the message was generated by the AI assistant.
    • function means that the message contains the results of a function call.

    A AIChatMessageRole can be one of the following values:

    • "system"
    • "user"
    • "assistant"
    • "function"
  • AIChatOptions

    Defines an interface that represents options for ai.chat(message, options).

    Members

    • endpoint? string

      The HTTP endpoint that the request should interface with.

    • frequencyPenalty? number

      The frequency penalty.

      Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

    • preferredModel? string

      The model that should be used.

      If not specified, then a default will be used.

      Currently, the following models are supported:

      • gpt-4
      • gpt-3.5-turbo
    • presencePenalty? number

      The presence penalty.

      Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

    • temperature? number

      The temperature that should be used.

      If not specified, then a default will be used.

    • topP? number

      The nucleus sampling probability.

  • AICreateOpenAIRealtimeSessionTokenFailure

    A unsuccessful response to a request to create a realtime session token using the OpenAI interface.

    Members

  • AICreateOpenAIRealtimeSessionTokenResult

    The response to a request to create a realtime session token using the OpenAI interface.

    A AICreateOpenAIRealtimeSessionTokenResult can be one of the following values:

  • AICreateOpenAIRealtimeSessionTokenSuccess

    A successful response to a request to create a realtime session token using the OpenAI interface.

    Members

    • clientSecret object

    • sessionId string

    • success true

  • AIDataContent

    Defines an interface that represents data that is passed to an AI chat model. This data can be used to represent images, videos, or other types of binary data that the model supports. Some models do not support this type of content.

    Members

    • base64 string

      The base 64 encoded data of the content.

    • mimeType string

      The MIME type of the content.

  • AIGenerateImageOptions

    Defines an interface that represents options for ai.generateImage(prompt, negativePrompt, options).

    Members

    • cfgScale? number

      How strictly the diffusion process adheres to the prompt text. Higher values keep the image closer to the prompt.

    • clipGuidancePreset? string

      The clip guidance preset.

    • height? number

      The desired height of the image(s) in pixels.

    • model? string

      The model that should be used to generate the image(s).

    • negativePrompt? string

      The description of what the generated image(s) should not look like.

    • numberOfImages? number

      The number of images that should be generated.

    • prompt string

      The description of what the generated image(s) should look like.

    • sampler? string

      The sampler to use for the diffusion process.

    • seed? number

      The random noise seed that should be used.

    • steps? number

      The number of diffusion steps to run.

    • stylePreset? string

      The style preset that should be used to guide the image model torwards a specific style.

    • width? number

      The desired width of the image(s) in pixels.

  • AIGenerateImageSuccess

    Defines an interface that represents a result from ai.generateImage(request, options).

    Members

  • AIGenerateSkyboxAction

    An event that is used to generate a skybox using AI.

    Members

    • negativePrompt? string

      The negative prompt to use for the skybox.

      Describes the things that you don't want the skybox to look like.

    • options AIGenerateSkyboxOptions

      The options that should be included in the request.

    • playerId? string

      The ID of the player that created this task. Set by remote action handlers when a task is recieved from a remote player.

    • prompt string

      The prompt to use for the skybox.

      Describes things that you want the skybox to look like.

    • taskId (string | number)

      The ID of the async task.

    • type "ai_generate_skybox"

      The type of the event. This helps determine how the event should be applied to the state.

    • uncopiable? boolean

      Whether the action can be structure cloned. If true, then the action should not be passed across message ports without special handling.

  • AIGenerateSkyboxBlockadeLabsOptions

    Options that are specific to Blockade Labs implementations for ai.generateSkybox(prompt, negativePrompt, options).

    Members

    • remixImagineId? number

      The ID of a previously generated skybox.

    • seed? number

      The random seed to use for generating the skybox.

    • skyboxStyleId? number

      The pre-defined style ID for the skybox.

  • AIGenerateSkyboxOptions

    Defines an interface that represents options for ai.generateSkybox(prompt, negativePrompt, options).

    Members

  • AIGenerateSkyboxRequest

    Defines an interface that represents a request for ai.generateSkybox(request).

    Members

    • negativePrompt? string

      The prompt that that describes what the generated skybox should avoid looking like.

    • options? AIGenerateSkyboxOptions

      The options that should be included in the request.

    • prompt string

      The prompt that describes what the generated skybox should look like.

  • AIGenerateSkyboxResult

    Defines an interface that represents the result from ai.generateSkybox(request).

    Members

    • fileUrl string

      The URL that the generated skybox is located at.

    • thumbnailUrl? string

      The URL that the thumbnail for the generated skybox is located at.

  • AIGeneratedImage

    Defines an interface that represents an AI generated image.

    Members

    • base64 string

      The base64 encoded image.

    • mimeType string

      The MIME Type of the image data.

    • seed? number

      The seed of the generated image.

    • url string

      The URL that can be used to display the image.

  • AISloydGenerateModelFailure

    A failed response to a request to generate a model using the Sloyd AI interface.

    Members

    • errorCode ("server_error" | "not_supported" | "invalid_record_key" | "unacceptable_request" | "record_not_found" | "not_authorized" | "action_not_supported" | "not_logged_in" | "subscription_limit_reached" | "not_subscribed" | "invalid_subscription_tier")

      The error code.

    • errorMessage string

      The error message.

    • reason? DenialReason

      The reason why the request was denied.

    • success false

  • AISloydGenerateModelOptions

    The options for generating a model using Sloyd AI.

    Members

    • baseModelId? string

      The ID of the model that the new model should be based on.

    • levelOfDetail? number

      The level of detail that should be used. Higher values will result in more detailed models. Should be between 0.01 and 1. Defaults to 0.5.

    • outputMimeType? ("model/gltf+json" | "model/gltf-binary")

      The MIME type that should be used for the model. If omitted, then "model/gltf+json" will be used.

    • prompt string

      The prompt to use for the model.

    • recordName? string

      The name of the record that should be used. If omitted, then the ID of the user will be used.

    • thumbnail? object

      The options for the thumbnail for the model. If omitted, then no thumbnail will be generated.

  • AISloydGenerateModelResponse

    The response to a request to generate a model using the Sloyd AI interface.

    A AISloydGenerateModelResponse can be one of the following values:

  • AISloydGenerateModelSuccess

    A successful response to a request to generate a model using the Sloyd AI interface.

    Members

    • confidence? number

      The confidence of the AI in the created model.

    • mimeType ("model/gltf+json" | "model/gltf-binary")

      The MIME type of the model.

    • modelData string

      The data for the model. If the mimeType is "model/gltf+json", then this will be a JSON string. If the mimeType is "model/gltf-binary", then this will be a base64 encoded string.

    • modelId string

      The ID of the model that was created.

    • name? string

      The name of the model.

    • success true

    • thumbnailBase64? string

      The base64 encoded thumbnail of the model.

  • AITextContent

    Defines an interface that represents text that is passed to an AI chat model.

    Members

    • text string

      The text of the content.

  • AIUrlContent

    Defines an interface that represents a URL that is passed to an AI chat model. This data can be used to represent images, videos, or other types of data that the model supports fetching. Some models do not support this type of content.

    Members

    • url string

      The URL that the content is available at.

  • RealtimeSessionRequest

    Defines a request to create a new realtime session.

    Members

    • inputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")

      The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate, single channel (mono), and little-endian byte order.

    • inputAudioNoiseReduction? object

      Configuration for input audio noise reduction. This can be set to null to turn off. Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.

    • inputAudioTranscription? object

      Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.

    • instructions? string

      The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.

      Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.

    • maxResponseOutputTokens? number

      Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

    • modalities? ("text" | "audio")[]

      The set of modalities the model can respond with. To disable audio, set this to ["text"].

    • model string

      The Realtime model used for this session.

    • outputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")

      The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, output audio is sampled at a rate of 24kHz.

    • temperature? number

      Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance.

    • toolChoice? string

      How the model chooses tools. Options are auto, none, required, or specify a function.

    • tools? object[]

      Tools (functions) available to the model.

    • turnDetection? object

      Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. Semantic VAD is more advanced and uses a turn detection model (in conjuction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.

    • voice? string

      The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, and verse.