Skip to main content
Version: Current

AI Types

  • AIChatContent​

    Defines an interface that represents the contents of an AI chat message.

    A AIChatContent can be one of the following values:

  • AIChatMessage​

    Defines an interface that represents a single chat message in a conversation with an AI.

    Members​

    • author? string​

      The name of the author of the message.

      This is required if the role is function.

    • content (string | AIChatContent[])​

      The contents of the message. This can be a string, an array of objects which represent the contents of the message.

    • finishReason? string​

      The reason why the message was finished.

    • role AIChatMessageRole​

      The role of the message.

      • system means that the message was generated by the system. Useful for telling the AI how to behave while.
      • user means that the message was generated by the user.
      • assistant means that the message was generated by the AI assistant.
      • function means that the message contains the results of a function call.

  • AIChatMessageRole​

    The role of a chat message.

    • system means that the message was generated by the system. Useful for telling the AI how to behave while.
    • user means that the message was generated by the user.
    • assistant means that the message was generated by the AI assistant.
    • function means that the message contains the results of a function call.

    A AIChatMessageRole can be one of the following values:

    • "system"
    • "user"
    • "assistant"
    • "function"

  • AIChatOptions​

    Defines an interface that represents options for ai.chat(message, options).

    Members​

    • endpoint? string​

      The HTTP endpoint that the request should interface with.

    • frequencyPenalty? number​

      The frequency penalty.

      Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.

    • preferredModel? string​

      The model that should be used.

      If not specified, then a default will be used.

      Currently, the following models are supported:

      • gpt-4
      • gpt-3.5-turbo

    • presencePenalty? number​

      The presence penalty.

      Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

    • temperature? number​

      The temperature that should be used.

      If not specified, then a default will be used.

    • topP? number​

      The nucleus sampling probability.

  • AICreateOpenAIRealtimeSessionTokenFailure​

    A unsuccessful response to a request to create a realtime session token using the OpenAI interface.

    Members​

  • AICreateOpenAIRealtimeSessionTokenResult​

    The response to a request to create a realtime session token using the OpenAI interface.

    A AICreateOpenAIRealtimeSessionTokenResult can be one of the following values:

  • AICreateOpenAIRealtimeSessionTokenSuccess​

    A successful response to a request to create a realtime session token using the OpenAI interface.

    Members​

    • clientSecret object​

    • sessionId string​

    • success true​

  • AIDataContent​

    Defines an interface that represents data that is passed to an AI chat model. This data can be used to represent images, videos, or other types of binary data that the model supports. Some models do not support this type of content.

    Members​

    • base64 string​

      The base 64 encoded data of the content.

    • mimeType string​

      The MIME type of the content.

  • AIGenerateImageOptions​

    Defines an interface that represents options for ai.generateImage(prompt, negativePrompt, options).

    Members​

    • cfgScale? number​

      How strictly the diffusion process adheres to the prompt text. Higher values keep the image closer to the prompt.

    • clipGuidancePreset? string​

      The clip guidance preset.

    • height? number​

      The desired height of the image(s) in pixels.

    • model? string​

      The model that should be used to generate the image(s).

    • negativePrompt? string​

      The description of what the generated image(s) should not look like.

    • numberOfImages? number​

      The number of images that should be generated.

    • prompt string​

      The description of what the generated image(s) should look like.

    • sampler? string​

      The sampler to use for the diffusion process.

    • seed? number​

      The random noise seed that should be used.

    • steps? number​

      The number of diffusion steps to run.

    • stylePreset? string​

      The style preset that should be used to guide the image model torwards a specific style.

    • width? number​

      The desired width of the image(s) in pixels.

  • AIGenerateImageSuccess​

    Defines an interface that represents a result from ai.generateImage(request, options).

    Members​

  • AIGenerateSkyboxAction​

    An event that is used to generate a skybox using AI.

    Members​

    • negativePrompt? string​

      The negative prompt to use for the skybox.

      Describes the things that you don't want the skybox to look like.

    • options AIGenerateSkyboxOptions​

      The options that should be included in the request.

    • playerId? string​

      The ID of the player that created this task. Set by remote action handlers when a task is recieved from a remote player.

    • prompt string​

      The prompt to use for the skybox.

      Describes things that you want the skybox to look like.

    • taskId (string | number)​

      The ID of the async task.

    • type "ai_generate_skybox"​

      The type of the event. This helps determine how the event should be applied to the state.

    • uncopiable? boolean​

      Whether the action can be structure cloned. If true, then the action should not be passed across message ports without special handling.

  • AIGenerateSkyboxBlockadeLabsOptions​

    Options that are specific to Blockade Labs implementations for ai.generateSkybox(prompt, negativePrompt, options).

    Members​

    • remixImagineId? number​

      The ID of a previously generated skybox.

    • seed? number​

      The random seed to use for generating the skybox.

    • skyboxStyleId? number​

      The pre-defined style ID for the skybox.

  • AIGenerateSkyboxOptions​

    Defines an interface that represents options for ai.generateSkybox(prompt, negativePrompt, options).

    Members​

  • AIGenerateSkyboxRequest​

    Defines an interface that represents a request for ai.generateSkybox(request).

    Members​

    • negativePrompt? string​

      The prompt that that describes what the generated skybox should avoid looking like.

    • options? AIGenerateSkyboxOptions​

      The options that should be included in the request.

    • prompt string​

      The prompt that describes what the generated skybox should look like.

  • AIGenerateSkyboxResult​

    Defines an interface that represents the result from ai.generateSkybox(request).

    Members​

    • fileUrl string​

      The URL that the generated skybox is located at.

    • thumbnailUrl? string​

      The URL that the thumbnail for the generated skybox is located at.

  • AIGeneratedImage​

    Defines an interface that represents an AI generated image.

    Members​

    • base64 string​

      The base64 encoded image.

    • mimeType string​

      The MIME Type of the image data.

    • seed? number​

      The seed of the generated image.

    • url string​

      The URL that can be used to display the image.

  • AISloydGenerateModelFailure​

    A failed response to a request to generate a model using the Sloyd AI interface.

    Members​

    • errorCode ("not_logged_in" | "not_supported" | "record_not_found" | "invalid_record_key" | "unacceptable_request" | "server_error" | "not_authorized" | "not_subscribed" | "invalid_subscription_tier" | "subscription_limit_reached" | "action_not_supported")​

      The error code.

    • errorMessage string​

      The error message.

    • reason? DenialReason​

      The reason why the request was denied.

    • success false​

  • AISloydGenerateModelOptions​

    The options for generating a model using Sloyd AI.

    Members​

    • baseModelId? string​

      The ID of the model that the new model should be based on.

    • levelOfDetail? number​

      The level of detail that should be used. Higher values will result in more detailed models. Should be between 0.01 and 1. Defaults to 0.5.

    • outputMimeType? ("model/gltf+json" | "model/gltf-binary")​

      The MIME type that should be used for the model. If omitted, then "model/gltf+json" will be used.

    • prompt string​

      The prompt to use for the model.

    • recordName? string​

      The name of the record that should be used. If omitted, then the ID of the user will be used.

    • thumbnail? object​

      The options for the thumbnail for the model. If omitted, then no thumbnail will be generated.

  • AISloydGenerateModelResponse​

    The response to a request to generate a model using the Sloyd AI interface.

    A AISloydGenerateModelResponse can be one of the following values:

  • AISloydGenerateModelSuccess​

    A successful response to a request to generate a model using the Sloyd AI interface.

    Members​

    • confidence? number​

      The confidence of the AI in the created model.

    • mimeType ("model/gltf+json" | "model/gltf-binary")​

      The MIME type of the model.

    • modelData string​

      The data for the model. If the mimeType is "model/gltf+json", then this will be a JSON string. If the mimeType is "model/gltf-binary", then this will be a base64 encoded string.

    • modelId string​

      The ID of the model that was created.

    • name? string​

      The name of the model.

    • success true​

    • thumbnailBase64? string​

      The base64 encoded thumbnail of the model.

  • AITextContent​

    Defines an interface that represents text that is passed to an AI chat model.

    Members​

    • text string​

      The text of the content.

  • AIUrlContent​

    Defines an interface that represents a URL that is passed to an AI chat model. This data can be used to represent images, videos, or other types of data that the model supports fetching. Some models do not support this type of content.

    Members​

    • url string​

      The URL that the content is available at.

  • ListedChatModel​

    Defines an interface that represents a listed chat model.

    Members​

    • isDefault? boolean​

      Whether this is the default model.

    • name string​

      The name of the model.

    • provider string​

      The provider of the model.

  • RealtimeSessionRequest​

    Defines a request to create a new realtime session.

    Members​

    • inputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")​

      The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate, single channel (mono), and little-endian byte order.

    • inputAudioNoiseReduction? object​

      Configuration for input audio noise reduction. This can be set to null to turn off. Noise reduction filters audio added to the input audio buffer before it is sent to VAD and the model. Filtering the audio can improve VAD and turn detection accuracy (reducing false positives) and model performance by improving perception of the input audio.

    • inputAudioTranscription? object​

      Configuration for input audio transcription, defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through the /audio/transcriptions endpoint and should be treated as guidance of input audio content rather than precisely what the model heard. The client can optionally set the language and prompt for transcription, these offer additional guidance to the transcription service.

    • instructions? string​

      The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior.

      Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.

    • maxResponseOutputTokens? number​

      Maximum number of output tokens for a single assistant response, inclusive of tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

    • modalities? ("text" | "audio")[]​

      The set of modalities the model can respond with. To disable audio, set this to ["text"].

    • model string​

      The Realtime model used for this session.

    • outputAudioFormat? ("pcm16" | "g711_ulaw" | "g711_alaw")​

      The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw. For pcm16, output audio is sampled at a rate of 24kHz.

    • temperature? number​

      Sampling temperature for the model, limited to [0.6, 1.2]. For audio models a temperature of 0.8 is highly recommended for best performance.

    • toolChoice? string​

      How the model chooses tools. Options are auto, none, required, or specify a function.

    • tools? object[]​

      Tools (functions) available to the model.

    • turnDetection? object​

      Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech. Semantic VAD is more advanced and uses a turn detection model (in conjuction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.

    • voice? string​

      The voice the model uses to respond. Voice cannot be changed during the session once the model has responded with audio at least once. Current voice options are alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, and verse.