AI Actions

• ai.chat(message: AIChatMessage, options?: AIChatOptions): Promise<AIChatMessage>

  Sends a chat message to the AI. Returns a promise that contains the response from the AI. Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a AIChatMessage and is the message that should be sent to the AI.

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat({
      role: "user",
      content: "Hello!"
  });
  console.log(`${response.role}: ${response.content}`);

  Ask the AI to describe an uploaded image.

  const files = await os.showUploadFiles();
  const firstFile = files[0];
  const base64 = bytes.toBase64String(new Uint8Array(firstFile.data));
  const response = await ai.chat({
     role: 'user',
     content: [
         {
             base64: base64,
             mimeType: firstFile.mimeType,
         },
         {
             text: 'please describe the image'
         }
     ]
  }, {
     preferredModel: 'gemini-pro-vision'
  });
  
  os.toast(response.content);

• ai.chat(messages: AIChatMessage[], options?: AIChatOptions): Promise<AIChatMessage>

  Sends a chat message to the AI. Returns a promise that contains the response from the AI. Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a AIChatMessage[] and

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat([
       {
           role: "system",
           content: "You are a helpful assistant."
       },
      {
           role: "user",
           content: "Hello!"
      }
  ]);
  console.log(`${response.role}: ${response.content}`);

  Build a basic chat bot.

  const messages = [
       {
           role: "system",
           content: "You are a helpful assistant."
       },
  ];
  
  while(true) {
       const userInput = await os.showInput();
       if (!userInput) {
           break;
       }
       messages.push({
           role: "user",
           content: userInput
       });
  
       const response = await ai.chat(messages);
       messages.push(response);
       os.toast(response.content);
  }
  
  os.toast("Goodbye!");

  Ask the AI to describe an uploaded image.

  const files = await os.showUploadFiles();
  const firstFile = files[0];
  const base64 = bytes.toBase64String(new Uint8Array(firstFile.data));
  const response = await ai.chat([{
     role: 'user',
     content: [
         {
             base64: base64,
             mimeType: firstFile.mimeType,
         },
         {
             text: 'please describe the image'
         }
     ]
  }], {
     preferredModel: 'gemini-pro-vision'
  });
  
  os.toast(response.content);

• ai.chat(message: string, options?: AIChatOptions): Promise<string>

  Sends a chat message to the AI. Returns a promise that contains the response from the AI. Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a string and is the message that should be sent to the AI.

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat("Hello!");
  console.log(response);

• ai.generateImage(request: AIGenerateImageOptions, options?: RecordActionOptions): Promise<AIGenerateImageSuccess>

  Generates an image from the given prompt.

  Returns a promise that resolves with the Base64 data of the generated image that can be used as the #formAddress of a bot.

  The first parameter is a AIGenerateImageOptions and is the request object that describes what the image should look like.

  The second parameter is optional and is a RecordActionOptions and is the options for the request.

  Examples

  Generate an image from a prompt.

  const imageResult = await ai.generateImage({
      prompt: "An oil painting of a grassy field.",
  });
  masks.formAddress = imageResult.images[0].url;

  Generate a image from a prompt and a negative prompt

  const imageResult = await ai.generateImage({
      prompt: "An oil painting of a grassy field.",
      negativePrompt: "realistic"
  });
  masks.formAddress = imageResult.images[0].url;

  Generate a image and upload it as a file record

  const imageResult = await ai.generateImage({
      prompt: "An oil painting of a grassy field.",
      negativePrompt: "realistic"
  });
  const image = imageResult.images[0];
  const blob = bytes.fromBase64Url(image.url);
  const file = await os.recordFile(recordKey, blob);
  console.log('file url', file.url);

• ai.generateImage(prompt: string, negativePrompt?: string, options?: AIGenerateImageOptions & RecordActionOptions): Promise<string>

  Generates an image from the given prompt.

  Returns a promise that resolves with the Base64 data of the generated image that can be used as the #formAddress of a bot.

  The first parameter is a string and is the string that describes what the image should look like.

  The second parameter is optional and is a string and is the string that describes what the image should avoid looking like.

  The third parameter is optional and is a AIGenerateImageOptions & RecordActionOptions and is the additional options that should be used.

  Examples

  Generate an image from a prompt.

  const image = await ai.generateImage("An oil painting of a grassy field.");
  masks.formAddress = image;

  Generate a image from a prompt and a negative prompt

  const image = await ai.generateImage("An oil painting of a grassy field.", "realistic");
  masks.formAddress = image;

  Generate a image and upload it as a file record

  const image = await ai.generateImage("An oil painting of a grassy field.");
  const blob = bytes.fromBase64Url(image);
  const file = await os.recordFile(recordKey, blob);
  console.log('file url', file.url);

• ai.generateSkybox(request: AIGenerateSkyboxRequest): Promise<AIGenerateSkyboxResult>

  Generates a skybox image from the given request object.

  Returns a promise that resolves with an object that contains a URL to the generated image that can be used as the #formAddress of a bot that has #form set to skybox.

  The first parameter is a AIGenerateSkyboxRequest and is the request object that describes what the skybox should look like.

  Examples

  Generate a skybox from a prompt.

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.");
  masks.formAddress = skybox;

  Generate a skybox from a prompt and a negative prompt

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.", "A skybox with a red sky and brown grass.");
  masks.formAddress = skybox;

• ai.generateSkybox(prompt: string, negativePrompt?: string, options?: AIGenerateSkyboxOptions): Promise<string>

  Generates a skybox image from the given prompt.

  Returns a promise that resolves with a URL to the generated image that can be used as the #formAddress of a bot that has #form set to skybox.

  The first parameter is a string and is the string that describes what the skybox should look like.

  The second parameter is optional and is a string and is the string that describes what the skybox should avoid looking like.

  The third parameter is optional and is a AIGenerateSkyboxOptions and is the additional options that should be used.

  Examples

  Generate a skybox from a prompt.

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.");
  masks.formAddress = skybox;

  Generate a skybox from a prompt and a negative prompt

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.", "A skybox with a red sky and brown grass.");
  masks.formAddress = skybox;

• ai.hume.getAccessToken(recordName?: string): Promise<AIHumeGetAccessTokenResult>

  Gets an access token for the Hume AI API. Returns a promise that resolves with the access token.

  The first parameter is optional and is a string and is the name of the record that the access token should be generated for. If omitted, then the user's player studio record will be used.

  Examples

  Get an access token for the Hume AI API.

  const accessToken = await ai.hume.getAccessToken();

  Get an access token for the Hume AI API for the given record.

  const accessToken = await ai.hume.getAccessToken('recordName');

• ai.sloyd.generateModel(request: AISloydGenerateModelOptions, options: RecordActionOptions): Promise<AISloydGenerateModelResponse>

  Generates a new 3D model using sloyd.ai.

  The first parameter is a AISloydGenerateModelOptions and is the options for the 3D model.

  The second parameter is a RecordActionOptions and is the options for the request.

  Examples

  Generate a chair model using Sloyd

  const model = await ai.sloyd.generateModel({
      prompt: 'a chair'
  });

  Generate a chair using a studio subscription

  const model = await ai.sloyd.generateModel({
      recordName: 'studioID',
      prompt: 'a chair'
  });

• ai.stream.chat(message: AIChatMessage, options?: AIChatOptions): Promise<AsyncIterable>

  Sends a chat message to the AI and streams the response back. Returns a promise that resolves with an async iterable that contains the responses from the AI.

  Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a AIChatMessage and is the message that should be sent to the AI.

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat({
      role: "user",
      content: "Hello!"
  });
  console.log(`${response.role}: ${response.content}`);

  Ask the AI to describe an uploaded image.

  const files = await os.showUploadFiles();
  const firstFile = files[0];
  const base64 = bytes.toBase64String(new Uint8Array(firstFile.data));
  const response = await ai.stream.chat({
     role: 'user',
     content: [
         {
             base64: base64,
             mimeType: firstFile.mimeType,
         },
         {
             text: 'please describe the image'
         }
     ]
  }, {
     preferredModel: 'gemini-pro-vision'
  });
  
  for await (let message of response) {
       os.toast(message.content);
  }

• ai.stream.chat(messages: AIChatMessage[], options?: AIChatOptions): Promise<AsyncIterable>

  Sends a chat message to the AI. Returns a promise that resolves with an async iterable that contains the responses from the AI.

  Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a AIChatMessage[] and

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.stream.chat([
       {
           role: "system",
           content: "You are a helpful assistant."
       },
      {
           role: "user",
           content: "Hello!"
      }
  ]);
  
  for await (let message of response) {
     console.log(`${message.role}: ${message.content}`);
  }

  Build a basic chat bot.

  const messages = [
       {
           role: "system",
           content: "You are a helpful assistant."
       },
  ];
  
  while(true) {
       const userInput = await os.showInput();
       if (!userInput) {
           break;
       }
       messages.push({
           role: "user",
           content: userInput
       });
  
       const response = await ai.stream.chat(messages);
  
       for await (let message of response) {
           messages.push(message);
           os.toast(message.content);
       }
  }
  
  os.toast("Goodbye!");

  Ask the AI to describe an uploaded image.

  const files = await os.showUploadFiles();
  const firstFile = files[0];
  const base64 = bytes.toBase64String(new Uint8Array(firstFile.data));
  const response = await ai.stream.chat([{
     role: 'user',
     content: [
         {
             base64: base64,
             mimeType: firstFile.mimeType,
         },
         {
             text: 'please describe the image'
         }
     ]
  }], {
     preferredModel: 'gemini-pro-vision'
  });
  
  for await (let message of response) {
       os.toast(message.content);
  }

• ai.stream.chat(message: string, options?: AIChatOptions): Promise<AsyncIterable>

  Sends a chat message to the AI and streams the response back. Returns a promise that resolves with an async iterable that contains the responses from the AI.

  Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a string and is the message that should be sent to the AI.

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.stream.chat("Hello!");
  
  for await (let message of response) {
     console.log(message);
  }