AI Actions

• ai.chat(message: AIChatMessage, options?: AIChatOptions): Promise<AIChatMessage>

  Sends a chat message to the AI. Returns a promise that contains the response from the AI. Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a AIChatMessage and is the message that should be sent to the AI.

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat({
      role: "user",
      content: "Hello!"
  });
  console.log(`${response.role}: ${response.content}`);

  Ask the AI to describe an uploaded image.

  const files = await os.showUploadFiles();
  const firstFile = files[0];
  const base64 = bytes.toBase64String(new Uint8Array(firstFile.data));
  const response = await ai.chat({
     role: 'user',
     content: [
         {
             base64: base64,
             mimeType: firstFile.mimeType,
         },
         {
             text: 'please describe the image'
         }
     ]
  }, {
     preferredModel: 'gemini-pro-vision'
  });
  
  os.toast(response.content);

• ai.chat(messages: AIChatMessage[], options?: AIChatOptions): Promise<AIChatMessage>

  Sends a chat message to the AI. Returns a promise that contains the response from the AI. Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a AIChatMessage[] and

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat([
       {
           role: "system",
           content: "You are a helpful assistant."
       },
      {
           role: "user",
           content: "Hello!"
      }
  ]);
  console.log(`${response.role}: ${response.content}`);

  Build a basic chat bot.

  const messages = [
       {
           role: "system",
           content: "You are a helpful assistant."
       },
  ];
  
  while(true) {
       const userInput = await os.showInput();
       if (!userInput) {
           break;
       }
       messages.push({
           role: "user",
           content: userInput
       });
  
       const response = await ai.chat(messages);
       messages.push(response);
       os.toast(response.content);
  }
  
  os.toast("Goodbye!");

  Ask the AI to describe an uploaded image.

  const files = await os.showUploadFiles();
  const firstFile = files[0];
  const base64 = bytes.toBase64String(new Uint8Array(firstFile.data));
  const response = await ai.chat([{
     role: 'user',
     content: [
         {
             base64: base64,
             mimeType: firstFile.mimeType,
         },
         {
             text: 'please describe the image'
         }
     ]
  }], {
     preferredModel: 'gemini-pro-vision'
  });
  
  os.toast(response.content);

• ai.chat(message: string, options?: AIChatOptions): Promise<string>

  Sends a chat message to the AI. Returns a promise that contains the response from the AI. Throws a CasualOSError if an error occurs while sending the message.

  This function can be useful for creating chat bots, or for using an Artificial Intelligence (AI) to process a message.

  The first parameter is a string and is the message that should be sent to the AI.

  The second parameter is optional and is a AIChatOptions and is the options that should be used.

  Examples

  Send a message to the AI and log the response.

  const response = await ai.chat("Hello!");
  console.log(response);

• ai.generateImage(request: AIGenerateImageOptions, options?: RecordActionOptions): Promise<AIGenerateImageSuccess>

  Generates an image from the given prompt.

  Returns a promise that resolves with the Base64 data of the generated image that can be used as the #formAddress of a bot.

  The first parameter is a AIGenerateImageOptions and is the request object that describes what the image should look like.

  The second parameter is optional and is a RecordActionOptions and is the options for the request.

  Examples

  Generate an image from a prompt.

  const imageResult = await ai.generateImage({
      prompt: "An oil painting of a grassy field.",
  });
  masks.formAddress = imageResult.images[0].url;

  Generate a image from a prompt and a negative prompt

  const imageResult = await ai.generateImage({
      prompt: "An oil painting of a grassy field.",
      negativePrompt: "realistic"
  });
  masks.formAddress = imageResult.images[0].url;

  Generate a image and upload it as a file record

  const imageResult = await ai.generateImage({
      prompt: "An oil painting of a grassy field.",
      negativePrompt: "realistic"
  });
  const image = imageResult.images[0];
  const blob = bytes.fromBase64Url(image.url);
  const file = await os.recordFile(recordKey, blob);
  console.log('file url', file.url);

• ai.generateImage(prompt: string, negativePrompt?: string, options?: AIGenerateImageOptions & RecordActionOptions): Promise<string>

  Generates an image from the given prompt.

  Returns a promise that resolves with the Base64 data of the generated image that can be used as the #formAddress of a bot.

  The first parameter is a string and is the string that describes what the image should look like.

  The second parameter is optional and is a string and is the string that describes what the image should avoid looking like.

  The third parameter is optional and is a AIGenerateImageOptions & RecordActionOptions and is the additional options that should be used.

  Examples

  Generate an image from a prompt.

  const image = await ai.generateImage("An oil painting of a grassy field.");
  masks.formAddress = image;

  Generate a image from a prompt and a negative prompt

  const image = await ai.generateImage("An oil painting of a grassy field.", "realistic");
  masks.formAddress = image;

  Generate a image and upload it as a file record

  const image = await ai.generateImage("An oil painting of a grassy field.");
  const blob = bytes.fromBase64Url(image);
  const file = await os.recordFile(recordKey, blob);
  console.log('file url', file.url);

• ai.generateSkybox(request: AIGenerateSkyboxRequest): Promise<AIGenerateSkyboxResult>

  Generates a skybox image from the given request object.

  Returns a promise that resolves with an object that contains a URL to the generated image that can be used as the #formAddress of a bot that has #form set to skybox.

  The first parameter is a AIGenerateSkyboxRequest and is the request object that describes what the skybox should look like.

  Examples

  Generate a skybox from a prompt.

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.");
  masks.formAddress = skybox;

  Generate a skybox from a prompt and a negative prompt

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.", "A skybox with a red sky and brown grass.");
  masks.formAddress = skybox;

• ai.generateSkybox(prompt: string, negativePrompt?: string, options?: AIGenerateSkyboxOptions): Promise<string>

  Generates a skybox image from the given prompt.

  Returns a promise that resolves with a URL to the generated image that can be used as the #formAddress of a bot that has #form set to skybox.

  The first parameter is a string and is the string that describes what the skybox should look like.

  The second parameter is optional and is a string and is the string that describes what the skybox should avoid looking like.

  The third parameter is optional and is a AIGenerateSkyboxOptions and is the additional options that should be used.

  Examples

  Generate a skybox from a prompt.

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.");
  masks.formAddress = skybox;

  Generate a skybox from a prompt and a negative prompt

  const skybox = await ai.generateSkybox("A skybox with a blue sky and green grass.", "A skybox with a red sky and brown grass.");
  masks.formAddress = skybox;