Yet another shortcut to make getting descriptions of screenshots easy

Hi all!

One day I realized that getting screenshots descriptions required too many steps: take a screenshot, have time to click the "Share" button, select ChatGPT, write a prompt, wait for a response - it seems that there are too many steps to just look at a meme received from a friend in the messenger :) There have long been addons for Windows and NVDA that allow you to perform the same task with the click of a button. If I understand correctly, the problem with iOS is that a third-party application simply does not have the ability to receive screen content, show windows on top of the current application, and the like, which is why all similar solutions involve sending the original image through the "Share" dialog. Therefore, we are limited to the capabilities provided within iOS Shortcuts app.

Acknowledgements:

• @aaron ramirez for excellent Shortcut, familiarization with which allowed me to understand that the capabilities of iOS Shortcuts are generally sufficient to solve such a problem;

• @alekssamos for great NVDA addon, which gave me the idea of ​​using the free api, as well as adding some additional features, like text recognition.

Quick Start

For those who are not interested in reading long manuals, installation instructions and feature descriptions, here is a link to the Shortcut itself, the initial setup of which should be intuitive. Simply assign a VoiceOver command for executing this shortcut, after which the Shortcut will take a screenshot and offer a menu with a choice of available options.

Current functionality

Currently the Shortcut uses Piccy Bot API to get free descriptions, but it is implemented in such a way that adding a new model whose API is ChatGPT compatible is not too difficult. The functionality supported by this Shortcut includes:

• Getting image descriptions using a given model;

• Optical text recognition from images using the OCR-Engine built into iOS;

• Additional questions about the image whose description the model generates;

• Using a screenshot as an input image or the ability to send your own image through the "Share" dialog;

• Displaying lists, tables and other formatting elements in the generated image description;

• Copying the last answer to the clipboard;

• Optional sound notification upon completion of description generation;

• The experimental feature for automatically turning the VoiceOver off while taking a screenshot, so you don't have to worry about the current Screen Curtain state (requires some additional actions);

• The ability to get answers in any language supported by the model (due to technical limitations of Shortcuts, the language itself must be specified manually).

Setting things up

1. Install Shortcut by following the link.

2. A dialog will appear asking you to set some settings:

   1. Play sound: Determines whether to play a sound after description generation is complete, enter 'y' or 'n';

   2. Disable VoiceOver: Experimental feature for disabling the VoiceOver before taking a screenshot and enabling it back after, increases the delay in the appearance of the options menu by about a second, enter 'y' or 'n';

   Note: this only works if you run Shortcut through a separate Shortcut-Executor.

   3. Description prompt: prompt that will be sent to the model to get an image description , enter '/default' to use a preset prompt or your own prompt;

   4. Language: the language in which the model will generate responses, enter the full name of the language, for example, 'English'.

3. If you want to use the feature for temporarily disable VoiceOver, install this Shortcut as well.

4. Assign VoiceOver command for executing this Shortcut:

   1. Go to the Settings --> Accessibility --> VoiceOver --> Commands --> All Commands --> Shortcuts;

   2. Depending on whether you have enabled the option to temporarily turn VoiceOver off, select CloudVision or CloudVisionExecutor.

   Note: If you choose CloudVisionExecutor, after each image description, the Shortcuts app will launch on top of the current app. This is a technical limitation of the method used to temporarily turn VoiceOver off; if this is inconvenient for you, use the default Shortcut, first turning off the screen curtain manually.

   3. Assign the gesture or the keyboard command for this Shortcut.

Usage instructions

1. Perform a VoiceOver Command or select CloudVision in the "Share" dialog to get an image description.

2. A menu will appear with the following options:

   1. Describe image: get an image description using the model selected in the Shortcut settings;

   2. Recognize text: Recognize text from an image using the OCR-Engine built into the system;

   3. Cancel: quit.

3. After selecting one of the options, it will take some time to receive a response (in the case of description generation, this can be about ten seconds, while text recognition occurs almost instantly).

4. The results of the image analysis will appear in a separate window. After familiarizing yourself with which, you can close the window using the button located in the upper right corner.

5. After viewing the generated description, a menu will appear with the following options:

   1. Chat with model: ask an additional question about the image being analyzed;

   2. Copy last answer: Copy the model's last answer to the clipboard;

   3. Cancel: quit.

6. After selecting the "Chat with model" option, a dialog will appear asking you to enter your question.

7. Similar to the original description, the generated response will appear in a separate window after a while.

8. After viewing the received response, you can continue asking follow-up questions or end your interaction with Shortcut.

Adding your own models

The following instructions involve extensive interaction with the Shortcut implementation. If necessary, detailed instructions for creating Shortcuts can be found here.

1. Open the Shortcuts app, find CloudVision Shortcut and, using the VoiceOver rotor, select an "Edit" action.

2. Find the description_model variable, in the text field located right before it, enter the human-readable name of the model you are adding.

3. Find the description_model_api_key variable, in the text field located right before it, enter your API key (if necessary).

4. Find the description_models variable, using the "Add New Item" button located right before it, create an entry for your model, selecting Dictionary as the value type.

5. For the key, enter the model name specified in step 2.

6. Click on a value to go to the dictionary editing screen.

7. Create the following entries with parameters for your model:

   1. Required, type: text, Key: 'url', value: URL to which requests will be executed, for example 'https://api.openai.com/v1/chat/completions';

   2. Optional, type: text, key: 'user_agent', value: User-Agent, with which requests to the model will be sent, if omitted, the default value is 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36';

   3. Required, type: text, key: 'model_name', value: The value of the model field in the request;

   4. Optional, type: text, key: 'request_messages_key', value: The key by which the request contains an array of the messages, if omitted, the default value is 'messages';

   5. Optional, type: dictionary, key: 'additional_parameters', value: dictionary whose elements will be directly added to the request, can be used to specify parameters such as 'max_tokens' or 'temperature', if omitted, the default value is empty dictionary, i.e. no additional parameters will be added to the request;

   6. Optional, type: text, key: 'response_messages_key', value: The key (or path) where the response contains the text of the received answer, if omitted, the default value is 'choices.1.message.content'.

   Note: If you want to omit any of the fields marked as optional, do not create an entry in the dictionary corresponding to that field.

8. After filling in all the specified fields, you can complete editing the Shortcut.

9. To switch between already added models, simply assign the description_model variable a value that matches the corresponding key in the description_models dictionary.

After all

I hope you find this Shortcut useful. You can leave any questions, bugs and suggestions in the comments below.