AI -> Image to Text

Ask for Help
1

Hi All,

Goal: The user uploads a UML / BPMN diagram (picture), and the app analyzes and evaluates it using AI mechanisms.

Question: Is there a way to invoke Python or JavaScript code from within Glide to perform image analysis (specifically using the gpt-4-vision-preview model)? Can Make/Zapier be used for this?

I tried using the AI Image2Text feature in Glide, but it doesn’t work well for me (by the way, which model does Glide use - from Google or OpenAI, and is there a way to customize the prompt used by Glide)?

Code:

from openai import OpenAI

api_key = ‘sk-…’

image_url = “https://cdn-images.visual-paradigm.com/tutorials/howtodrawsequencediagram_screenshots/23-final-uml-sequence-diagram.png”

client = OpenAI(api_key=api_key)

response = client.chat.completions.create(
model=“gpt-4-vision-preview”,
messages=[
{
“role”: “user”,
“content”: [
{“type”: “text”, “text”: “What does the diagram show - describe it in Polish and find any errors in it.”},
{
“type”: “image_url”,
“image_url”: {
“url”: image_url,
“detail”: “high”
},
},
],
}
],
max_tokens=700,
)

Best Regards,
Andrzej

2

I believe they use OpenAI’s model under the hood, but I’m not entirely sure. I assume you are using the “Image to Text” column, but “Complete chat” might actually be what you need instead.

image
image359×610 22.5 KB

It allows you to add an image, and ask a question, whilst “Image to text” doesn’t allow you to specify a prompt, I believe.

3

@ThinhDinh Thank you very much!
First tests show that it works much better. There is not always a stable answer - but I will definitely test this method.

2 Likes
4

Ooh this could be helpful for my recent new question- thanks!

2 Likes
5

Can you check if it works for you? I am trying to use an image in different ways but always receive the same error:

Complete chat: Invalid type for messages[1].content[1]image_url: expected an object, but got a string instead.

@David

6

Please try a model that has the “vision” capability. In my test, “gpt-4-1106-vision-preview” works.

7

Maybe it worked, but now it doesn’t (anymore). I was just trying gpt-4-turbo-2024-04-09 (also tried gpt-4o etc.) and got the same error:

Invalid type for 'messages[1].content[1].image_url': expected an object, but got a string instead.

Tried with different fields, too (conditional, template, file). Any ideas? Has the integration changed?

8

image

If you’re using Image to Text (which is the column mentioned in the post title), it’s working for me.

1 Like
9

Ehm I see, no, I am using “Complete chat” with the OpenAI integration.

10

Unrelated but I swore Image to Text previously cost only 1 update and not 2…

11

@Jeremy Can you check this? Thank you.

Also, as you have a paid plan, you can chat with the team using the Intercom messenger.

We have now launched the intercom messenger in the builder. Instead of submitting a ticket, folks with paid plans can use the messenger to contact support. We'll be updating docs as soon as we have the full details!

1 Like