Analyze nutrition info

[Simon_Hill]

Anyone have a recommended approach the following feature?

User uploads a photo of their plate of food
Analyze the dish and return the following:

Name of dish (descriptive title)
List of ingredients
Estimated amount of fat, carbs, protein, sugar (grams)

I have tested this outside of Glide, using ChatGPT directly. I simply attached an image, and entered a prompt that asked for the above response. It worked really well. At first, I thought it would be pretty easy to achieve the same using Glide.

I just upgraded to a business plan, and I was hoping that this would provide access to a straightforward solution involving an AI integration. For example, image to text where I could use a prompt That would instruct the AI to perform the analysis and return the data, preferably in JSON Format. However, it seems that the image to text option Does not include a prompt, and simply has ‘ Describe’ as the action.

Does this mean I’m going to have to create my own AI call action and set up tokens with ChatGPT? If so, what are the Glide steps required and which integrations would be needed? For awareness, some months back I did set up a ChatGPT account and I do have a token that can be used (In the end, I didn’t use it because Glide introduced the new AI column).

Thanks in advance for any tips and pointers.

[ThinhDinh]

Linking this here for other readers, since you have already responded on the other thread.

AI -> Image to Text

I believe they use OpenAI’s model under the hood, but I’m not entirely sure. I assume you are using the “Image to Text” column, but “Complete chat” might actually be what you need instead. [image] It allows you to add an image, and ask a question, whilst “Image to text” doesn’t allow you to specify a prompt, I believe.

[Simon_Hill]

So I got this working with 1 key barrier: When the result is returned most of the columns are not automatically updating. Can you provide a hint how to fix?

I return the result into a rich text column, see image. This works great.

Here is the prompt I am using:

Here is the prompt I am using.

The attached is a photo of a meal. Generate the following:

1. A descriptive title for the recipe
2. Ingredients
3. Estimated amount of fat in grams
4. Estimated amount of carbohydrates in grams
5. Estimated amount of protein in grams
6. Estimated amount of sugar in grams

Output format as a JSON object consistent with this output:

{
“Title”: “Hearty Mediterranean Chicken Pasta Salad”,
“NutritionalEstimates”: {
“Fat”: “20 grams”,
“Carbs”: “50 grams”,
“Protein”: “30 grams”,
“Sugar”: “7.5 grams”
},
“Ingredients”: [
“Chicken”,
“Pasta (rotini)”,
“Avocado”,
“Cherry tomatoes”,
“Red onions”,
“Basil”
]
}

And this generates this:

In addition, I have a column that is a JSON Object that looks at the result and produces a JSON object. That object looks good too. I then use several Query JSON columns to extract the elements like title, ingredients, fat, protein, etc…

However, the issue I have is that whilst the JSON Object updates as soon as the AI Complete Chat integration has been carried out, none of the Query JSON columns auto update. They go blank. I have to manually go back into each and link them again with the JSON Object column.

Is the issue something to do with a delay in returning the result? If I go into the data editor I can update the Query JSON’s by editing and selecting the attribute again… but not sure why they don’t just refresh as soon as a new JSON Object is available

Should become automatically:

[ThinhDinh]

Why do you have to get an AI result column and an additional JSON object column? Couldn’t you just parse directly from the AI Result column?

[Simon_Hill]

I guess I’m relatively new to using JSON. Does Parse mean to read out the wanted data? Can I use Query JSON with a rich text column? Let me try:

Ok so it will point to the RT column but the formatting seems off and it doesn’t see the JSON structure properly.

Here is with the AI text to JSON Object:

And this is when it is pointing directly to the RT column:

Any tips? Is the JSON result not formatted properly - something the AI to text column is sorting out for me but requiring the manual refresh?

[Darren_Murphy]

@Simon_Hill

Ok so it will point to the RT column but the formatting seems off and it doesn’t see the JSON structure properly.

Why are you using a Rich Text column?
That’s probably introducing additional formatting that is invalidating the returned JSON.
Just use a normal Text type column.

[Simon_Hill]

Ok I changed to text column but that doesn’t help.

I also asked CHATGPT and it recommended I updated my prompt in the AI Complete Chat column. Having done that it seems to work on PC but now when I try using my mobile device, the upload image component seems to have stopped working. Really strange. I can delete the image but after picking another to upload it does nothing… maybe a bug. Same behavior even after I delete the image picker and add a new one.

[Simon_Hill]

Well I have deleted the image picker and replaced it. No luck. I deleted the Image column (user specific). No luck… the image picker has just stopped working on mobile device. It deletes the image but will not upload. Strange. It works on desktop editor view…

[ThinhDinh]

I saw you posting a new thread, has this issue been fixed?

[Simon_Hill]

Yes it resolved itself, thanks.