Workaround for searchable database containing 100,000+ rows?

[tourglide]

Hi there, I’m in the process of building a new app that contains a searchable database of a long list of songs (100,000+). The issue is with the current Glide row limits (personal and paid), it would seem that the app would not be feasible for the Glide platform, which is unfortunate since Glide is a great platform.

Might there be any known workarounds or other options, perhaps a mechanism to pull a database feed from a separate resource? The 100,000+ song list wouldn’t need to be modified, rather just searched and referenced.

Thank you!

[Eric_Penn]

You’ll have to be mindful of your data structure.

Row Owners, =filter & =vlookup are your friends.

Using spreadsheet formulas could get you there but will be some loss of performance.

[ptrnelson]

Hey Eric, can you expand on this a bit. I’m in a similar pickle as @tourglide but have around 350K rows.

[Mark_Turrell]

I had an issue where I needed to do a post code lookup in my app to return the town and German ‘Land’. It’s a 5 digit code (PLZ) … with minimal and unreliable logic… and 12,000 possibilities.

The solution is to put that into a separate Google sheet (import csv). Next send a webhook with the PLZ to integromat, which does a Google Sheet lookup to pick the right values, then does a search for the row that is using that plz.

It’s very quick, and saves rows in my app.

[V88]

And how does Integromat “report back to your app though?

[adriasoce]

Making a lookup and paste the values on the row that you are using in order to update the data on the app i think

[Mark_Turrell]

This should give you an idea of the scenario
@Darren_Murphy is the expert - this is one of his

[V88]

Sure, but obviously still with the usual 2-3 min delay?

[Mark_Turrell]

The webhook is sent immediately to Integromat - so about 2 seconds
Then integromat runs every minute (much better than Zapier)
The script in integromat takes 6 - 14 seconds to run (with the end point of writing to GS)
Then I have my sad wait time for the Google sheet:

• if I am updating a row, then GS updates Glide on a 3 min cycle
• if I am adding a row, then GS syncs immediately with Glide (so about 5 - 8 seconds)

One of these days we will have a webhook to write into Glide directly (esp if you need Glide Tables). Oh that will be a happy day…

[Uzo]

Just use glide action to set columns in google sheets with a values to filter, and have gs do the filtering… and glide to show that sheet data, it will take 1 to 5 seconds to complete

[Mark_Turrell]

It depends on the size of the data set. An array formula with a lookup in GS runs every time… every row. This could get painfully slow very quickly, and is unnecessary because ideally the value is fixed once you get it.

A lot depends on the use case and the specifics of the App.

[Uzo]

I tested on 32k rows… runs smoothly

[V88]

What if Glide supported, optionally, waiting for a webhook response. The app could show a spinner whilst waiting and write the response into a column. Almost like a REST API call. Now that would be a nice feature.

[Lucas_Pires]

Glide has a limit of 50k searchable rows - per sheets in Google Sheets - and they’ll be searchable

Is this limit increase in Glide tables @jason?

[Lucas_Pires]

Btw @Messias_Carvalho already done a video about it Take a look

Limite de 25000 lignes

I ran tests on my account, where I have +175,000 rows available and it is possible to reach 50,000 rows for each table on the google sheet. here’s in the video when i show it. [Testando a capacidade do Glide! Será que suporta?]

[Uzo]

just tested with 120k rows… under 3 seconds