App doesn't work with 65k rows of data

Hello,

When my account was free and had been limited to 500 rows of data the app was working fast and fine. Now, when I upgraded to the pro account I connected a google sheet with one list of 15k rows and another list with 40k rows. Now my app has several refreshes on start. Search also works incorrect. In one tab it founds data, in another tab some data is lost.

How can I increase the limit of rows in database to make my app working correctly? 60k rows is just nothing to work with. I don’t understand such limit.

I kindly ask to help me with that.

60k may not be much as far as databases are concerned, but Google sheets aren’t exactly a true database, and the Google API has limits and can only handle so much data transfer. That’s a lot of data to continuously move back and forth between Glide servers and Google servers.

Keep in mind, if your data is not behind row owners, then that entire 60k of data is downloaded and cached to a user device every time they open that app. Along with all computed columns being recalculated.

2 Likes

So what’s the solution? Can I switch from google sheets to some DB in glide?

You can try using glide tables instead of Google sheets. It can help in some ways because you eliminate that extra step of having data sync back and forth between google sheets and glide’s server copy of that google sheet data. From the app’s point of view, it only can only see and synchronize it’s local db copy with the copy on Glide’s server. Eliminating one leg of the data transfer process to google can help things run more efficiently…at least on the back end.

There is no easy way to transfer data from a google sheet to a glide table. There are some tricks, like an inline list with an add row action to manually click through and have the data added to a new table. Or some have used robots/macros to automate that manual process to transfer data.

However, I won’t guarantee that the app itself will run a whole lot better. Like I said, it only synchronizes with Glide’s copy of the database. You are still forcing users to download, cache, and compute 65k rows of data every time they open the app fresh. That’s a lot of data to download and compute. The question is, does the user need access to all of that data? Can you implement row owners in any capacity to reduce the amount of data a user downloads, to only their owned rows? While there is technically no limit to the number of rows a pro app can use, there are technical bottlenecking issues you start to run into once you exceed 25k rows, which is the limit that glide technically supports but does not currently enforce.

You could look into transferring data from the Google sheet to a glide table, but it won’t be a quick solution. I guess I would look for ways to restructure your data to make it more efficient and/or implement security, such as row owners to reduce the amount of data that a user sees. Some glide users have implemented google scripts or sheet queries to move data in and out of the sheet (to another spreadsheet entirely, or just another sheet tab that’s not utilized by the app) based on user input, or time based queries or triggers. I’m not a big fan of those scripting or sheet query solutions though.

Most importantly, do not try to display all that data at one time on the screen. At the very least I hope you have some categorization to filter out data appropriately. It won’t stop the caching of all data and won’t necessarily prevent all computations from occuring, but will definitely reduce the amount of data that has to be rendered on the screen.

5 Likes

Thanks for your reply.
To be more focused on the problem, the data is a list of medications (15k) and a list of interactions of those medications (40k). This data can not be personalised, the only thing I can do (and actually did) is the limitation when displaying it on the screen. The rest of data is a first aid kit which is a additional glide table. It will growth with the number of users.

I can not see any way to structure, limit or divide this data to several tables besides the 3 I already have.

Please provide a detailed way to import a google sheet to a glide table.

PS Link to the app: pills-app.net

@santer1982 you could use a robot tool to import data to glide tables. But it might not help you enough as Jeff stated

2 Likes

@Jeff_Hager @Krivo Thank you for your answers. Transferring data with RPA was a long process but helped.
There are many disadvantages though, but at least it works.

1 Like