Do you have the option to directly connect to the database? If so, you could use the Database Connector module from the app store.
If the amount of data is too big, rest is not an option. We had the same issue on a data migration we just did some time ago. Rest was too slow and needed too many ressources. Excel import is also not an option if your excel sheet has too many lines.
A database based migration can be the best approach on gughe data amounts.
Thanks Andreas for the response.
Initial load using database connectors –> As per my understanding, database connectors can only be used within Microflow. So you suggest to retrieve data in a microflow and populate mendix tables.
Ongoing data sync – we cannot use database connectors due to security reasons and will have to go with rest services. With rest do you know if Mendix supports Bulk APIs.
Thanks. I will check on Process queue module.
Bulk API – > To add more information, as mentioned we have to sync data between On Premise ERP to Mexdix app. We are thinking of writing data from On Premises to Azure Cloud DB ( our Mendix runtime will be deployed on azure) , and then write a Logic app to invoke MEndix REst Service to write data to Mendix tables ( ERP DB→ Azure Cloud DB → Mendix DB).
I was going through mendix documentation ( Publish rest services) and I couldn’t see any option of specifying number of records a rest service can handle or there is no Queue in Mendix which we can use to write streams of data which we can be used to write data to mendix DB. Do you know if we can write a microflow to consume data from a Queue sitting on azure cloud.