Retrieve huge amount of data from REST (get) call API
I am trying to retrieve a bout 50,000 lines of data, which is data for different cities. as data is too huge API gets error after 20 minutes. Is there any method by which i can call the REST Api and just get data for one city. i means limiting the retrieve rate. My purpose is to call API for each city one by one or when need.
Muhammad Sarfaraz Malik
Assuming that the api supports pagination a solution couldbe like;
- create a process queue mechanism (see appstore) for the 50k of data, whereby each queued action retrieve 1k over the api or per city (or what your system could handle) and persist it to your db, whereafter you can do your own custom processing
You could also try retrieving the cities data via OData. That technique can usually handle a larger dataload.