Hello, i must find a expert of scrapy framework.
When i run this command: scrapy runspider "[url removed, login to view]" the script must return the json datas in the std out.
Then i think that is necessary to create a correct pipeline and use a memory area to temporary store the datas and reload it from main script.
Do you undestand the problem?
I have experience developing scrappers for several tasks. This looks like a fun challenge. I do see some concerns loading the whole data set in memory, but they could be solved easily.