I'm looking for an expert in crawling/data processing/data mining to process a list with 30 000 00 domains.
What I need is someone to crawl these 30 000 000 domains ( + up to 100 pages on each domain so this will make up to 3 000 000 000 urls which need to be crawled) and to collect all external backlinks found on them.
This will create a very large log file ( maybe around 50 gigs) and i want the expert to help me upload that log on a server and to be able to search inside the log file.
If you have more question regarding this project, please contact me.
42 freelancers are bidding on average $637 for this job
Hello There, I have questions regarding this project. Would you be able to answer them before. My name is Kal. Looking forward to hear from you soon. Thanks. Kalpanasekhar
Hi, I understand the project requirements and willing to start as soon as possible. Kindly share the details and I will deliver with high quality. Looking forward to work with you. Thanks & Regards Khushi India
Hi, Thanks for the opportunity to bid on this project. I have done such data gathering and research in the past for various clients so that I will provide good service. Regards, Jai
Hello, I read your requirements and understand very-well and ready to start work from now. Let's discuss more in chat, waiting for your reply. Thank you
Hi there! Have done this type of work before and am a big data/scraping expert ! Please feel free to message me with any additional information or with questions if any ! Thanks! -Steve
Hello, Im interested to work on this. I gone through the job description. Im looking forward to get more details about the log file and format of the logfile. Thanks, Sheik Mubarak S