Website scraper and then data analysis

Hi there,

This is a two part project to be written using mainly python (though C may also be used for calculations / libraries that cannot be done / do not exist in Python.)

1. Web scraper.

I need a web scraper that will regularly scrape data from a series of web pages. This part of the program should be able to take a list of proxies to be used if there is an error and access is denied to a website.

2. Data analysis.

Statistical analysis will be performed on the data gained from the web scraper. This will involve a regression analysis to predict sales using more than 20 independent variables.

The program overall should be able to run in the background on my Windows 7 machine. The refresh rate for the raw data gained from the scrape should be variable. When opened, I will like a very simple presentation of the statistical analysis. This GUI does not need to look pretty at all. It just needs to be easily read and functional (i.e. no colours, large enough text.)

Finally, the winning bidder will need to sign one of these electronic non-disclosure agreements (all costs covered):

[url removed, login to view]

Thanks for taking the time to consider this project. I look forward to further discussion with the successful bidder.



The following is a very simple example of the sort of statistical analysis I refer to in this project's description:

Using an SQLite database, the program will pull numerical information from various pages. For example, an item with comments on it:

Item x


The number of comments and number of sales Item X has had would then be scrapped from this site and saved under the particular item's ID number. The database logic for this would look like this:


When enough data has been gathered, there will be a facility within the program that will allow me to perform a statistical analysis to see how well NumberOfComments predicts or correlates with NumberOfSales and how significant this prediction is (p-value.)

The winning bidder will be given a highly detailed directive that will be extremely straight forward. I am also available nearly all the time to answer any queries at all through skype or IM.



Skills: C Programming, HTML, Mathematics, Python, Statistics

See more: windows gui programming, variables in programming, variable programming, variable in programming, statistical programming, raw presentation, python programming website, python gui programming, python functional programming, program website in python, programming variables, programming variable, programming libraries, programming a website costs, needs analysis, machine programming, if then programming, gui programming python, gui programming in python, functional programming python, functional programming in python, data programming, c programming website, scraper analysis, scraper website

About the Employer:
( 12 reviews ) Sydney, Australia

Project ID: #1265973

13 freelancers are bidding on average $220 for this job


Details in PMB

$150 USD in 0 days
(175 Reviews)

I can deliver the project as I am an expert in scrapping

$250 USD in 5 days
(79 Reviews)

Hi, I'm a professional developer, pretty good in protocol & network traffic analysis. Will provide you with a quick and good solution.

$240 USD in 2 days
(5 Reviews)

I have extensive experience writing web scrapers, with python in particular. I also have considerable experience using SQlite and I will have no problem writing this software. No problem with signing the NDA and I'm More

$250 USD in 4 days
(1 Review)

I have much experience on web scraping and database operation with Python. I can finish this requirement on time.

$200 USD in 7 days
(1 Review)

Ready to start please check your PM

$250 USD in 4 days
(0 Reviews)

it will be my pleasure

$250 USD in 3 days
(1 Review)

very good at programming and very good knowledge of python,mathematics and c language

$175 USD in 90 days
(0 Reviews)

Ready to do your Project.

$200 USD in 20 days
(0 Reviews)

Hello, I have read and understand your requirement. I have done a lot scraper work from different sites. I will create scraper and data analysis of scrap data in C#, php or Java and give you a quality work. Regards More

$250 USD in 10 days
(0 Reviews)

I am a research scientist who has worked on SciPy and other Python related projects. Have built python based APIs for Piratebay and other websites. Have built various django based apps. If you would want to see my code More

$200 USD in 10 days
(0 Reviews)

please chk pmb

$250 USD in 2 days
(0 Reviews)

Working as a Software Engineer with Software Pattern, Islamabad, Pakistan (April. ’11 to date) • Developed different web crawlers that extract important information from the website according to the set criteria. Afte More

$200 USD in 10 days
(0 Reviews)