Basically from a top-level view. this is what we need; please note, the following example will follow specific
data flow relative to the intended operation.
1: Using python-beautiful-soup, scrape 2 pre-defined websites for specific data, using simple
regexp to identify the data as a potential match.
2: Send scraped data to an Administrator dashboard (HTML5) for manual verification and formatting
of scraped data.
2a: Administrator dashboard (html5) requires 2 main functions:
Function 1; Discard the data due to false positive
Function 2; Forward the data to MYSQL DB
3: MYSQL DB
3a: MYSQL DB will have a separate DB for each specific END-USER, before previous administrator
sends the scrapped data to the DB he/she will specif-icy which end-user DB the data will reside in.
3b: After scraped data has been placed into it's according DB the data will be pushed out to
END-USERS dashboard (HTML5) in which they can view the scrapped data.
Run down ~
Python Requirements : Beautiful Soup Scraping, Regex pattern identification, insert into ADMIN DB
to populate ADMIN dashboard (HTML5)
ADMIN DASHBOARD REQUIREMENTS (HTML5) : Fully functional HTML5 dashboard specifically for ADMIN
use, does not need to be complex, simply populates with scraped data from python, and allows
ADMIN to discard data or format for forwarding to MYSQL DB. Also ability to enter manual data that
will be pushed to DB same way as scraped data.
MYSQL DB: Multiple separated DB's for each END-USER. Data placed in each DB will be pushed out
to end users HTML5 dashboard for further manual analysis. ( Also would be useful some mechanism in
the DB or in the python to make sure the same data is not collected twice.
END-USER HTML 5 DASHBOARD : This dashboard will be a little more complex than the Administrator dashboard
it will include only 6 pages, 5 graph pages containing various types of scraped data and 1
main home screen.
As I am aware this description is very vague, I am hoping this will be enough for you to gain a good
idea on the projects size and time requirements in order for you to give me a quote.
Please note, further in-detail description of the project will require NDA's etc so I am trying
to save time on both sides.
!!! NOTE !!! Developer MUST be from Australia or able to converse developer to client directly. Strictly NO middle men.
HTML5 Dashboards will be based off current fully functional models only need to modify
26 freelancers are bidding on average $4198 for this job
Kindly accept my bid for your HTML5 Dashboard project and be assured of a high quality work with our highly experienced and dedicated developers. Thanks
HI We are interested in this project. We are very clear with your requirements. I have few doubts , whether you already took call on Python OR still open for suggestion about more fast technologies for scrapping?
Thanks for your consideration, please see our message and if you would like to discuss the project further do get in touch and I’ll give you a call.