We have a web based reporting system that we'd like to scrape. The system is built and hosted by another company, so we'd like to build a Django application that can do this.
So far I have built a quick proof of concept PHP script using the PHP cURL bindings to login and run these reports and return a print out of the results. So I have already uncovered some of the non standard things this web app does (like actually returning a response after returning a 301 header) and proven that we can automate pulling reports. I have also built some Django models for this project to get signoff from my stakeholders and also to communicate to the person doing the work what we want at the end. My proof of conecpt is just that. I did it to make sure that we could do what we needed to do.
So what you will need to do.
1. Use cURL for Python to do what I have done in PHP and build an object structure in Python to do what I have done structurally.
2. Take the response from the current web reporting system which is basically a HTML table and store the results in a MySQL database. I have been advised that BeautifulSoup is probably a good tool for this. See [url removed, login to view] The only change that I'd make to how I currently do things is to chunk the reporting time frame down to a month at a time. So for example if we run a report that spans 2 years we actually run 24 reports against the reporting server and log each piece of data to the database and then return the whole 24 month period to the user.
3. Once the report is run it needs to be emailed to the report owner email address with an excel attachment.
4. Make these reports able to be executed as Django Admin tasks
5. Make a django admin command to import all customers into the system from the reporting system. There are probably < 1000 customers.
Notes this application must run under:
Apache 2x running mod_wsgi
Build using buildout (sample django app already does this)
I have included the sample [url removed, login to view] that I have built. It is pretty much complete. I'm happy to not use the generic relations if they prove difficult to implement.
Brief description of models.
reportingUserAccount: These are the user accounts that we use to connect to our reporting server.
These two tables are not complete but are just the fields that we pass up to the reporting server to get it to return data to us. I have a list of these fields I just have not got to making a DJango model for them yet. There are currently two types of reports that are run product and sales lists.
These tables specify the report jobs, where the data is sent to and which user we run the reports as. Reflecting on this it might be better to be modelled in one model.
customer: This model stores the name and customer id of the customers in the remote reporting system.
reportingRun: Every time a report is run there is an entry in this table.
These tables store the data returned from the remote reporting system. Because the data in the remote system changes over time we may store the same data across multiple runs. This is not going to be an issue.
Looking to make some money?
- Set your budget and the time frame
- Outline your proposal
- Get paid for your work
Bids on this Project
St. Petersburg, Russian Federation
For last 6 years I'm writing in Python. During this period I've successfully finished a lot of projects. Most of them are web applications based on Django, Pyramid, Pylons or Tornado frameworks. Some projects are the collections of scripts for various process automation. My experience is limited by back-end development. But I have strong business relationships with experienced front-end developers so we can build high quality modern web applications.
New Delhi, India
I've around 3 years of web development experience with Django/Python and have worked remotely with various early stage startups. I run a django development studio, see http://cloudshuffle.com. Some prior examples of django related work are: * Teaspiller.com - an online marketplace for tax experts. * FertilityPlanit.com, built their self serve ad network + billing and payments. * Fratmusic.fm - a reddit like music aggregation site with 35-45K unique hits daily. * Tivix.com - a social media platform for fund raising using Facebook. Helped out with S3 issues, scaling, general dev. * Emriq.com - a web based medical records platform for small to medium clinics. * Snapshopinc.com - built the backend for a top 10 ranked iPhone app.
KEY strengths : - Good communication skill. - Having 4+Year Exp. in web solutions & software Development Services - Well organized and collect detail requirement prior to start. - Vast experience with different development environments, coding styles and frameworks. - Follow best practices secure coding and key process documentations. - Provide best and organized database structure for future growth. - Provide easy to upgrade options in coding style. - Maintain daily status reports and well timed delivery. Working in following Systems: 1. PHP Development-Smarty, Cake PHP, Zend Framework ASP/ASP Dot NET Development-AJAX, Web Services 2. Website Design-XHTML, CSS, Flash 3. Search Engine Optimisation, On-Page, Off-page, Internet Marketing, Link Campaign, PPC 4. Web Hosting- Windows/Linux Server Management (Shared or Dedicated), Email Management, Domain Name 5. Application Development - Blackberry Applications, iPhone Applications, Android Applications 6. CMS-Joomla, Drupal, Dot Net Nuke, WordPress 7. CRM Applications-Typo3/Vtiger/SugarCRM 8. Shopping Carts-Zencart /X-cart/ OSCommerce/ Magento 9. Payment Gateway Integration-PayPal/ WorldPay/ Google Checkout/ Protx/ Secpay etc 10. Development and Web Hosting at one place 11. Managed Server Support Services Working in following Technologies: 1. PHP Development 2. ASP/ASP Dot Net Development 3. CRM Applications 4. Website Design 5. Joomla/Drupal CMS Dot Net Nuke 6. OS Commerce 7. Flash Development 8. Cake PHP 9. Sugar CRM Experts 10. Shopping Carts ZenCarts X-Carts Zend Framework Xhtml/CSS Website Design Blackberry Application Development Wordpress CMS Development i-phone Applications Development Typo/Vtiger CRM Applications Payment Gateway Integration PayPal Integration Internet Marketing SEO Google Checkout Integration Working in following domains: 1. Content management system 2. Ecommerce, B2C, B2B 3. Social networking & community websites 4. Job Portals 5. Classifieds websites 6. Forums 7. Blog websites 8. Wikis 9. Search engine optimizations Working in following Business Categories Building / Construction Cars / Automotives Comparison Websites Directories Educational Fashion / Clothing Finance Insurance IT & Technology Jewellery Manufacturing Marketing Medical Music & Entertainment Property Recruitment Restaurants / Food Social Networking Specialists & Services Sports Travel Advertising E-Commerce Finally what about Our Services PHP Development-Smarty, Cake PHP, Zend Framework ASP/ASP Dot NET Development-AJAX, Web Services Website Design-XHTML, CSS, Flash Application Development - Blackberry Applications, iPhone Applications, Android Applications CMS- Joomla, Drupal, Dot Net Nuke, WordPress CRM Applications-Typo3/ Vtiger/SugarCRM Shopping Carts-Zencart X-cart/ OSCommerce/ Magento Payment Gateway Integration-PayPal/ WorldPay/ Google Checkout/ Protx/ Secpay etc
Abakan, Russian Federation
Professional development of web and desktop applications. Technologies: (J)Python, Django, Node.JS, Java, Ruby, Ruby On Rails
I'am Hadi Wijaya Utama, freelance software developer from Indonesia with an special interest on free software, GNU/Linux and Python programming language. 4 years of experience working on Network Application Develops (pcapy, spynner, twisted & scapy), Distributed Messages/Tasks Processing (AMQP & RabbitMQ), Web Development (Django, Pyramid, Tornado, Zope & Google App Engine). Restful service (API) builds, Cloud SaaS specialist.
I'm a senior software engineer alone. Development experience more than ten years. I am good at C/C++ and delphi. In order to operate computer easily, i often develop some tools with python and shell(bash). I am a pragmatic programmer, "Simple Is Beautiful" is my principle of programming. TDD and Scrum are my favorite software process, because it improves quality and more.
Working with Drupal and Django since last three years, experience in scaling complex sites. Also hands on experience with jquery and modern UI/UX.