Wordpress Plugin: Sitemap Scraper and Visual Page/Category Creation

In Progress

Plugin Overview

This plugin is meant to help someone build a brand new website STRUCTURE in wordpress. Meaning, analyze the top 10 results in Google, extract out the page and category and post titles from sitemaps (or crawl through a site manually) and then allow the user to visually pick and choose which titles to use for home pages/category titles or post titles. I broke down the workflow in 2 stages.

Stage 1


A) Allow for user to enter in a keyword

B) Scrape the top 10 results of google and get the domain name.

C) Put those top 10 domains into an array [1-10]

D) Now, parse through each domain and see if it has a sitemap attached to it - ie - [url removed, login to view]

E) If it does - Extract out the category/page/post names from the sitemap

E) If it does not, use scrapy or some other php class to scrape through a website based on 'X' level of depth to find all the internal links

i. User input in 2 as the level depth, and the plugin will parse 2 levels deep to identify the internal link structure. Maybe use the following routines (or variations obviously)?

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

F) The plugin will then give scores to each of the page title/category title/post title (the first website in rank one gets 100 points for each title/category since Google has ranked it as being important first, and each site under it starts with a sliding value due to it's position..2nd website starts with 90, third starts with 80, and so forth. If there are any titles that MATCH the original 1st position website, that title will get a score of 100)

Please see referenced document

Stage 2


This is the trickier part, as the above is more along the lines of scraping and parsing with regex or some other DOM component. Stage 2 now provides those Titles and Categories in a visual box list to enable a user to manually drag and drop and create their structure based on the titles and categories. There is an abandoned plugin in the wordpress plugin directory called Visual Site Manager. It sort of doesn't work 100% right now because the devs haven't updated it.

[url removed, login to view]

here's a good write up of the plugin:

[url removed, login to view]

You can test install it and see how it works, but again, all the funcationality isn't there right now.

What I like about it is the usage of JIT Spacetree Visualization.

[url removed, login to view]

Obviously if it's a brand new site, all that the main canvas would have is the top box representing the root of the site. By being able to drag and drop each box onto the tree map that JIT enables you to build, one can take a predefined set of titles scraped and ranked in order of importance from other known authority sites and then build a site structure based on that.

Now, if someone has an existing site, then the main JIT canvas will show the existing site structure and allow someone to change titles on there (click on node, bring up edit text screen to change info) or just drag and drop new categories/pages/postings into the site structure.

Please have wordpress, jit, and scraping experience

Skills: MySQL, PHP, Web Scraping, WordPress

See more: jit spacetree, wordpress php sitemap, www dom com, write xml code website, wordpress website manager, wordpress demos, what is a tree node, what ie wordpress, website creation with wordpress, website creation in google, website code in wordpress, web devs, visual workflow, tree to box, tree in order, top 10 work from home sites, sort routines, scrapy org, s.a.p. experience, root info, regex match list, regex info, php parse tree, parsing input, order of tree

Project ID: #4987553

2 freelancers are bidding on average $165 for this job


Dear Client, I can help in your project. We have already experience of working on similar projects. Please see below to get idea of our experience: Amazon/Ebay Bots: http://sigma-dns.sigmavirtual.com/PDemo1/Am More

$144 USD in 3 days
(259 Reviews)

Your requirement is very much clear to us so we are ready to start immediately. Thanks Chirag our past job for data mining and scraping https://www.freelancer.com/projects/Data-Mining/Data-Mining-from-web-site.ht More

$187 USD in 3 days
(35 Reviews)

Hi, we are one of the best developers on freelancer. We can provide you an excellent solution as we are professional and experience. We can develop your desire application within timeline. Please see my PMB. Have a nic More

$180 USD in 5 days
(16 Reviews)

Hello Sir, We have checked your job detail and as per that we are able to do your job and we have very good experience in Word Press, Joomla, Joomla component, Modules, PHP, HTML,Magento etc . As well knowledge on More

$144 USD in 3 days
(15 Reviews)

sir i am a software engineer and i am interested in this task i have 3 years experience in this field i did too many projects and know recently start freelancing to prove my skills and expertise if you are interested t More

$185 USD in 15 days
(18 Reviews)

Hi, We are very clear with the specification mentioned and ready to start your project immediately. We are pleased about having the opportunity to work together. Kindly do review PMB for further details. Sir , TI Wo More

$142 USD in 6 days
(3 Reviews)

Respected Customer G'Day,,This is Hussein in charge of sales/Marketing. our strong technical experts will explain you every function and beat of features. After clarifications if you feel comfortable and satisfied t More

$233 USD in 5 days
(1 Review)