Text mining in Python to identify interesting document files

Closed

I want to do a simple text mining task on a large number of files with python code.

The files are stored in a few larger network shares and sum up to about a million of files, with about 100 different filetypes. The "text and document" filetypes that are considered extra interesting are microsoft office files, pdf and text (doc, docx, xls, xlsx, xlsm, ppt,pptx, pdf, txt, html, xml, etc..). But there are also a few binary files, movies and others that are considered as less interesting. I have a list of about 100 interesting words organized in a textfile (txt) , one word per row. Now I want to identify all files that contain one or more instances of the words in their filename, path or in their file contents. I would like to get this task solved in python. I am not an experienced python programmer and I would like to get the code well written, well annotated and easy to modify. Communication and code should be in english. The code should work preferable in Windows, MacOSX and Linux.

I would like

1) A script to list all the files (not folders) in a networks share. Number the list, one file per row. List interesting file information, Columns separated by semicolon (;). Something like this.

File Counter; Full path; creation date; modifcation date; file owner, filetype; etc..

1; C:/mypath/[url removed, login to view]; date; date; owner; text document

2; C:/mypath/[url removed, login to view]; date; date; owner; Microsoft Word

2) One or several scripts to scan the names and contents (based on the file of interesting words) of the files from script 1). "Document filetypes and textfiles" (see above) shall be scanned for both the content and the full path (filname+path). Other files don't have to be scanned for content but need to be scanned for the full path. The script(s) shall report the action on each file and the result (name scan: yes/no/error, content scan: yes/no/error). If an error occurs with a file (e.g. for reading or parsing) this need to be stated in the result of the file scan, but should not interrupt the scan. The number of matches in the content and by which word should be stated. The number of unique positive (identified) words in the list of words, i.e. must be 0-about 100 should also be stated for each file.

Final output should be a similar list as in 1) but with additional columns containing. e.g.

Path Scan; Content Scan; #Matches; Words matched; #Unique matches

(Yes/No/Error); (Yes/No/Error); Integer; Monkey, Bannanas; 0-100,

Provide me with a way to run this search as easy and quickly as possible. (There are a lots of files and speed is important.) I don't want to wait two weeks for the search and I don't want to find out that the scripts run into an error after three hour and stops.

The bidders with recommendations, high evaluations, strong background in python and text mining are preferred.

Skills: Big Data, Data Mining, Python

See more: text mining python, python text mining, python text mining pdf, report mining python, ppt python text mining, interesting text mining python, python list files network share, python mining files, python code identify text, identify text python, text mining python identify interesting document files, yes positive communication, yes in binary, windows action script, text now, sum binary, speed date script, simple binary code, simple binary, search binary, python find, programmer in english, now to get a programmer, need python programmer, list get python

Project ID: #2473761

Awarded to:

Blender3D

I have significant experience with data extraction and machine learning, so creating a script like this should not be a problem for me. Documenting the code and communicating with you won't be an issue either, as I spe More

$220 CAD in 10 days
(18 Reviews)
5.1

8 freelancers are bidding on average $209 for this job

RhynoProgramming

I have experience in text mining, but I will have to learn some things about network interfacing. I will complete this project in the time allotted.

$250 CAD in 14 days
(6 Reviews)
4.3
psfever

Hi Sir, i am good with Python and have practiced Data Mining since 2 years.

$250 CAD in 10 days
(7 Reviews)
3.6
antfilfon

I have very good experience in data mining algorithms and Python.

$200 CAD in 10 days
(1 Review)
3.2
serenek

I have extensive knowledge in text mining and data mining as I had taken courses related to those while at school(Stanford University)

$50 CAD in 2 days
(0 Reviews)
0.0
azlyth

Being a system administrator who deals in Python, this is right up my alley. We'll just have to clarify what your network setup is.

$200 CAD in 3 days
(0 Reviews)
0.0
jonincanada

Sounds like a hefty challenge. I'm up for that. Based in Toronto

$250 CAD in 3 days
(0 Reviews)
2.5
F0m6pPB1U

Custom software development (<b><i>Removed by Admin</i></b>)

$250 CAD in 1 day
(0 Reviews)
0.0