Make program in python for plagiatrizm detection

Plagiarism detection - NEED IT IN 5 days TOPS! Find common phrases and sentences between documents (source and suspicious). Find all plagiarized parts. There can be 4 cases: - Copy paste - Copy paste + word order change - Copy paste + paraphrasing - Copy paste + word order change and paraphrasing I suggest here using multithreaded Needleman Wunch algorithm for document similarity comparison and plWordnet for synonyms and paraphrasing checks. Input file There are sets of document pairs suspected to be plagiarized and files that previous have been plagiarized from. In plain text format named as [login to view URL], [login to view URL], where XXXX in pair number, in first there is something plagiarized from the second one. Output file In plagiarism was detected we want to save all information in XML file as follows: <?xml version="1.0" encoding="UTF-8"?> <alignment document="[login to view URL]" source="[login to view URL]"> <passage documentFrom="123" documentTo="123" sourceFrom="123" sourceTo="123" /> <passage documentFrom="234" documentTo="234" sourceFrom="234" sourceTo="234" /> </alignment> Tag passage means that plagiarism was detected: • documentFrom – beginning index of recognized plagiarized fragment from [login to view URL] • documentTo- ending index of recognized plagiarized fragment from [login to view URL], • sourceFrom- beginning index of recognized plagiarized fragment from [login to view URL], • sourceTo- ending index of recognized plagiarized fragment from [login to view URL] m. save all to suspiciousXXXX-sourceXXXX.xml. For entire task, it will be a set of XML files. Measures In order to measure quality, I will use • precision,: Claude, Webb, Geoffrey I., “Encyclopedia of Machine Learning and Data Mining Sammut”, 2017, precision • recall: Claude, Webb, Geoffrey I., “Encyclopedia of Machine Learning and Data Mining Sammut”, 2017, precision and recall • granularity,: Potthast, Martin, et al. “An evaluation framework for plagiarism detection.” Proceedings of the 23rd international conference on computational linguistics: Posters. Association for Computational Linguistics, 2010. • pladget score (main score),: Potthast, Martin, et al. “An evaluation framework for plagiarism detection.” Proceedings of the 23rd international conference on computational linguistics: Posters. Association for Computational Linguistics, 2010. Trial set Trial set is attahed: • pl/en – division between PL and EN documents, • src (inside pl/en) – source documents, • susp (inside pl/en) – suspicious documents, • xml (inside pl/en) – proper answers. Evaluation tool Is attached as JAR file that needs newest Java 8. Arguments: • -e evaluation method, • -i path to ZIP file with reesulting XML files, • -t path to folder with answers. Example: java -jar [login to view URL] -i c:\\[login to view URL] -t c:\\dataset -e TASK1 Baseline THE BASELINE SLUTION TO THIS TASK in general can be based on suffix array. To find Longest Common Substring between documents. In pre processing this documents will be: • Remove special characters, • Normalize white symbols in text, • Remove EN stop-words, • Remove PL stop-words, Such data is then divided in 15-grams phrases and put into suffix array. The result of this is as follows: • precision: 0.861901, recall 0.123821, granularity: 1.352459, plagdet: 0.175451 Nong, Ge, Sen Zhang, and Wai Hong Chan. “Linear suffix array construction by almost pure induced-sorting.” Data Compression Conference, 2009. DCC’09.. IEEE, 2009.