a Kb@s`ddlZddlmZddlmZddlmZddlmZGdddej Z Gdd d ej Z dS) N)closing)data) PorterStemmer)SnowballStemmerc@s4eZdZddZddZddZddZd d Zd S) SnowballTestcCs\tdd}|ddksJ|ddks.J|ddks@J|dd ksRJ|d d ksdJ|d d ksvJ|d d ksJ|d d ksJ|ddksJtdd}|d dksJ|dd ksJ|ddksJtd}|ddksJ|ddksJ|ddks0J|dd ksDJ|ddksXJdS)z this unit testing for test the snowball arabic light stemmer this stemmer deals with prefixes and suffixes arabicTu&الْعَرَبِــــــيَّةuعربuالعربيةu فقالواuقالuالطالباتuطالبuفالطالباتuوالطالباتuالطالبونu اللذانuمنFuاللذuالكلماتuكلمNrstem)selfZ ar_stemmerr 7lib/python3.9/site-packages/nltk/test/unit/test_stem.py test_arabic s(  zSnowballTest.test_arabiccCstd}|ddksJdS)Nrussianuавантненькаяuавантненькr)r Zstemmer_russianr r r test_russian'szSnowballTest.test_russiancCs`td}tddd}|ddks&J|ddks8J|ddksJJ|ddks\JdS)NgermanT)Zignore_stopwordsu SchränkeZschrankZkeinenZkeinr)r Zstemmer_germanZstemmer_german2r r r test_german+s  zSnowballTest.test_germancCs0td}|ddksJ|ddks,JdS)NspanishZ VisionadoZvisionZalgueZalgurr stemmerr r r test_spanish5szSnowballTest.test_spanishcCstd}|ddksJdS)Nenglishzy'syrrr r r test_short_strings_bug=sz#SnowballTest.test_short_strings_bugN)__name__ __module__ __qualname__r rrrrr r r r r s  rc@sDeZdZddZddZddZddZd d Zd d Zd dZ dS) PorterTestcCsFttdjdd}|WdS1s80YdS)Nz*stemmers/porter_test/porter_vocabulary.txtutf-8encoding)rrfindopenread splitlinesr fpr r r _vocabularyCs zPorterTest._vocabularycCsLt|d}t||D].\}}||}||ksJd||||qdS)N)modez*{} should stem to {} in {} mode but got {})rzipr&r format)r Z stemmer_modeZexpected_stemsrZwordZ true_stemZour_stemr r r _test_against_expected_outputKs  z(PorterTest._test_against_expected_outputcCsPttdjdd&}|tj|Wdn1sB0YdS)azTests all words from the test vocabulary provided by M Porter The sample vocabulary and output were sourced from https://tartarus.org/martin/PorterStemmer/voc.txt and https://tartarus.org/martin/PorterStemmer/output.txt and are linked to from the Porter Stemmer algorithm's homepage at https://tartarus.org/martin/PorterStemmer/ z-stemmers/porter_test/porter_martin_output.txtrrN) rrr r!r*rZMARTIN_EXTENSIONSr"r#r$r r r test_vocabulary_martin_modeXs  z&PorterTest.test_vocabulary_martin_modecCsPttdjdd&}|tj|Wdn1sB0YdS)Nz+stemmers/porter_test/porter_nltk_output.txtrr) rrr r!r*rZNLTK_EXTENSIONSr"r#r$r r r test_vocabulary_nltk_modejs z$PorterTest.test_vocabulary_nltk_modecCstttdjdd&}|tj|Wdn1sB0Y|tjtdjdddS)Nz/stemmers/porter_test/porter_original_output.txtrr) rrr r!r*rZORIGINAL_ALGORITHMr"r#r$r r r test_vocabulary_original_modets "  z(PorterTest.test_vocabulary_original_modecCstddksJdS)zTest for bug https://github.com/nltk/nltk/issues/1581 Ensures that 'oed' can be stemmed without throwing an error. ZoedoNrr )r r r r test_oed_bugszPorterTest.test_oed_bugcCslt}|ddksJ|ddks*J|jddddks@J|ddksRJ|jddddkshJd S) zTest for improvement on https://github.com/nltk/nltk/issues/2507 Ensures that stems are lowercased when `to_lowercase=True` ZOnZonIiF)Z to_lowercaseZGithubZgithubNr/)r Zporterr r r test_lowercase_options z PorterTest.test_lowercase_optionN) rrrr&r*r+r,r-r0r3r r r r rBs  r) Zunittest contextlibrZnltkrZnltk.stem.porterrZnltk.stem.snowballrZTestCaserrr r r r s     9