OgL SSKJr SSKJr SSKJr SSKJr SSKrSSKrSSK r SSK J r \Rr \Rr SrSrS r0r"S S 5r"S S 5rSrSrSrSrSr0SS_SS_SS_SS_SS_SS_SS_SS_S S!_S"S_S#S$_S%S&_S'S(_S)\ R2"5S$- _S*S_S+S,_S-S._S/S0SS0S1S$S2S0S3.ErS4rS5rS6rS7rS8rS9r g):)absolute_import)division)print_function)unicode_literalsN)chainz<>c>\rSrSrSrSrS SjrS SjrS SjrSr g) _MetercXlX lgNfm)selffasttext_modelmeters V/Users/admin/workspace/ai/Jarvis/env/lib/python3.13/site-packages/fasttext/FastText.py__init___Meter.__init__s cURRU5nURRU5nU(a [ U6upEO/SpT[ R "USS9[ R "USS94$)z>Return scores and the gold of each sample for a specific labelFcopy)r get_label_idr scoreVsTruezipnparray)rlabellabel_id pair_listy_scoresy_trues r score_vs_true_Meter.score_vs_truesc66&&u-FF&&x0 "I Hf "Bfxxu-rxxU/KKKrNc0U(a7URRU5nURRU5nOURR 5nU(a [ U6upEO/SpT[ R"USS9[ R"USS94$)zReturn precision/recall curverFr)rrrprecisionRecallCurveLabelprecisionRecallCurverr r!)rr"r#r$ precisionrecalls rprecision_recall_curve_Meter.precision_recall_curve+sx vv**51H88BI335I  #Y Iv!#Rvxx .e0LLLrcU(a8URRU5nURRX15nU$URR U5nU$)z#Return precision for a given recall)rrrprecisionAtRecallLabelprecisionAtRecall)rr-r"r#r,s rprecision_at_recall_Meter.precision_at_recall:sQ vv**51H55hGI008IrcU(a8URRU5nURRX15nU$URR U5nU$)z#Return recall for a given precision)rrrrecallAtPrecisionLabelrecallAtPrecision)rr,r"r#r-s rrecall_at_precision_Meter.recall_at_precisionDsO vv**51HVV228GF VV--i8F rrr) __name__ __module__ __qualname____firstlineno__rr'r.r3r8__static_attributes__rrrr r s L Mrr c"\rSrSrSrS"SjrS#SjrSrSrSr S r S$S jr S$S jr S r S rSrS%SjrSrS&SjrSrSrS'SjrS'SjrS%SjrSrS(SjrS(SjrS)SjrS*SjrSr\S5r\S5r Sr!S r"S!r#g)+ _FastTextOak This class defines the API to inspect models and should not be used to create objects. It will be returned by functions such as load_model or train. In general this API assumes to be given only unicode for Python2 and the Python3 equvalent called str for any string-like arguments. All unicode strings are then encoded as UTF-8 and fed to the fastText C++ API. Nc[R"5UlUbURRU5 SUlSUlUR U5 gr)fasttextr loadModel_words_labelsset_args)r model_pathargss rr_FastText.__init__ZsD""$  ! FF  Z (   drc XU(a#/SQnUHn[X[X55 M gg)N)lrdimwsepochminCount minCountLabelminnmaxnneg wordNgramslossbucketthread lrUpdateRatetr"verbosepretrainedVectors)setattrgetattr)rrI arg_namesarg_names rrG_FastText.set_argsbs, I(&(?@&+ rc6URR5$r)risQuantrs r is_quantized_FastText.is_quantized{svv~~rcNURR5nUR$)z;Get the dimension (size) of a lookup vector (hidden layer).)rgetArgsrM)ras r get_dimension_FastText.get_dimension~s FFNN uu rcUR5n[R"U5nURR X15 [ R "U5$)z&Get the vector representation of word.)rjrCVectorr getWordVectorr r!)rwordrMbs rget_word_vector_FastText.get_word_vectors>  " OOC  Q%xx{rcURS5S:wa [S5eUS- nUR5n[R"U5nUR R X15 [R"U5$)z Given a string, get a single vector represenation. This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed and the null character.  2predict processes one line at a time (remove '\n')) find ValueErrorrjrCrmrgetSentenceVectorr r!)rtextrMrps rget_sentence_vector_FastText.get_sentence_vectorse 99T?b RS S    " OOC    )xx{rc:URRXU5$r)rgetNN)rrokon_unicode_errors rget_nearest_neighbors_FastText.get_nearest_neighborssvv||D%566rc<URRXX4U5$r)r getAnalogies)rwordAwordBwordCrrs r get_analogies_FastText.get_analogiessvv""5;KLLrc8URRU5$)zc Given a word, get the word id within the dictionary. Returns -1 if word is not in the dictionary. )r getWordIdrros r get_word_id_FastText.get_word_ids vv%%rc8URRU5$)zf Given a label, get the label id within the dictionary. Returns -1 if label is not in the dictionary. )r getLabelId)rr"s rr_FastText.get_label_ids vv  ''rc8URRU5$)zG Given a subword, return the index (within input matrix) it hashes to. )r getSubwordId)rsubwords rget_subword_id_FastText.get_subword_idsvv""7++rctURRX5nUS[R"US54$)z4 Given a word, get the subwords and their indicies. r)r getSubwordsr r!)rrorpairs r get_subwords_FastText.get_subwordss4vv!!$9Awa)))rcUR5n[R"U5nURR X15 [ R "U5$)zC Given an index, get the corresponding vector of the Input Matrix. )rjrCrmrgetInputVectorr r!)rindrMrps rget_input_vector_FastText.get_input_vectors@  " OOC  a%xx{rcHSn[U5[:Xa8UVs/sH oe"U5PM nnURRXX45upxXx4$U"U5nURR XX45n U (a [ U 6upO/SpU [ R"U SS94$s snf)aE Given a string, get a list of labels and a list of corresponding probabilities. k controls the number of returned labels. A choice of 5, will return the 5 most probable labels. By default this returns only the most likely label and probability. threshold filters the returned labels by a threshold on probability. A choice of 0.5 will return labels with at least 0.5 probability. k and threshold will be applied together to determine the returned labels. This function assumes to be given a single line of text. We split words on whitespace (space, newline, tab, vertical tab) and the control characters carriage return, formfeed and the null character. If the model is not supervised, this function will throw a ValueError. If given a list of strings, it will return a list of results as usually received for a single line of text. cPURS5S:wa [S5eUS- nU$)Nrtrurvrwrxentrys rcheck _FastText.predict..checks-zz$2% !VWW TMELrrFr)typelistrmultilinePredictpredictrr r!) rrzr thresholdrrr all_labels all_probs predictionsprobslabelss rr_FastText.predicts.  : .23dUE%LdD3$(FF$;$;% !J( (;D&&..)NK #[ 1 v!#Rv288E66 64sBcURR5(a [S5e[R"URR 55$)ze Get a reference to the full input matrix of a Model. This only works if the model is not quantized. Can't get quantized Matrix)rrcrxr r!getInputMatrixrds rget_input_matrix_FastText.get_input_matrixs= 66>>  9: :xx--/00rcURR5(a [S5e[R"URR 55$)zf Get a reference to the full output matrix of a Model. This only works if the model is not quantized. r)rrcrxr r!getOutputMatrixrds rget_output_matrix_FastText.get_output_matrixs= 66>>  9: :xx..011rcURRU5nU(aUS[R"US54$US$)z Get the entire list of words of the dictionary optionally including the frequency of the individual words. This does not include any subwords. For that please consult the function get_subwords. rr)rgetVocabr r!)r include_freqrrs r get_words_FastText.get_wordss?vv/0 GRXXd1g./ /7NrcURR5nUR[R:XaEURR U5nU(aUS[ R"US54$US$URU5$)z Get the entire list of labels of the dictionary optionally including the frequency of the individual labels. Unsupervised models use words as labels, which is why get_labels will call and return get_words for this type of model. rr) rrhmodel model_name supervised getLabelsr r!r)rrrrirs r get_labels_FastText.get_labelssr FFNN  77j++ +66##$45DQ$q'!233Aw>>,/ /rcSn[U5[:Xa2UVs/sH oC"U5PM nnURRX5$U"U5nURR X5$s snf)z Split a line of text into words and labels. Labels must start with the prefix used to create the model (__label__ by default). cPURS5S:wa [S5eUS- nU$)Nrtruz3get_line processes one line at a time (remove '\n')rrs rr!_FastText.get_line..check(s-zz$2% !WXX TMELr)rrrmultilineGetLinegetLine)rrzrrrs rget_line_FastText.get_line"sc   : .23dUE%LdD366**4B B;D66>>$9 9 4sA,c:URRU5 g)z Save the model to the given pathN)r saveModel)rpaths r save_model_FastText.save_model5s rc:URRXU5$)z2Evaluate supervised model using file given by path)rtestrrrrs rr_FastText.test9svv{{4I..rc:URRXU5$)z Return the precision and recall score for each label. The returned value is a dictionary, where the key is the label. For example: f.test_label(...) {'__label__italian-cuisine' : {'precision' : 0.7, 'recall' : 0.74}} )r testLabelrs r test_label_FastText.test_label=svv33rcN[XRRX55nU$r)r rgetMeter)rrrrs r get_meter_FastText.get_meterHstVV__T56 rc JURR5n U(d U RnU(d U RnU(d U RnU(d U R nU(aU(d [ S5eUcSnURRXX4XVXxX5 g)zN Quantize the model reducing the size of the model and it's memory footprint. z"Need input file path if retrainingN)rrhrOrLrXr[rxquantize) rinputqoutcutoffretrainrOrLrXr[dsubqnormris rr_FastText.quantizeMsz" FFNN GGEBXXFiiG 5AB B =E  %Vd rcURRUR[R5UR[R55 g)zS Set input and output matrices. This function assumes you know what you are doing. N)r setMatricesastyper float32)r input_matrix output_matrixs r set_matrices_FastText.set_matricesos9     +]-A-A"**-M rc^URcUR5UlUR$r)rErrds rwords_FastText.wordsxs$ ;; ..*DK{{rc^URcUR5UlUR$r)rFrrds rr_FastText.labels~s$ << ??,DL||rc$URU5$r)rqrs r __getitem___FastText.__getitem__s##D))rcXR;$r)rrs r __contains___FastText.__contains__szz!!r)rFrEr)NNr) strict)r)rr)Fr)rr)ru) NFrFNNNNF)$r:r;r<r=__doc__rrGrerjrqr{rrrrrrrrrrrrrrrrrrrpropertyrrrrr>rrrr@r@OsA2   7M&(, *,7\12 0$:&/ 4     D   *"rr@cUS:Xa[R$US:Xa[R$US:Xa[R$[ S5e)NcbowskipgramrzUnrecognized model name)rrrrrxstrings r_parse_model_stringr sH  """ $$$233rcUS:Xa[R$US:Xa[R$US:Xa[R$US:Xa[R$[ S5e)NnshssoftmaxovazUnrecognized loss name) loss_namer r r rrxrs r_parse_loss_stringrsU ~|| ~||     }}122rc[US5US'[US5US'[US5[:Xa[ US5US'[ R "5nUR5H)up4[X#U5 X1;dMURU5 M+ SUl SUl URS::aURS:XaSUlU$)NrrVautotuneModelSizerrr)r rrintstrrCrIitemsr] setManualoutput saveOutputrUrSrW)rImanually_set_argsrirvs r _build_argsrs'W 6DM%d6l3DL D$ %&#-$'-@(A$B ! A**,a ! KKNAHAL||qQVVq[ HrcN[R"5nURU5$)z?Given a string of text, tokenize it and return a list of tokens)rCtokenize)rzrs rrrsA ::d rc[US9$)z8Load a model given a filepath and return a model object.)rH)r@)rs r load_modelrs  %%rrrrLg?rMdrNrOrPrQrRrSrTrUrrVr rWirXrYrZg-C6?r" __label__rrf1i,)r[r\seedautotuneValidationFileautotuneMetricautotunePredictionsautotuneDurationrcXSSSSSS.n0n[5n[[X 5UR55HIupxXt;aXGnXr;a[ SU-5eXu;a[ SU-5eXU'UR U5 MK UR5HupxXu;dM XU'M XV4$) NrPrUrYr"r\) min_count word_ngramslr_update_rate label_prefixpretrained_vectorsz unexpected keyword argument '%s'z!multiple values for argument '%s')setrrr TypeErroradd) arg_listarg_dictr_default_values param_mapretrr` arg_values r read_argsr:s#(1 I C!&s9'?AQ!R   *H  $>IJ J ??(JK K!H h'"S"0!5!5!7  %M"8  ##rc >[R5nURSSSSSSS.5 /SQn[XX25upE[ XE5n[ US9n[ R"URU5 URURR55 U$) a% Train a supervised model and return a model object. input must be a filepath. The input text does not need to be tokenized as per the tokenize function, but it must be preprocessed and encoded as UTF-8. You might want to consult standard preprocessing scripts such as tokenizer.perl mentioned here: http://www.statmt.org/wmt07/baseline.html The input file must must contain at least one label per line. For an example consult the example datasets which are part of the fastText repository such as the dataset pulled by classification-example.sh. g?rrr r)rLrPrRrSrVr)rrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr"r[r\r&r'r(r)r*rrI) unsupervised_defaultrupdater:rr@rCtrainrrGrh)kargskwargssupervised_defaultr_rIrrifts rtrain_supervisedrDs.224!   I6(yUDD,A  B NN244KK  Irc/SQn[XU[5up4[X45n[US9n[R "UR U5 URUR R55 U$)aZ Train an unsupervised model and return a model object. input must be a filepath. The input text does not need to be tokenized as per the tokenize function, but it must be preprocessed and encoded as UTF-8. You might want to consult standard preprocessing scripts such as tokenizer.perl mentioned here: http://www.statmt.org/wmt07/baseline.html The input field must not contain any labels or use the specified label prefix unless it is ok for those words to be ignored. For an example consult the dataset pulled by the example script word-vector-example.sh, which is part of the fastText repository. )rrrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr"r[r\r<) r:r=rr@rCr?rrGrh)r@rAr_rIrrirCs rtrain_unsupervisedrF5s^I,(yBVWDD,A  B NN244KK  Irc[S5e)Nz`cbow` is not supported any more. Please use `train_unsupervised` with model=`cbow`. For more information please refer to https://fasttext.cc/blog/2019/06/25/blog-post.html#2-you-were-using-the-unofficial-fasttext-module Exceptionr@rAs rrras  g rc[S5e)Nz`skipgram` is not supported any more. Please use `train_unsupervised` with model=`skipgram`. For more information please refer to https://fasttext.cc/blog/2019/06/25/blog-post.html#2-you-were-using-the-unofficial-fasttext-modulerHrJs rrrgs  o rc[S5e)Nz`supervised` is not supported any more. Please use `train_supervised`. For more information please refer to https://fasttext.cc/blog/2019/06/25/blog-post.html#2-you-were-using-the-unofficial-fasttext-modulerHrJs rrrms  Y r)! __future__rrrrfasttext_pybindrCnumpyr multiprocessing itertoolsrrrEOSBOWEOWdisplayed_errorsr r@r rrrr cpu_countr=r:rDrFrrrrrrrWs'%'"        22jy"y"x 4 3 $ &  Z$ 3 !   Q   Q A A 1! D g o'')A-C !" [#$  3:$89x)X  r