o 4ѹg !@s<ddlZddlmZddlZddlmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZd e ed efddZdeded e ede ededede ede ededededededeede ede ef dd Zd ejfd!d"Zd&d#d$Zed%kredSdS)'N)Counter)Path)List)Optional)get_commandline_args)build_tokenizer) TextCleaner) g2p_classes)str2bool) str_or_nonefieldreturncCs|}zBd|vr5|jddd\}}|dkrd}n t|}|dkr'td|dkr0d}nt|}nt|}|d}|dkrEtdWntyTtd |w|dur`td|}|St|d|}|S) aConvert field string to slice Note that field string accepts 1-based integer. Examples: >>> field2slice("1-") slice(0, None, None) >>> field2slice("1-3") slice(0, 3, None) >>> field2slice("-3") slice(None, 3, None) -)maxsplitNrz1-based stringzmust be 1 or more valuez)Format error: e.g. '2-', '2-5', or '-5': )stripsplitint ValueError RuntimeErrorslice)r s1s2Zslicr\/Users/admin/.pyenv/versions/3.10.0/lib/python3.10/site-packages/funasr/bin/tokenize_text.py field2slices2      rinputoutput delimiter token_type space_symbolnon_linguistic_symbolsbpemodel log_levelwrite_vocabularyvocabulary_sizeremove_non_linguistic_symbolscutoff add_symbolcleanerg2pc stj|dd|dkrtj}n t|jddd}|dkr tj}nt|}|jjddd|jd dd}t |}t |||||| |d }t }|durMt |}|D]E}| }|durq||}||}|durld |}n||}||}||}| s|d |d qO|D] }||d 7<qqO| sdS| D]%}z |d\}}Wntytd|w|}||vr||=qttfddt|ddd}| dkr| t| krtd| |d| t| }| D]8}z |d\}}t|}Wntytd|w|}|dkr!t|d |}|||dfq|D] \}}||d q,t|}tdd|D}td|||dddS)Nz>%(asctime)s (%(module)s:%(lineno)d) %(levelname)s: %(message)s)levelformatrrzutf-8)encodingT)parentsexist_okw)r r#rr!r"r'Zg2p_type  r:z Format error: e.g. ':0': cs |dkSNrrxr(rrs ztokenize..cSs |d Sr6rr7rrrr:s )keyrzvocabulary_size is too small: css |] \}}|dur|VqdSNr).0r2crrr sztokenize..z OOV rate = dz %)logging basicConfigsysstdinropenstdoutparentmkdirrrrrrstriprjoinZ text2tokenswriterrrlistfiltersorteditemslenrinsertsumvaluesinfo) rrr rr r!r"r#r$r%r&r'r(r)r*r+ZfinZfoutp tokenizercounterlinetokenstZ symbol_and_idsymbolidxZwords_and_countsr2r>Z total_countZ invocab_countrr9rtokenizeAs            "r]cCsLtjdtjd}|jddddddd |jd d d d d|jddd dd|jdddd|jdddgddd|jddddd|jd d!d"d|jd#dd$d|jd%td&d'|jd(td)d*d+|jd,tgd-dd.d/|jd0ttdd1d/|d2}|jd3td)d4d+|jd5td6d7d+|jd8d6td9d:|jd;t gd|S)?NzTokenize texts) descriptionZformatter_classz --log_levelcSs|Sr<)upperr7rrrr:szget_parser..INFO)CRITICALERRORWARNINGr`DEBUGNOTSETzThe verbose level of logging)typedefaultchoiceshelpz--inputz-iTz!Input text. - indicates sys.stdin)requiredriz--outputz-oz#Output text. - indicates sys.stdoutz--fieldz-fz?The target columns of the input text as 1-based integer. e.g 2-)riz --token_typez-tchar)rkZbpewordZphnz Token type)rgrhriz --delimiterz-dz The delimiter)rgriz--space_symbolzzThe space symbolz --bpemodelzThe bpemodel file pathz--non_linguistic_symbolsz non_linguistic_symbols file path)rfriz--remove_non_linguistic_symbolsFz'Remove non-language-symbols from tokens)rfrgriz --cleaner)NZtacotronZjaconvZ vietnameseZkorean_cleanerzApply text cleaning)rfrhrgriz--g2pz&Specify g2p method if --token_type=phnzwrite_vocabulary mode relatedz--write_vocabularyz4Write tokens list instead of tokenized text per linez--vocabulary_sizerzVocabulary sizez--cutoffz0cut-off frequency used for write-vocabulary mode)rgrfriz --add_symbolappendzBAppend symbol e.g. --add_symbol ':0' --add_symbol ':1')rfrgactionri) argparseArgumentParserZArgumentDefaultsHelpFormatter add_argumentr r r add_argument_grouprstr)parsergrouprrr get_parsers rvcCs:tttjdt}||}t|}tdi|dS)N)filer)printrrCstderrrv parse_argsvarsr])cmdrtargskwargsrrrmain s  r__main__r<)ro collectionsrrApathlibrrCtypingrrZfunasr.utils.cli_utilsrZ funasr.tokenizer.build_tokenizerrZfunasr.tokenizer.cleanerrZ"funasr.tokenizer.phoneme_tokenizerr Zfunasr.utils.typesr r rsrrboolrr]rprvr__name__rrrrsh          .      z Q