o /ѹg0@sddlZddlmZddlmZmZmZddlmZm Z m Z ddl m Z ddl mZddlmZe ZGd d d ZGd d d ZdS) N)Optional)GitErrorInvalidParameterNotLoginException)DEFAULT_DATASET_REVISIONDEFAULT_REPOSITORY_REVISIONMASTER_MODEL_BRANCH) get_logger)GitCommandWrapper get_endpointc @seZdZdZeddfdededeedeedeef dd Zd d Zd d Z d%dedefddZ defddZ eedfdedeedeedee fddZ efdeded eefd!d"Zefdeded eefd#d$ZdS)& Repositoryz8A local representation of the model git repository. N model_dir clone_fromrevision auth_tokengit_pathc Cs ||_tj||_tj||_|sd}t|ddlm }|r%||_ n| |_ t }| s6tdt ||_tj|jdd||} t|jr`|} |j| } | r`| | kr`dS|j|j|j | |j|| rw||j|j|j|j|j r|j|j|j dSdS)a"Instantiate a Repository object by cloning the remote ModelScopeHub repo Args: model_dir (str): The model root directory. clone_from (str): model id in ModelScope-hub from which git clone revision (str, optional): revision of the model you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' Raises: InvalidParameter: revision is None. 0a non-default value of revision cannot be empty.rModelScopeConfigz)git lfs is not installed, please install.Texist_okN)rospathdirnamemodel_base_dirbasenamemodel_repo_namermodelscope.hub.apirr get_tokenr Zis_lfs_installedloggererror git_wrappermakedirs_get_model_id_urllistdir_get_remote_urlremove_token_from_urlcloneZgit_lfs_install add_user_infoconfig_auth_token) selfrrrrrerr_msgrr#url remote_urlr0]/Users/admin/.pyenv/versions/3.10.0/lib/python3.10/site-packages/modelscope/hub/repository.py__init__s@          zRepository.__init__cCstd|d}|S)N/.gitr )r,Zmodel_idr.r0r0r1r%PszRepository._get_model_id_urlcC.z |j|j}W|Styd}Y|SwN)r#get_repo_remote_urlrrr,remoter0r0r1r'T zRepository._get_remote_urloriginmasterr9branchcCs|jj|j||ddS)zPull remote branch Args: remote (str, optional): The remote name. Defaults to 'origin'. branch (str, optional): The remote branch. Defaults to 'master'. )r9r=N)r#pullr)r,r9r=r0r0r1r>[szRepository.pullfile_name_suffixcCs"td|tj|jdfdS)zAdd file suffix to lfs list. Args: file_name_suffix (str): The file name suffix. examples '*.safetensors' z5printf ' %s filter=lfs diff=lfs merge=lfs -text '>>%sz.gitattributesN)rsystemrjoinr)r,r?r0r0r1 add_lfs_typeds zRepository.add_lfs_typeFcommit_message local_branch remote_branchforcecCs|dus t|tsd}t|t|tstd|jstd|j|j|j|j |j |j |j |j}|jj |jdd|j|j||jj|j|j|||ddS)a Push local files to remote, this method will do. Execute git pull, git add, git commit, git push in order. Args: commit_message (str): commit message local_branch(str, optional): The local branch, default master. remote_branch (str, optional): The remote branch to push, default master. force (bool, optional): whether to use forced-push. Raises: InvalidParameter: no commit message. NotLoginException: no auth token. N commit_message must be provided!force must be bool'Must login to push, please login first.T all_filesrepo_dirtokenr.rDrE) isinstancestrrboolrrr#r+rr*rrr7addcommitpush)r,rCrDrErFmsgr.r0r0r1rTos*   zRepository.pushtag_namemessagerefcCsR|dus|dkrd}t||dus|dkrd}t||jj|j|||ddS)aCreate a new tag. Args: tag_name (str): The name of the tag message (str): The tag message. ref (str, optional): The tag reference, can be commit id or branch. Raises: InvalidParameter: no commit message. NzFWe use tag-based revision, therefore tag_name cannot be None or empty.z=We use annotated tag, therefore message cannot None or empty.)rMrVrWrX)rr#tagr)r,rVrWrXrUr0r0r1rZs zRepository.tagcCs$|||||jj|j|ddS)zCreate tag and push to remote Args: tag_name (str): The name of the tag message (str): The tag message. ref (str, optional): The tag ref, can be commit id or branch. Defaults to MASTER_MODEL_BRANCH. )rMrVN)rZr#Zpush_tagr)r,rVrWrXr0r0r1 tag_and_pushs zRepository.tag_and_push)r;r<)__name__ __module__ __qualname____doc__rrPrr2r%r'r>rBrQrTrrZr[r0r0r0r1rs` ;   - rc @seZdZdZeddfdededeedeedeef dd Zd efd d Zed fdedeedee fddZ ddZ ddZ dS)DatasetRepositoryzEA local representation of the dataset (metadata) git repository. N repo_work_dir dataset_idrrrcCs||_|r t|tsd}t||d|_|jsd}t|tj|j|_ tj |j|_ |s7d}t|||_ ddl m}|rF||_n||_t||_tj|jdd|j|d |_d S) au Instantiate a Dataset Repository object by cloning the remote ModelScope dataset repo Args: repo_work_dir (str): The dataset repo root directory. dataset_id (str): dataset id in ModelScope from which git clone revision (str, optional): revision of the dataset you want to clone from. Can be any of a branch, tag or commit hash auth_token (str, optional): token obtained when calling `HubApi.login()`. Usually you can safely ignore the parameter as the token is already saved when you login the first time, if None, we will use saved token. git_path (str, optional): The git command line path, if None, we use 'git' Raises: InvalidParameter: parameter invalid. z"dataset_work_dir must be provided!r3z%dataset_work_dir can not be root dir!rrrTr)rbN)rbrOrPrrstriprarrr repo_base_dirr repo_namerrrrr r r#r$ _get_repo_urlrepo_url)r,rarbrrrr-rr0r0r1r2s*    zDatasetRepository.__init__returncCsht|jr|}|j|}|r||jkrdStd |j|j |j |j |j|j |j|jS)NrYzCloning repo from {} )rr&rar'r#r(rgr!infoformatr)rdrrer)r,r/r0r0r1r)s   zDatasetRepository.cloneFrCr=rFcCs|dus t|tsd}t|t|tstd|jstd|j|j|j|j |j |j | }|j |}|j|j|jj|jdd|j|j||jj|j|j|||ddS)aPush local files to remote, this method will do. git pull git add git commit git push Args: commit_message (str): commit message branch (str, optional): which branch to push. force (bool, optional): whether to use forced-push. Raises: InvalidParameter: no commit message. NotLoginException: no access token. NrGrHrITrJrL)rOrPrrQrrr#r+rar*rdrer'r(r>rRrSrT)r,rCr=rFrUr/r0r0r1rTs*   zDatasetRepository.pushcCstd|dS)Nz /datasets/r4r )r,rbr0r0r1rf1szDatasetRepository._get_repo_urlcCr5r6)r#r7rarr8r0r0r1r'4r:z!DatasetRepository._get_remote_url) r\r]r^r_rrPrr2r)rQrTrfr'r0r0r0r1r`s8 / - r`)rtypingrZmodelscope.hub.errorsrrrZmodelscope.utils.constantrrrZmodelscope.utils.loggerr gitr Z utils.utilsr r!rr`r0r0r0r1s    3