o
/ѹg/ @ s d dl Z d dlZd dlZd dlZd dlmZmZ d dlmZ d dl m
Z
m Z d dl m
Z
d dlmZ e Zee d dkZ G d d
d
eZG d d d eZdS )
N)movermtree)Dict) FILE_HASH)MODELSCOPE_ENABLE_DEFAULT_HASH_VALIDATION) compute_hash)
get_loggerFalsetruec @ st e Zd ZdZdZdZdZ defddZdd Z d
d Z
d d
Z dd Z dd Z
dd Zdd Zdd Zdd ZdS )FileSystemCachez.mscz.mdlidz.mvcache_root_locationc K s t j|dd || _| dS )zBase file system cache interface.
Args:
cache_root_location (str): The root location to store files.
kwargs(dict): The keyword arguments.
Texist_okN)osmakedirsr
load_cache)selfr
kwargs r `/Users/admin/.pyenv/versions/3.10.0/lib/python3.10/site-packages/modelscope/hub/utils/caching.py__init__ s zFileSystemCache.__init__c C s | j S N)r
r r r r get_root_location. s z!FileSystemCache.get_root_locationc C sd g | _ tj| jtj}tj|r0t|d}t
|| _ W d d S 1 s)w Y d S d S )Nrb) cached_filesr pathjoinr
r
KEY_FILE_NAMEexistsopenpickleload)r cache_keys_file_pathfr r r r 1 s
"zFileSystemCache.load_cachec C sd t j| jtj}t \}}t|d}t
| j | W d n1 s&w Y t || dS )zSave cache metadata.wbN)
r r r r
r r tempfilemkstempr! r" dumpr r )r r$ fdfnr% r r r save_cached_files9 s
z!FileSystemCache.save_cached_filesc C dS )zCheck the key is in the cache, if exist, return the file, otherwise return None.
Args:
key(str): The cache key.
Raises:
None
Nr r keyr r r get_fileD s zFileSystemCache.get_filec C r- )zPut file to the cache.
Args:
key (str): The cache key
location (str): Location of the file, we will move the file to cache.
Raises:
None
Nr )r r/ locationr r r put_fileO s
zFileSystemCache.put_filec C s& || j v r| j | | dS dS )zvRemove cache key in index, The file is removed manually
Args:
key (dict): The cache key.
N)r remover, r. r r r
remove_key[ s
zFileSystemCache.remove_keyc C s | j D ] }||kr dS qdS )NTF)r )r r/ Z
cache_filer r r r e s
zFileSystemCache.existsc C s t | j | dS )zRemove all files and metadata from the cache
In the case of multiple cache locations, this clears only the last one,
which is assumed to be the read/write one.
N)r r
r r r r r clear_cachel s
zFileSystemCache.clear_cachec C s t | S r )hashlibsha256encode hexdigestr. r r r hash_namet s zFileSystemCache.hash_nameN)__name__
__module__ __qualname__r MODEL_META_FILE_NAMEMODEL_META_MODEL_IDMODEL_VERSION_FILE_NAMEstrr r r r, r0 r2 r4 r r5 r: r r r r r s$
r c s e Zd ZdZd fdd Zdd Zdd Zd efd
d Zd d
Z dd Z
dd Z dd Z dd Z
dd Zdd Zdd Zd d Z ZS ) ModelFileSystemCachezLocal cache file layout
cache_root/owner/model_name/individual cached files and cache index file '.mcs'
Save only one version for each file.
Nc sp |du s|du rt tj| | nt tj||| tjd||f i| _| |
| _ dS )aY Put file to the cache
Args:
cache_root(`str`): The modelscope local cache root(default: ~/.cache/modelscope/)
owner(`str`): The model owner.
name('str'): The name of the model
Returns:
Raises:
None
model_id = {owner}/{name}
Nz%s/%s) superr r r r load_model_metar r?
model_metasave_model_metaload_model_versionZcached_model_revision)r Z
cache_rootownername __class__r r r ~ s
z ModelFileSystemCache.__init__c C sj t j| jtj}t j|r-t|d}t || _
W d d S 1 s&w Y d S tj di| _
d S )Nr unknown) r r r r
r r> r r! r" r# rE r? r Zmeta_file_pathr% r r r rD s
"z$ModelFileSystemCache.load_model_metac C s\ t j| jtj}t j|r,t|d}| W d S 1 s%w Y d S d S )Nr)
r r r r
r r@ r r! readstrip)r model_version_file_pathr% r r r rG s
$z'ModelFileSystemCache.load_model_version
revision_infoc C s t j| jtj}t|d+}t|tr$d|d |d f }| | n
| | W d d S W d d S 1 s r! r" r) rE rM r r r rF s
"z$ModelFileSystemCache.save_model_metac C sN | j D ]!}||d kr$tj| j|d }tj|r | S | | qdS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
Returns:
path: the full path of the file.
PathN)r r r r r
r r4 )r file_path cached_filecached_file_pathr r r get_file_by_path s
z%ModelFileSystemCache.get_file_by_pathc C sj | j D ]/}||d kr2|d |s||d r2tj| j|d }tj|r-| S | | qdS )zRetrieve the cache if there is file match the path.
Args:
file_path (str): The file path in the model.
commit_id (str): The commit id of the file
Returns:
path: the full path of the file.
rZ rT N)r
startswithr r r r
r r4 )r r[ commit_idr\ r] r r r get_file_by_path_and_commit_id s
z3ModelFileSystemCache.get_file_by_path_and_commit_idc C sZ | |}| jD ]"}||kr*tj| j|d }tj|r"| S | | dS qdS )zCheck if exist cache file.
Args:
model_file_info (ModelFileInfo): The file information of the file.
Returns:
str: The file path.
rZ N)$_ModelFileSystemCache__get_cache_keyr r r r r
r r4 )r model_file_info cache_keyr\ orig_pathr r r get_file_by_info s
z%ModelFileSystemCache.get_file_by_infoc C s |d |d d}|S )NrZ rT )rZ rT r )r rc rd r r r Z__get_cache_key s z$ModelFileSystemCache.__get_cache_keyc C s | |}d}|d }tj| j|d }| jD ]B}|d |krZ|d |d s2|d |d rZ|t }|durZtj|rZt rGt
|}n|}||krQd} q[t d| d q|rjtj|redS |
| dS )a Check the file is cached or not. Note existence check will also cover digest check
Args:
model_file_info (CachedFileInfo): The cached file info
Returns:
bool: If exists and has the same hash, return True otherwise False
FrZ rT NTzFile [z?] exists in cache but with a mismatched hash, will re-download.)rb r r r r
r r_ r r enable_default_hash_validationr loggerinfor4 ) r rc r/ Z is_existsr[ Zcache_file_pathZ
cached_keyZ
expected_hashZcache_file_sha256r r r r sB
zModelFileSystemCache.existsc C sZ | j D ]'}|d |d kr*| | tj| j|d }tj|r't| dS qdS )zWe in cache, remove it.
Args:
model_file_info (ModelFileInfo): The model file information from server.
rZ N)r r4 r r r r
r r3 )r rc r\ r[ r r r remove_if_exists- s
z%ModelFileSystemCache.remove_if_existsc C sp | | | |}tj| j|d }tj|}tj|s'tj|dd t || | j
| | |S )a[ Put model on model_file_location to cache, the model first download to /tmp, and move to cache.
Args:
model_file_info (str): The file description returned by get_model_files.
model_file_location (str): The location of the temporary file.
Returns:
str: The location of the cached file.
rZ Tr N)
rj rb r r r r
dirnamer r r r appendr, )r rc Zmodel_file_locationrd Zcache_full_pathZcache_file_dirr r r r2 < s
z ModelFileSystemCache.put_file)NN)r; r< r= __doc__r rD rG r rX rY rF r^ ra rf rb r rj r2
__classcell__r r rJ r rB x s )rB )r6 r r" r' shutilr r typingr Zmodelscope.hub.constantsr r Zmodelscope.hub.utils.utilsr Zmodelscope.utils.loggerr rh getenvrP lowerrg objectr rB r r r r s a