o .ѹgM@sddlZddlmZddlmZmZmZddlm Z ddl Z ddl m Z ddl mZmZddlZddlZddlmZzddlZWneyPeded dwz ddlZdd lmZWneymed ed dwd Zz!ddlZddlmZddlmmZddlZddl Zddl!Z!WneyeddZYnwGdddeZ"ddZ# d3ddZ$d4ddZ% d5ddZ& d6ddZ' d6dd Z(d!d"Z)d7d#d$Z*d8d%d&Z+d8d'd(Z,Gd)d*d*ej-j.Z/d+d,Z0Gd-d.d.ej1Z2erGd/d0d0ej3Z4d1d2Z5dS dS)9N)UMAP)warncatch_warningsfilterwarnings) TypingError)spectral_layout)check_random_state check_array)KDTreeaThe umap.parametric_umap package requires Tensorflow > 2.0 to be installed. You can install Tensorflow at https://www.tensorflow.org/install or you can install the CPU version of Tensorflow using pip install umap-learn[parametric_umap] z/umap.parametric_umap requires Tensorflow >= 2.0)opszEThe umap.parametric_umap package requires Keras >= 3 to be installed.z#umap.parametric_umap requires KerasTz7Torch and ONNX required for exporting to those formats.Fc seZdZddddddddddddif fdd Zd fdd Z d fd d Zd!d d Zfd dZddZd!ddZ ddZ d"ddZ    d#ddZ ddZ ddZZS)$ParametricUMAPNF?rc stjd i|||_||_||_||_||_||_||_||_ d|_ | |_ | |_ | |_ d|_d|_| |_| |_d|_t|jtrItj|jd|_tjjddd|_|jdurt|jdjd|jkrvt d !|jdjd|jdSdS) af Parametric UMAP subclassing UMAP-learn, based on keras/tensorflow. There is also a non-parametric implementation contained within to compare with the base non-parametric implementation. Parameters ---------- batch_size : int, optional size of batch used for batch training, by default None dims : tuple, optional dimensionality of data, if not flat (e.g. (32x32x3 images for ConvNet), by default None encoder : keras.Sequential, optional The encoder Keras network decoder : keras.Sequential, optional the decoder Keras network parametric_reconstruction : bool, optional Whether the decoder is parametric or non-parametric, by default False parametric_reconstruction_loss_fcn : bool, optional What loss function to use for parametric reconstruction, by default keras.losses.BinaryCrossentropy parametric_reconstruction_loss_weight : float, optional How to weight the parametric reconstruction loss relative to umap loss, by default 1.0 autoencoder_loss : bool, optional [description], by default False reconstruction_validation : array, optional validation X data for reconstruction loss, by default None global_correlation_loss_weight : float, optional Whether to additionally train on correlation of global pairwise relationships (>0), by default 0 landmark_loss_fn : callable, optional The function to use for landmark loss, by default the euclidean distance landmark_loss_weight : float, optional How to weight the landmark loss relative to umap loss, by default 1.0 keras_fit_kwargs : dict, optional additional arguments for model.fit (like callbacks), by default {} NMbP?@Z clipvaluerzNDimensionality of embedder network output ({}) doesnot match n_components ({}))"super__init__dimsencoderdecoderparametric_reconstruction%parametric_reconstruction_loss_weight"parametric_reconstruction_loss_fcnautoencoder_loss batch_sizeloss_report_frequencyglobal_correlation_loss_weightlandmark_loss_fnlandmark_loss_weight prev_epoch_XZ window_valsreconstruction_validationkeras_fit_kwargsparametric_model isinstance random_stateintkerasutilsZset_random_seedn_training_epochs optimizersAdam optimizeroutputsshape n_components ValueErrorformat)selfrrrrrrrrr$r r!r"r%kwargs __class__rX/Users/admin/.pyenv/versions/3.10.0/lib/python3.10/site-packages/umap/parametric_umap.pyr0sD4  zParametricUMAP.__init__c|jdu|du@r+tttjtjgg|jdt||j}t||jf}|durFt |}t |}||krFt d|d|d|j dkr_|durSt d||_ t j|||dSt j|||dS aFit X into an embedded space. Optionally use a precomputed distance matrix, y for supervised dimension reduction, or landmarked positions. Parameters ---------- X : array, shape (n_samples, n_features) Contains a sample per row. If the method is 'exact', X may be a sparse matrix of type 'csr', 'csc' or 'coo'. Unlike UMAP, ParametricUMAP requires precomputed distances to be passed seperately. y : array, shape (n_samples) A target array for supervised dimension reduction. How this is handled is determined by parameters UMAP was instantiated with. The relevant attributes are ``target_metric`` and ``target_metric_kwds``. precomputed_distances : array, shape (n_samples, n_samples), optional A precomputed a square distance matrix. Unlike UMAP, ParametricUMAP still requires X to be passed seperately for training. landmark_positions : array, shape (n_samples, n_components), optional The desired position in low-dimensional space of each sample in X. Points that are not landmarks should have nan coordinates. NrzLength of x = z5, length of landmark_positions = z, while it must be equal. precomputedzTPrecomputed distances must be supplied if metric is precomputed.landmark_positions)r#npstackarraynanr1list transform concatenatelenr3metric_Xrfitr5XyZprecomputed_distancesr>Zlen_XZlen_landr7rr9rIs8 zParametricUMAP.fitcr:r;)r#r?r@rArBr1rCrDrErFr3rGrHr fit_transformrJr7rr9rMs8 zParametricUMAP.fit_transformcCs(|r|n|j}|jjt|||jdS)aTransform X into the existing embedded space and return that transformed output. Parameters ---------- X : array, shape (n_samples, n_features) New data to be transformed. batch_size : int, optional Batch size for inference, defaults to the self.batch_size used in training. Returns ------- X_new : array, shape (n_samples, n_components) Embedding of the new data in low-dimensional space. rverboseN)rrpredictr? asanyarrayrO)r5rKrrrr9rDszParametricUMAP.transformcs.|jr|jjt||j|jdSt|S)aTransform X in the existing embedded space back into the input data space and return that transformed output. Parameters ---------- X : array, shape (n_samples, n_components) New points to be inverse transformed. Returns ------- X_new : array, shape (n_samples, n_features) Generated data points new data in data space. rNN) rrrPr?rQrrOrinverse_transform)r5rKr7rr9rR2s   z ParametricUMAP.inverse_transformcCsF|j}t|j|j|j|j|j|j|j||j |j |j |j |j d |_dS)zDefine the model in keras) negative_sample_raterr!parametric_reconstruction_loss_fnrrr rr!r"r/N)r UMAPModelZ_a_brSrrrrr rr!r"r/r&)r5Zprlwrrr9 _define_modelFs  zParametricUMAP._define_modelc Cs:|jdkr|j}|jdurt|dg|_nt|jdkr,t|t|gt|j}|jrAt |dks=t |dkrAt d|durMt |tj dd}t||j|j|j|j|j|d \}|_}}} |_tt|tjd |_tt| tjd |_|jdurd} t|} t|j|j|j|j| |j| \|_|_| t!||j|j"} |jr|j#durt|jdkrt|j#t|j#gt|j|_#|j#t$|j#fd |j#if} nd} |jj%|f|j"|j&| | d |j'}t(|d s|j)|_*n|j)+D]}|j*||j)|7<q|jj,||j-d}|ifS)Nr<rrr zMData should be scaled to the range 0-1 for cross-entropy reconstruction loss.z allow-nan)dtypeZforce_all_finiter=rreconstruction)Zepochssteps_per_epochvalidation_data_history)rO).rGrHrr?r1rFreshaperCrmaxminrr float32construct_edge_datasetgraph_n_epochsrr Z edge_weightr rA expand_dimsastypeint64headtailr&prepare_networksrrr2rWr)rr$ zeros_likerIr,r%hasattrhistoryr]keysrPrO)r5rKrdinitr(r> edge_datasetZn_edgesrhriinit_embeddingn_datar[r\rmkey embeddingrrr9_fit_embed_dataYs  "        zParametricUMAP._fit_embed_datacCstdd|jDS)Ncss.|]\}}t||r|dvr||fVqdS))r/rrr&N) should_pickle).0kvrrr9 s z.ParametricUMAP.__getstate__..)dict__dict__itemsr5rrr9 __getstate__szParametricUMAP.__getstate__Tc Cs@|jdurtj|d}|j||rtd||jdur6tj|d}|j||r6td||jdurQtj|d}|j||rQtd|t Bt dtj|d}t |d }t ||t jWdn1sxwY|rtd |WddSWddS1swYdS) N encoder.keraszKeras encoder model saved to {} decoder.keraszKeras decoder model saved to {}zparametric_model.keraszKeras full model saved to {}ignore model.pklwbz*Pickle of ParametricUMAP model saved to {})rospathjoinsaveprintr4rr&rropenpickledumpHIGHEST_PROTOCOL)r5 save_locationrOencoder_outputdecoder_outputparametric_model_output model_outputoutputrrr9rs8       "zParametricUMAP.save{Gz?uniformcCs||_||_||_|jdkr.ttjjt|jdt |jd|dd|_ ||j |_ dS|jdkrF|dur;t d||_ ||j |_ dSt d)aAdd some points from a dataset X as "landmarks." Parameters ---------- X : array, shape (n_samples, n_features) Old data to be retained. sample_pct : float, optional Percentage of old data to use as landmarks. sample_mode : str, optional Method for sampling points. Allows "uniform" and "predefined." landmark_loss_weight : float, optional Multiplier for landmark loss function. rrF)replaceZ predeterminedNz'Choice of sample_mode is not supported.) sample_pct sample_moder"rCr?randomchoiceranger1r)Zprev_epoch_idxr#r3)r5rKrrr"idxrrr9 add_landmarkss(  zParametricUMAP.add_landmarkscCs d|_dSN)r#r~rrr9remove_landmarks! zParametricUMAP.remove_landmarkscCsD|j}t|jd|j}t||}td|jd}tj|||S)z(Exports trained parametric UMAP as ONNX.rrN) rPumapNetrr2 weight_copiertorchZrandnZonnxZexport)r5rkmpmZ dummy_inputrrr9to_ONNX$s  zParametricUMAP.to_ONNX)NNNrT)rrrN)__name__ __module__ __qualname__rrIrMrDrRrWrurrrrr __classcell__rrr7r9r /s>gA E  n % /r cCs|}||jd}|dur|jddkrd}nd}d|j|j|jt|k<|||j}|j}|j}|j}||||||fS)a< gets elements of graphs, weights, and number of epochs per edge Parameters ---------- graph_ : scipy.sparse.csr.csr_matrix umap graph of probabilities n_epochs : int maximum number of epochs per edge Returns ------- graph scipy.sparse.csr.csr_matrix umap graph epochs_per_sample np.array number of epochs to train each sample for head np.array edge head tail np.array edge tail weight np.array edge weight n_vertices int number of vertices in graph rNr'irX) ZtocooZsum_duplicatesr1datar_floatZeliminate_zerosrowcol)rcrdgraph n_verticesepochs_per_samplerhriweightrrr9get_graph_elements2s  rspectralcCsF|durtd}t|tr$|dkr$|jdd|jd|fdtj}|St|trZ|dkrZt||||||d}dt | } || tj|j d |jd|gd tj}|St |} t | jd krtj| dd jd| jdkrt| } | j| d d \} } t| dddf}| |j d|| jd tj}|S| }|S)a*Initialize embedding using graph. This is for direct embeddings. Parameters ---------- init : str, optional Type of initialization to use. Either random, or spectral, by default "spectral" Returns ------- embedding : np.array the initialized embedding Nrg$g$@r)lowhighsizer)rGZ metric_kwds-C6?)Zscaleraxis)rxrr)rr'strrr1rfr?rarabsr_normalrArFuniquer querymean)Z _raw_datarr2r(rGZ _metric_kwdsrortZinitialisationZ expansionZ init_datatreedistindZnndistrrr9init_embedding_from_graphhsX$  rr cCst||d| S)a convert distance representation into log probability, as a function of a, b params Parameters ---------- distances : array euclidean distance between two points in embedding a : float, optional parameter based on min_dist, by default 1.0 b : float, optional parameter based on min_dist, by default 1.0 Returns ------- float log probability in embedding space rN)r log1p)Z distancesabrrr9#convert_distance_to_log_probabilitysrrcCs>| t|}d| t|||}||}|||fS)a Compute cross entropy between low and high probability Parameters ---------- probabilities_graph : array high dimensional probabilities log_probabilities_distance : array low dimensional log probabilities EPS : float, optional offset to ensure log is taken of a positive number, by default 1e-4 repulsion_strength : float, optional strength of repulsion between negative samples, by default 1.0 Returns ------- attraction_term: float attraction term for cross entropy loss repellant_term: float repellent term for cross entropy loss cross_entropy: float cross entropy umap loss r N)r Z log_sigmoid)probabilities_graphlog_probabilities_distanceZEPSrepulsion_strengthZattraction_termZrepellant_termZCErrr9compute_cross_entropys  rc Cs|dur/ttjj|dtjtjjdddtjjdddtjjdddtjj|ddg}|durf|rfttjj|fdtjjdddtjjdddtjjdddtjjt|ddd tj|g}||fS) a` Generates a set of keras networks for the encoder and decoder if one has not already been predefined. Parameters ---------- encoder : keras.Sequential The encoder Keras network decoder : keras.Sequential the decoder Keras network n_components : int the dimensionality of the latent space dims : tuple of shape (dim1, dim2, dim3...) dimensionality of data n_data : number of elements in dataset # of elements in training dataset parametric_reconstruction : bool Whether the decoder is parametric or non-parametric init_embedding : array (optional, default None) The initial embedding, for nonparametric embeddings Returns ------- encoder: keras.Sequential encoder keras network decoder: keras.Sequential decoder keras network N)r1drelu)units activationz)rnameZrecon)rrr) r*Z SequentiallayersZInputFlattenZDenser?productZReshape)rrr2rrrrrqrrr9rjs0&     rjcsdddjddkr dnddurjddkrdndfdd }fd d }t||\} } } } } }durItt|d gt| | d t| | d }}tjt t |}||tj }||tj }t j j||f}|}|d}|jdd}|j|t j jjd}|j|t j jjd}|d}|t || | | fS)a Construct a tf.data.Dataset of edges, sampled by edge weight. Parameters ---------- X : array, shape (n_samples, n_features) New data to be transformed. graph_ : scipy.sparse.csr.csr_matrix Generated UMAP graph n_epochs : int # of epochs to train each edge batch_size : int batch size parametric_reconstruction : bool Whether the decoder is parametric or non-parametric landmark_positions : array, shape (n_samples, n_components), optional The desired position in low-dimensional space of each sample in X. Points that are not landmarks should have nan coordinates. cSs||Srr)Ztensorindexrrr9 gather_indexDsz,construct_edge_dataset..gather_indexg& .>g?TFNcs^rt|gtjgd}t|gtjgd}n t|}t|}||||fS)Nr)tf py_functionragather)edge_to edge_from edge_to_batchedge_from_batch)rKrgather_indices_in_pythonrr9gather_XOs   z(construct_edge_dataset..gather_Xcstdtdi}dkr||d<r||d<dur4r,t|gtjgd|d<nt||d<||f|fS)Numaprglobal_correlationrZ landmark_to)r repeatrrrar)rrrrr0)rr!gather_landmark_indices_in_pythonr r>rrr9 get_outputs[s  z+construct_edge_dataset..get_outputsir)r)Zdrop_remainder)Znum_parallel_callsr)nbytesrr)r?r`rrfrZ permutationrrFrgrrZDatasetZfrom_tensor_slicesshufflebatchmapZ experimentalZAUTOTUNEZprefetch)rKrcrdrrr r>rr_rrhrirrZ edges_to_expZedges_from_expZ shuffle_maskrpr)rKrrrrr r>rr9rb's<     rbcCsztt|d}tt|d}WdStjtjj t tjj tjj t ttfyC}ztd||WYd}~dSd}~wtyb}ztd|d|d|WYd}~dSd}~ww) a Checks if a dictionary item can be pickled Parameters ---------- key : try key for dictionary element val : None element of dictionary Returns ------- picklable: bool whether the dictionary item can be pickled base64zDid not pickle {}: {}NFzFailed at pickling :z due to T)codecsencoderdumpsdecodeloads PicklingErrorrerrorsZInvalidArgumentError TypeErrorZ InternalErrorZ NotFoundError OverflowErrorrAttributeErrorrr4r3)rsvalZpickledrerrr9rvs,  rvcCstj|d}tt|d}|rtd|tj|d}tj|r5t j ||_ |r5td|tj|d}tj|rPt j ||_ td|tj|d}tj|rkt j ||_td ||S) a Load a parametric UMAP model consisting of a umap-learn UMAP object and corresponding keras models. Parameters ---------- save_location : str the folder that the model was saved in verbose : bool, optional Whether to print the loading steps, by default True Returns ------- parametric_umap.ParametricUMAP Parametric UMAP objects rrbz-Pickle of ParametricUMAP model loaded from {}rz"Keras encoder model loaded from {}rz"Keras decoder model loaded from {}r&zKeras full model loaded from {}N)rrrrloadrrr4existsr*modelsZ load_modelrrr&)rrOrmodelrrrrrr9load_ParametricUMAPs$   rc Cst|}|tj|ddd}|dur#|}tj|t|d|d}ntj||jd}|tj|ddd}t|jdg}dg}tj|dd}tj|dd}t|}t|}t|d}t|d}t |||f} t |||f} t |||f} t |||f} t | t| t||j} t | |ddf} t| } t | t| dd||f} |stj | dd } | S) zAdapted from TF Probability.rTrkeepdimsNrYrZint32rr) r Zconvert_to_tensorrZconjrYrFr1cast transposer^matmulsqueeze) xrLrZ event_axisZ sample_axisZx_permedZy_permedZn_eventsZ n_samplesZ x_permed_flatZ y_permed_flatZcovrrr9 covariancesD     r cCs>|tj|ddd}|dur|tj|ddd}t|||dS)NrTrr rLr)r stdr r rrr9 correlation$src@seZdZddZdS) StopGradientcCs t|Sr)r stop_gradientr5r rrr9call,rzStopGradient.callN)rrrrrrrr9r+s rcCs tjttj||ddS)Nrr)r*Z activationsrr rnorm)rLy_predrrr9_default_landmark_loss0s rcsdeZdZ         dfdd Zdd Zdd d Zdd d ZddZddZddZ Z S)rUNFr rX umap_modelcstj|d||_||_||_| |_| |_||_||_||_ | |_ | |_ | |_ |p1t jjddd}|j|dt j|_t j|_|durQt jjdd|_n||_| dur]t|_ dS| |_ dS)N)rrrr)r/T)Z from_logits)rrrrrr rrS umap_loss_a umap_loss_brr!r"r*r-r.compilerrflattenrZ SeedGeneratorseed_generatorlossesZBinaryCrossentropyrTr)r5rrrSrrr/rTrrr rr!r"rr7rr9r7s2      zUMAPModel.__init__cCsZ|\}}||}||}||d}|jr+|jr||}n|t|}||d<|S)N) embedding_toembedding_fromrZ)rrrrr r)r5inputsZto_xZfrom_xrrrZembedding_to_reconrrr9ris   zUMAPModel.callcKsg}|jD]}|tj|tjdq||||jdkr+|| |||j r7|| ||d|vrD|| ||t |S)Nrrr)rappendr rr*backendZfloatx _umap_lossr _global_correlation_lossr_parametric_reconstruction_loss_landmark_losssum)r5r rLrZ sample_weightr6rlossrrr9 compute_loss|s   zUMAPModel.compute_losscCs|d}|d}tj||jdd}tj||jdd}t|d}tjjt||jd}tj dkr;t ||} n||} tj tj||ddtj|| ddgdd} t| |j|j} t|d} tj t| ft| |jfgdd} t| | |d\}}}t|S) Nrrrrseed tensorflowr)r)r rrSr1r*rrZarangerconfigr"rrrErrrrZoneszerosrr)r5rrrrZembedding_neg_toZ repeat_negZrepeat_neg_batch_dimZshuffled_indicesZembedding_neg_fromZdistance_embeddingrrrZattraction_lossZrepellant_lossZce_lossrrr9r#sB     zUMAPModel._umap_lossc Cs||d}||d}dd}||}||}t|dd}t|dd}tj|dd|dddd }tj|dd|dddd }|tjj|j|jd d }t t t |dt |dd }| |j S) NrrcSs|t|t|Sr)r rr)r rrr9z_scoresz3UMAPModel._global_correlation_loss..z_scoreirrrrr*g|=)r rL) rr Zcliprr*rrr1rr rrer ) r5rLrr Zz_xr/ZdxZdzZcorr_drrr9r$s"" z"UMAPModel._global_correlation_losscCs||d|d}||jS)NrZ)rTr)r5rLrr(rrr9r%s  z)UMAPModel._parametric_reconstruction_losscCsZ|d}tjt|t|d|dd}tjt|t||d}||||jS)Nrr)x1Zx2)r whereisnanrkr!r")r5rLrZy_toZclean_y_pred_toZ clean_y_torrr9r&s  zUMAPModel._landmark_loss) NNFr rXFNr r)NNNN)r ) rrrrrr)r#r$r%r&rrrr7r9rU6s"2  4rUcs$eZdZfddZddZZS)rcsJtt|t|d|_tdd|_tdd|_td||_dS)Nr) rrrnnZLineardense1dense2dense3dense4)r5ZindimZoutdimr7rr9rs zPumapNet.__init__cCsT||}t|}||}t|}||}t|}||}t|}|Sr)r4Frr5r6r7rrrr9forwards        zPumapNet.forward)rrrrr9rrrr7r9rs rcs|}tt|d}dd|DfddtdD}|}t|D]}|d|d|||d<t|d||||d<q*|D] }t ||||<qN| ||S) a4Copies weights from a parametric UMAP encoder to pytorch. Parameters ---------- km : encoder extracted from parametric UMAP. pm: a PumapNet object. Will be overwritten. Returns ------- pm : PumapNet Object. Net with copied weights. rcSsg|]}|qSrr)rwr rrr9 6sz!weight_copier..cs"g|] }d|ddqS)r.r)split)rwiall_keysrr9r:7s"rz.biasz.weightN) Z get_weightsr)rFZ state_dictrnrr?rrZ from_numpyZload_state_dict)rrZkweightsZn_layersZpm_namesZpyt_state_dictr=rsrr>r9r's     r)r)r r )rr rr)NF)6Znumpyr?rrwarningsrrrZnumbarrZ umap.spectralrZ sklearn.utilsrr rrZsklearn.neighborsr r,r ImportErrorr*r Ztorch_importedrZtorch.nnr3Ztorch.nn.functionalZ functionalr8Z torch.onnxZ torchvisionr rrrrrjrbrvrr rrZLayerrrZModelrUModulerrrrrr9s~             7 : 2 K k ( / ;K %!