U eLO@sdZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl m Z ddlTz ddlZWn$ek red ed YnXed zRddlmmZdd lmZdd lmZddlmZddlmZddl m!Z!Wn.ek r edededYnXddddZ"ej#$dej%&dZ'd&e(e(e(e(ej)j*dddZ+eej,ddd Z-Gd!d"d"Z.d#d$Z/e0d%kre/dS)'zThe LangSAM model for segmenting objects from satellite images using text prompts. The source code is adapted from the https://github.com/luca-medeiros/lang-segment-anything repository. Credits to Luca Medeiros for the original implementation. N)Image)sam_model_registry) SamPredictor)hf_hub_download)*zInstalling rasterio...rasterioignore) build_model)box_ops)predict)SLConfig)clean_state_dictzInstalling GroundingDINO...zgroundingdino-pyz5Please restart the kernel and run the notebook again.zDhttps://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pthzDhttps://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pthzDhttps://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth)vit_hZvit_lZvit_bZ TORCH_HOMEz~/.cache/torch/hub/checkpointscpu)repo_idfilenameckpt_config_filenamedevicereturnc Csht|||d}t|}t|}||t|||d}tj|dd}|jt|ddd| |S)a Loads a model from HuggingFace Model Hub. Args: repo_id (str): Repository ID on HuggingFace Model Hub. filename (str): Name of the model file in the repository. ckpt_config_filename (str): Name of the config file for the model in the repository. device (str): Device to load the model onto. Default is 'cpu'. Returns: torch.nn.Module: The loaded model. )rrZforce_filenamer)Z map_locationmodelFstrict) rr fromfiler totorchloadload_state_dictreval) rrrrZcache_config_fileargsrZ cache_file checkpointr!R/home/ankuromar296_gmail_com/.local/lib/python3.8/site-packages/samgeo/text_sam.py load_model_hf4s"  r#)imagerc CsFttjdgddttdddgddd gg}||d \}}|S) z Transforms an image using standard transformations for image-based models. Args: image (Image): The PIL Image to be transformed. Returns: torch.Tensor: The transformed image as a tensor. i i5)max_sizeg ףp= ?gv/?gCl?gZd;O?gy&1?g?N)TZComposeZ RandomResizeZToTensorZ Normalize)r$ transformZimage_transformed_r!r!r"transform_imageUs r)c @seZdZdZd'ddZddZddZd d Zd d Zd dZ dde j iddfddZ de j iddfddZ d(ddZd)d d!Zd*d"d#Zd+d%d&ZdS),LangSAMzg A Language-based Segment-Anything Model (LangSAM) class which combines GroundingDINO and SAM. rcCsZttjrdnd|_|||d|_d|_d|_d|_ d|_ d|_ d|_ dS)zInitialize the LangSAM instance. Args: model_type (str, optional): The model type. It can be one of the following: vit_h, vit_l, vit_b. Defaults to 'vit_h'. See https://bit.ly/3VrpxUh for more details. cudarN) rrr+Z is_availablebuild_groundingdino build_samsourcer$masksboxesphraseslogits prediction)self model_typer!r!r"__init__ps zLangSAM.__init__cCsHt|}t|}tj|}|j|dd|j|jdt||_ dS)zBuild the SAM model. Args: model_type (str, optional): The model type. It can be one of the following: vit_h, vit_l, vit_b. Defaults to 'vit_h'. See https://bit.ly/3VrpxUh for more details. Tr)rN) SAM_MODELSrrZhubZload_state_dict_from_urlrrrrsam)r4r5Zcheckpoint_urlr8Z state_dictr!r!r"r-s   zLangSAM.build_samcCs"d}d}d}t||||j|_dS)zBuild the GroundingDINO model.zShilongLiu/GroundingDINOzgroundingdino_swinb_cogcoor.pthzGroundingDINO_SwinB.cfg.pyN)r#r groundingdino)r4Z ckpt_repo_idZ ckpt_filenamerr!r!r"r,szLangSAM.build_groundingdinoc CsVt|}t|j|||||jd\}}}|j\} } t|t| | | | g}|||fS)a Run the GroundingDINO model prediction. Args: image (Image): Input PIL Image. text_prompt (str): Text prompt for the model. box_threshold (float): Box threshold for the prediction. text_threshold (float): Text threshold for the prediction. Returns: tuple: Tuple containing boxes, logits, and phrases. )rr$caption box_thresholdtext_thresholdr) r)r r9rsizer Zbox_cxcywh_to_xyxyrTensor) r4r$ text_promptr;r<Z image_transr0r2r1WHr!r!r" predict_dinos  zLangSAM.predict_dinocCs\t|}|j||jj||jdd}|jjdd||jj dd\}}}| S)z Run the SAM model prediction. Args: image (Image): Input PIL Image. boxes (torch.Tensor): Tensor of bounding boxes. Returns: Masks tensor. NF)Z point_coordsZ point_labelsr0Zmultimask_output) npZasarrayr8 set_imager'Zapply_boxes_torchshapeZ predict_torchrrr)r4r$r0Z image_arrayZtransformed_boxesr/r(r!r!r" predict_sams     zLangSAM.predict_samcCsJt|tr@|drt|}tj|s8td|d||_nd|_dS)zoSet the input image. Args: image (str): The path to the image file or a HTTP URL. http Input path  does not exist.N) isinstancestr startswith download_fileospathexists ValueErrorr.)r4r$r!r!r"rEs   zLangSAM.set_imageNFc  Kst|tr|drt|}tj|s8td|d||_t |D} | d} | j |_ | j|_t| ddddddf}W5QRXn|}t|} ||_|||||\}}}tg}t|dkr|||}|d}|dkrtd dStj| d |d }tt||D]H\}\}}t|tjrP| !"|}||dk|d"|7}q$|dk|}|dk rt#|||jfd |i|||_$||_%||_&||_'||_(| r||||fS| r g}|j%D](}| !}|)|d|dfq|SdS) ap Run both GroundingDINO and SAM model prediction. Parameters: image (Image): Input PIL Image. text_prompt (str): Text prompt for the model. box_threshold (float): Box threshold for the prediction. text_threshold (float): Text threshold for the prediction. output (str, optional): Output path for the prediction. Defaults to None. mask_multiplier (int, optional): Mask multiplier for the prediction. Defaults to 255. dtype (np.dtype, optional): Data type for the prediction. Defaults to np.uint8. save_args (dict, optional): Save arguments for the prediction. Defaults to {}. return_results (bool, optional): Whether to return the results. Defaults to False. Returns: tuple: Tuple containing masks, boxes, phrases, and logits. rHrIrJrrCrNrrNo objects found in the image..rdtyperY)*rKrLrMrNrOrPrQrRr.ropenread transposer'crsr fromarrayrDarrayr$rBrZtensorlenrGZsqueezenelementprint zeros_like enumeratezipr>rnumpyastypearray_to_imager/r0r1r2r3append)r4r$r?r;r<outputmask_multiplierrY save_argsZreturn_resultsZ return_coordskwargssrcimage_np image_pilr0r2r1r/ mask_overlayiboxmaskZboxlistr!r!r"r sx               zLangSAM.predictTc  Ks0ddl} tj|st|t|trHt| tj|d}| t|tsZt dt |D]\} }tj tj |d}| rtdt| dttt|dt|d|d tj||d }|j||||f||||d | qb| r,tj|d }t||| r,td |ddS)a4 Run both GroundingDINO and SAM model prediction for a batch of images. Parameters: images (list): List of input PIL Images. out_dir (str): Output directory for the prediction. text_prompt (str): Text prompt for the model. box_threshold (float): Box threshold for the prediction. text_threshold (float): Text threshold for the prediction. mask_multiplier (int, optional): Mask multiplier for the prediction. Defaults to 255. dtype (np.dtype, optional): Data type for the prediction. Defaults to np.uint8. save_args (dict, optional): Save arguments for the prediction. Defaults to {}. merge (bool, optional): Whether to merge the predictions into a single GeoTIFF file. Defaults to True. rNz*.tifz6images must be a list or a directory to GeoTIFF files.zProcessing image rz of z: z...z _mask.tif)rjrkrYrlz merged.tifzSaved the merged prediction to .)globrOrPrQmakedirsrKrLlistjoinsortrRrdsplitextbasenamerbzfillr`r Z merge_rasters)r4Zimagesout_dirr?r;r<rkrYrlmergeverbosermrvrrr$r|rjr!r!r" predict_batchMsB    4   zLangSAM.predict_batch EPSG:4326cKsd|jdkrtddS|j}t|jf||d|}|dkrPt||j||St||j||dS)aSave the bounding boxes to a vector file. Args: output (str): The path to the output vector file. dst_crs (str, optional): The destination CRS. Defaults to "EPSG:4326". **kwargs: Additional arguments for boxes_to_vector(). NPlease run predict() first.)r0dst_crs)r0rbtolistZ rowcol_to_xyr.Zboxes_to_vectorr])r4rjrrmr0Zcoordsr!r!r" save_boxess  zLangSAM.save_boxes offviridis皙?rrc  KsZddl} ddlm} ddlm}| d|j}|dkrDtddSt|dkr\tddS| j |d| |j |r|j D]X}| }|j|d|df|d|d|d |d||d d }| |q~d | krd | d <d| krd| d<| j |||d|dk r| || || dk rV| rF| j| f| nt|j| |jdS)aShow the annotations (objects with random color) on the input image. Args: figsize (tuple, optional): The figure size. Defaults to (12, 10). axis (str, optional): Whether to show the axis. Defaults to "off". cmap (str, optional): The colormap for the annotations. Defaults to "viridis". alpha (float, optional): The alpha value for the annotations. Defaults to 0.4. add_boxes (bool, optional): Whether to show the bounding boxes. Defaults to True. box_color (str, optional): The color for the bounding boxes. Defaults to "r". box_linewidth (int, optional): The line width for the bounding boxes. Defaults to 1. title (str, optional): The title for the image. Defaults to None. output (str, optional): The path to the output image. Defaults to None. blend (bool, optional): Whether to show the input image. Defaults to True. kwargs (dict, optional): Additional arguments for matplotlib.pyplot.savefig(). rNr rrV)figsizerrCrUnone)Z linewidthZ edgecolorZ facecolorZdpidZ bbox_inchesZtight)cmapalpha)warningsZmatplotlib.pyplotZpyplotZmatplotlib.patchespatchesfilterwarningsr3rbr`ZfigureZimshowr$r0rrfZ RectangleZgcaZ add_patchtitleaxisZsavefigrhr.)r4rrrrZ add_boxesZ box_colorZ box_linewidthrrjblendrmrZpltrannsrsrectr!r!r" show_annssJ            zLangSAM.show_annscKst||fd|i|dS)agSave the result to a vector file. Args: image (str): The path to the image file. output (str): The path to the vector file. simplify_tolerance (float, optional): The maximum allowed geometry displacement. The higher this value, the smaller the number of vertices in the resulting geometry. simplify_toleranceN)raster_to_vector)r4r$rjrrmr!r!r"rs zLangSAM.raster_to_vector SATELLITEcKst|f||d|S)a<Show the interactive map. Args: basemap (str, optional): The basemap. It can be one of the following: SATELLITE, ROADMAP, TERRAIN, HYBRID. out_dir (str, optional): The path to the output directory. Defaults to None. Returns: leafmap.Map: The map object. )basemapr~)Z text_sam_gui)r4rr~rmr!r!r"show_maps zLangSAM.show_map)r)Nr) rrrrTrrNNT)N)rN)__name__ __module__ __qualname____doc__r6r-r,rBrGrErDuint8r rrrrrr!r!r!r"r*ksD   s B  P r*c Cstjdd}|jdddd|jdddd|jd d td d |jd d tdd |}t|j }| d}|j }|j }W5QRXt }t |ddddddf}|}|||j|j|j\} } } } | dkrtdn\tj|dtjd} tt| D]:}| |}| |}| |dk|d7} q | dkdtj} tjddd| jd| jdd| j ||d }|!| dW5QRXdS)Nr*) descriptionz--imageTzpath to the image)requiredhelpz--promptz text promptz--box_thresholdg?z box threshold)defaulttyperz--text_thresholdztext thresholdrTrUrrVrWrXrrSzmask.tifwZGTiff)ZdriverheightwidthcountrYr]r')"argparseArgumentParser add_argumentfloat parse_argsrrZr$r[r\r'r]r*rr^copyr promptr;r<rarbrDrcZint64ranger`rrfrgrrFrYwrite)parserrrnror'r]rrpZ image_np_copyr/r0r1r2rqrrrsrtdstr!r!r"main sv      r__main__)r)1rrOrrrfrDrZPILrZsegment_anythingrrZhuggingface_hubrcommonr ImportErrorrbZinstall_packagerZ!groundingdino.datasets.transformsZdatasetsZ transformsr&Zgroundingdino.modelsr Zgroundingdino.utilr Zgroundingdino.util.inferencer Zgroundingdino.util.slconfigr Zgroundingdino.util.utilsrr7environgetrP expanduserZ CACHE_PATHrLnnModuler#r>r)r*rrr!r!r!r"sf            !$@