a `i@sddlmZmZmZddlZddlmZmZm Z m Z ddlm Z ddl mZmZddlZddlmZddlmZzddlmZmZWneyd \ZZYn0z ddlZWneydZYn0d d lmZm Z m!Z!Gd d d eZ"Gddde"Z#Gddde"Z$Gddde$Z%Gddde%Z&Gddde&Z'Gddde$Z(GdddeZ)Gddde)Z*Gddde*Z+Gd d!d!e)Z,Gd"d#d#e*Z-Gd$d%d%e)Z.Gd&d'd'e)Z/Gd(d)d)e/Z0Gd*d+d+e/Z1Gd,d-d-e/Z2Gd.d/d/e/Z3Gd0d1d1e/Z4Gd2d3d3e,Z5Gd4d5d5e)Z6Gd6d7d7e)Z7Gd8d9d9e)Z8Gd:d;d;e)Z9Gdd?d?e)Z;Gd@dAdAeZdBdCe?@DddgZAdS)D)absolute_importdivisionprint_functionN)dshape isnumericRecordOption) coretypes)concatuniqueisnull)cuda)cuda_atomic_nanmincuda_atomic_nanmax)NN)Exprngjitnansum_missingc@s$eZdZdZddZeddZdS) Preprocessz#Base clase for preprocessing steps.cCs ||_dSNcolumnselfrr4lib/python3.9/site-packages/datashader/reductions.py__init__szPreprocess.__init__cCs|jfSrrrrrrinputsszPreprocess.inputsN)__name__ __module__ __qualname____doc__rpropertyrrrrrrsrc@seZdZdZddZdS)extractz=Extract a column from a dataframe as a numpy array of values.cCsvtrNt|tjrNddl}||jjjdkr2tj}nd}| ||jj |dSt|t j rf||jj S||jjSdS)NrfZfillna)cudf isinstance DataFramecupyrdtypekindnpnanZarray to_gpu_arrayxrZDatasetdatavalues)rdfr+nullvalrrrapply&s  z extract.applyN)r r!r"r#r6rrrrr%$sr%c@s4eZdZdZeddZddZddZdd Zd S) CategoryPreprocessz*Base class for categorizing preprocessors.cCs|jSz"Returns name of categorized columnrrrrr cat_column6szCategoryPreprocess.cat_columncCs tddS)z7Returns list of categories corresponding to input shapezcategories not implementedNNotImplementedErrorr input_dshaperrr categories;szCategoryPreprocess.categoriescCs tddS)zValidates input shapezvalidate not implementedNr:r in_dshaperrrvalidate?szCategoryPreprocess.validatecCs tddS)z3Applies preprocessor to DataFrame and returns arrayzapply not implementedNr:rr4rrrr6CszCategoryPreprocess.applyN) r r!r"r#r$r9r>rAr6rrrrr74s  r7c@s(eZdZdZddZddZddZdS) category_codesz:Extract just the category codes from a categorical column.cCs|j|jjSr)measurerr>r<rrrr>Iszcategory_codes.categoriescCs4|j|jvrtdt|j|jtjs0tddS)Nspecified column not foundzinput must be categorical)rdict ValueErrorr)rDctZ Categoricalr?rrrrALs zcategory_codes.validatecCs6tr"t|tjr"||jjjS||jjjjSdSr)r(r)r*rcatZcodesr0r3rBrrrr6Rszcategory_codes.applyN)r r!r"r#r>rAr6rrrrrCGsrCc sneZdZdZejejejejej ej ej ej ej h Zd fdd ZfddZddZd d Zd d ZZS)category_moduloz A variation on category_codes that assigns categories using an integer column, modulo a base. Category is computed as (column_value - offset)%modulo. rcst|||_||_dSr)superroffsetmodulo)rrrMrL __class__rrras zcategory_modulo.__init__cst|j|jfSr)rK_hashable_inputsrLrMrrNrrrPfsz category_modulo._hashable_inputscCstt|jSr)listrangerMr?rrrr>iszcategory_modulo.categoriescCs2|j|jvrtd|j|j|jvr.tddS)NrEzinput must be an integer column)rrFrGrD IntegerTypesr?rrrrAls zcategory_modulo.validatecCs8||j|j|j}tr.t|tjr.|S|jSdSr)rrLrMr(r)r*r0r3)rr4resultrrrr6rszcategory_modulo.apply)r)r r!r"r#rHbool_Zuint8Zuint16uint32Zuint64Zint8Zint16Zint32Zint64rSrrPr>rAr6 __classcell__rrrNrrJXs( rJcs>eZdZdZd fdd ZfddZddZd d ZZS) category_binninga A variation on category_codes that assigns categories by binning a continuous-valued column. The number of categories returned is always nbins+1. The last category (nbin) is for NaNs in the data column, as well as for values under/over the binned interval (when include_under or include_over is False). Parameters ---------- column: column to use lower: lower bound of first bin upper: upper bound of last bin nbins: number of bins include_under: if True, values below bin 0 are assigned to category 0 include_over: if True, values above the last bin (nbins-1) are assigned to category nbin-1 TcsTt||d||_||t||_||_|r8dn||_|rJ|dn||_dS)Nrr)rKrbin0floatbinsizenbins bin_underbin_over)rrlowerupperr\Z include_underZ include_overrNrrrs zcategory_binning.__init__cst|j|j|j|jfSr)rKrPrYr[r]r^rrNrrrPsz!category_binning._hashable_inputscCs|j|jvrtddS)NrE)rrFrGr?rrrrAs zcategory_binning.validatecCsrtrt|tjrtdnT||jj}||j|jt }|j ||dk<|j |||j k<|j |t |<|SdS)Nz'this feature is not implemented in cudar)r(r)r*r;rr3rYr[Zastypeintr]r^r\r.Zisnan)rr4valueindexrrrr6s  zcategory_binning.apply)TT) r r!r"r#rrPrAr6rWrrrNrrXys  rXcsPeZdZdZfddZeddZeddZdd Zd d Z d d Z Z S)category_valueszVExtract a category and a value column from a dataframe as (2,N) numpy array of values.cst|||_dSr)rKr categorizer)rreZ value_columnrNrrrs zcategory_values.__init__cCs|jj|jfSrrerrrrrrszcategory_values.inputscCs|jjSr8rfrrrrr9szcategory_values.cat_columncCs|jjSr)rer>r<rrrr>szcategory_values.categoriescCs |j|Sr)rerAr?rrrrAszcategory_values.validatecCs|j|}trvt|tjrvddl}||jjjdkr>t j }nd}| |}| ||jj |d}|j ||fddS||jj}t j ||fddSdS)Nrr&r'axis)rer6r(r)r*r+rr,r-r.r/Zasarrayr0stackr3)rr4ar+r5brrrr6s   zcategory_values.apply) r r!r"r#rr$rr9r>rAr6rWrrrNrrds   rdc@sleZdZdZdddZddZddZed d Zdd d Z dddZ ddZ dddZ ddZ ddZdS) Reductionz"Base class for per-bin reductions.NcCs ||_dSrrrrrrrszReduction.__init__cCs0|j|jvrtdt|j|js,tddS)NrEzinput must be numeric)rrFrGrrDr?rrrrAs zReduction.validatecCs|jSr)_dshaper?rrr out_dshapeszReduction.out_dshapecCs t|jfSrr%rrrrrrszReduction.inputsFcCs|fSrrrrrrr _build_basesszReduction._build_basescCsdSNrrrqrrr _build_tempsszReduction._build_tempscCs|jSr)_createrrrrr _build_createszReduction._build_createcCs6|r|jdur|jS|jSn|jdur,|jS|jSdSr)r_append_no_field_cuda _append_cuda_append_no_field_appendrrZschemarrrr _build_appends  zReduction._build_appendcCs|jSr)_combinervrrr_build_combineszReduction._build_combinecCs|jSr) _finalizervrrr_build_finalizeszReduction._build_finalize)N)F)F)F)r r!r"r#rrAror$rrrrtrwr}rrrrrrrms     rmc@s<eZdZdZd ddZeddZddZed d d Z dS)OptionalFieldReductionzOBase class for things like ``count`` or ``any`` for which the field is optionalNcCs ||_dSrrrrrrrszOptionalFieldReduction.__init__cCs|jdurt|jfSdSrs)rr%rrrrrszOptionalFieldReduction.inputscCsdSrrr?rrrrAszOptionalFieldReduction.validateFcKstj|dfi|SNrr1 DataArraybasesrkwargsrrrrsz OptionalFieldReduction._finalize)N)F) r r!r"r#rr$rrA staticmethodrrrrrrs  rc@seZdZdZeejZee ddZ ee ddZ ee j dddd Zee j ddd d Zed d ZeddZdS)countzCount elements in each bin, returning the result as a uint32. Parameters ---------- column : str, optional If provided, only counts elements in ``column`` that are not ``NaN``. Otherwise, counts every element. cCs|||fd7<dSNrrxyaggrrrrzszcount._append_no_fieldcCs t|s|||fd7<dSrr rrrfieldrrrr{sz count._appendT)ZdevicecCstj|||fddSr)nb_cudaatomicaddrrrrrx#szcount._append_no_field_cudacCs t|stj|||fddSrr rrrrrrrry(szcount._append_cudacCs|j|ddS)Nu4r,ZzerosshapeZ array_modulerrrru.sz count._createcCs|jdddS)Nrrrir,sumZaggsrrrr~2szcount._combineN)r r!r"r#rrHrVrnrrrzr{rZjitrxryrur~rrrrr s$        rc@seZdZdZefddZddZdddZed d Z ed d Z d dZ ddZ eddZ ddZdddZd ddZddZddZdS)!byayApply the provided reduction separately per category. Parameters ---------- cats: str or CategoryPreprocess instance Name of column to aggregate over, or a categorizer object that returns categories. Resulting aggregate has an outer dimension axis along the categories present. reduction : Reduction Per-category reduction function. cCs~t|tr||_nt|tr(t||_ntd|jj|_|jjt|ddf|_||_ |j durrt |j|j |_ n|j|_ dS)NzEfirst argument must be a column name or a CategoryPreprocess instancer) r)r7restrrC TypeErrorrgetattrcolumns reduction val_columnrd preprocess)rr9rrrrrAs     z by.__init__cCs tt|||j|jfSr)hashtyperPrerrrrr__hash__Rsz by.__hash__Fcstfddj|DS)Nc3s|]}tj|VqdSrrre).0tmprrr Vz"by._build_temps..)tuplerrtrqrrrrtUszby._build_tempscCs |jdSrrrrrrr9Xsz by.cat_columncCs |jdSrrrrrrr\sz by.val_columncCs|j||j|dSr)rrArr?rrrrA`s z by.validatecs2|j|}|j|ttfdd|DS)Ncsg|] }|fqSrr)rcZ red_shaperr grz!by.out_dshape..)rer>rrorr)rr=catsrrrrods  z by.out_dshapecCs|jfSr)rrrrrrisz by.inputscstjjfddS)Ncsj|f|Sr)rrwrZn_catsrorrros z"by._build_create..)lenrDfields)rrorrrrwms zby._build_createcs>j|}t|dkr(|dur(|Stfdd|DS)Nrrc3s|]}tj|VqdSrr)rbaserrrrvrz"by._build_bases..)rrrrr)rrrrrrrrrs zby._build_basescCs|j|||Sr)rr}r|rrrr}xszby._build_appendcCs|jjSr)rr~rvrrrr{szby._build_combinecs$tj|dfdd }|S)NFcs:|djg7<|dj<jj|fd|i|S)NZdimsZcoordsr)r9rrrrrrrfinalizesz$by._build_finalize..finalize)F)rQrer>)rrrrrrr~szby._build_finalizeN)F)F)F)r r!r"r#rrrrtr$r9rrArorrwrrr}rrrrrrr7s"       rc@sZeZdZdZeejZee ddZ e Z ee ddZ e Z eddZedd Zd S) anyzWhether any elements in ``column`` map to each bin. Parameters ---------- column : str, optional If provided, only elements in ``column`` that are ``NaN`` are skipped. cCsd|||f<dSNTrrrrrrzszany._append_no_fieldcCst|sd|||f<dSrr rrrrr{sz any._appendcCs|j|ddS)Nboolrrrrrrrusz any._createcCs|jdddS)Nrrrrrrrrr~sz any._combineN)r r!r"r#rrHrUrnrrrzrxr{ryrur~rrrrrs    rc@speZdZdZeeejZe dddZ e ddZ e ddZ e ed d Ze ed d Ze d dZdS) _upsamplez+"Special internal class used for upsamplingFcKstj|dfi|Srrrrrrrsz_upsample._finalizecCs t|jfSrrprrrrrsz_upsample.inputscCs|j|ddSNf8r)emptyrrrrrusz_upsample._createcCsdSrrrrrrr{sz_upsample._appendcCsdSrrrrrrrysz_upsample._append_cudacCstj|ddSNrrhr.Znanmaxrrrrr~sz_upsample._combineN)F)r r!r"r#rrrHfloat64rnrrr$rrurr{ryr~rrrrrs      rc@s8eZdZdZeeejZe ddZ e dddZ dS) FloatingReductionzBBase classes for reductions that always have floating-point dtype.cCs|j|tjddSr)fullr.r/rrrrruszFloatingReduction._createFcKstj|dfi|SrrrrrrrszFloatingReduction._finalizeN)F) r r!r"r#rrrHrrnrrurrrrrrs  rc@sHeZdZdZeddZeeddZeeddZedd Z d S) _sum_zerozSum of all elements in ``column``. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. cCs|j|ddSrrrrrrrusz_sum_zero._createcCs t|s|||f|7<dSrr rrrrr{sz_sum_zero._appendcCs t|stj|||f|dSrrrrrrrysz_sum_zero._append_cudacCs|jdddS)Nrrrrrrrrr~sz_sum_zero._combineN) r r!r"r#rrurr{ryr~rrrrrs    rc@sReZdZdZeeejZd ddZ e d ddZ e e ddZ e d d Zd S)raSum of all elements in ``column``. Elements of resulting aggregate are nan if they are not updated. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. FcCs"|rt|jt|jfS|fSdSr)rrrrqrrrrrszsum._build_basescKsH|r.|\}}t||tj}tj|fi|Stj|dfi|SdSr)r.wherer/r1r)rrrsumsZanysrrrrr s z sum._finalizecCs>t|s:t|||fr&||||f<n|||f|7<dSrr rrrrr{sz sum._appendcCs t|ddSr)rrrrrr~sz sum._combineN)F)F)r r!r"r#rrrHrrnrrrrrr{r~rrrrrs    rcsTeZdZdZeddZd ddZdfdd Zeed d Z ed d Z Z S)m2a`Sum of square differences from the mean of all elements in ``column``. Intermediate value for computing ``var`` and ``std``, not intended to be used on its own. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. cCs|j|dddS)Ngrr)rrrrrru0sz m2._createFcCst|jt|jfSrrrrrqrrrrt4szm2._build_tempscs |r tdtt||||S)NzIThe 'std' and 'var' reduction operations are not yet supported on the GPU)rGrKrr}r|rNrrr}7szm2._build_appendcCsXt|sT|dkrTt||}t|||d}|||f||||7<dS)Nrr)r r.r)rrrrrrZu1urrrr{=s z m2._appendcCsntjdddLtj|dd|jdd}tj|||||dddWdS1s`0YdS)NignoreZdivideZinvalidrrh)r.errstateZnansumr)ZMsrnsZmurrrr~Hsz m2._combine)F)F) r r!r"r#rrurtr}rr{r~rWrrrNrr#s    rc@s<eZdZdZeeddZeeddZeddZdS) minzMinimum value of all elements in ``column``. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. cCs>t|||fr||||f<n|||f|kr:||||f<dSrr rrrrr{Xsz min._appendcCst|||f|dSr)rrrrrry`szmin._append_cudacCstj|ddSr)r.Znanminrrrrr~esz min._combineN r r!r"r#rrr{ryr~rrrrrOs  rc@s<eZdZdZeeddZeeddZeddZdS) maxzMaximum value of all elements in ``column``. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. cCs>t|||fr||||f<n|||f|kr:||||f<dSrr rrrrr{ssz max._appendcCst|||f|dSr)rrrrrry{szmax._append_cudacCstj|ddSrrrrrrr~sz max._combineNrrrrrrjs  rcs eZdZdZfddZZS) count_cata]Count of all elements in ``column``, grouped by category. Alias for `by(...,count())`, for backwards compatibility. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be categorical. Resulting aggregate has a outer dimension axis along the categories present. cstt||tdSr)rKrrrrrNrrrszcount_cat.__init__)r r!r"r#rrWrrrNrrs rc@s6eZdZdZeeejZdddZ e d ddZ dS) meanzMean of all elements in ``column``. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. FcCst|jt|jfSrrrqrrrrrszmean._build_basescKs`|\}}tjddd(t|dk||tj}Wdn1sD0Ytj|fi|SNrrrr.rrr/r1r)rrrrcountsrrrrrs6zmean._finalizeN)F)F r r!r"r#rrrHrrnrrrrrrrrrs  rc@s6eZdZdZeeejZdddZ e d ddZ dS) varzVariance of all elements in ``column``. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. FcCst|jt|jt|jfSrrrrrrqrrrrrszvar._build_basescKsb|\}}}tjddd(t|dk||tj}Wdn1sF0Ytj|fi|SrrrrrrrZm2srrrrrs 6z var._finalizeN)F)Frrrrrrs  rc@s6eZdZdZeeejZdddZ e d ddZ dS) stdzStandard Deviation of all elements in ``column``. Parameters ---------- column : str Name of the column to aggregate over. Column data type must be numeric. ``NaN`` values in the column are skipped. FcCst|jt|jt|jfSrrrqrrrrrszstd._build_basescKsh|\}}}tjddd.t|dkt||tj}Wdn1sL0Ytj|fi|Sr)r.rrZsqrtr/r1rrrrrrs <z std._finalizeN)F)Frrrrrrs  rc@sNeZdZdZeeejZe ddZ e ddZ e ddZ e dd Z d S) firstaFirst value encountered in ``column``. Useful for categorical data where an actual value must always be returned, not an average or other numerical calculation. Currently only supported for rasters, externally to this class. Parameters ---------- column : str Name of the column to aggregate over. If the data type is floating point, ``NaN`` values in the column are skipped. cCs tddSNz/first is currently implemented only for rastersr:rrrrr{sz first._appendcCs tddSrr:rrrrrusz first._createcCs tddSrr:rrrrr~szfirst._combinecKs tddSrr:rrrrrrszfirst._finalizeNr r!r"r#rrrHrrnrr{rur~rrrrrrs    rc@sNeZdZdZeeejZe ddZ e ddZ e ddZ e dd Z d S) lastaLast value encountered in ``column``. Useful for categorical data where an actual value must always be returned, not an average or other numerical calculation. Currently only supported for rasters, externally to this class. Parameters ---------- column : str Name of the column to aggregate over. If the data type is floating point, ``NaN`` values in the column are skipped. cCs tddSNz.last is currently implemented only for rastersr:rrrrr{sz last._appendcCs tddSrr:rrrrru sz last._createcCs tddSrr:rrrrr~sz last._combinecKs tddSrr:rrrrrszlast._finalizeNrrrrrrs    rc@sNeZdZdZeeejZe ddZ e ddZ e ddZ e dd Z d S) modeatMode (most common value) of all the values encountered in ``column``. Useful for categorical data where an actual value must always be returned, not an average or other numerical calculation. Currently only supported for rasters, externally to this class. Implementing it for other glyph types would be difficult due to potentially unbounded data storage requirements to store indefinite point or line data per pixel. Parameters ---------- column : str Name of the column to aggregate over. If the data type is floating point, ``NaN`` values in the column are skipped. cCs tddSNz.mode is currently implemented only for rastersr:rrrrr{-sz mode._appendcCs tddSrr:rrrrru1sz mode._createcCs tddSrr:rrrrr~5sz mode._combinecKs tddSrr:rrrrr9szmode._finalizeNrrrrrrs   rc@s<eZdZdZddZddZddZdd Zed d Z d S) summaryavA collection of named reductions. Computes all aggregates simultaneously, output is stored as a ``xarray.Dataset``. Examples -------- A reduction for computing the mean of column "a", and the sum of column "b" for each bin, all in a single pass. >>> import datashader as ds >>> red = ds.summary(mean_a=ds.mean('a'), sum_b=ds.sum('b')) cKs$tt|\}}||_||_dSr)zipsorteditemskeysr3)rrksZvsrrrrMszsummary.__init__cCstt|t|jt|jfSr)rrrrr3rrrrrRszsummary.__hash__cCs|jD]}||qdSr)r3rA)rr=vrrrrAUs zsummary.validatecs$ttfddt|j|jDS)Ncsg|]\}}||fqSr)ro)rkrr@rrrZrz&summary.out_dshape..)rrrrr3r?rrrroYs zsummary.out_dshapecCstttdd|jDS)Ncss|] }|jVqdSr)r)rrrrrr_rz!summary.inputs..)rr r r3rrrrr]szsummary.inputsN) r r!r"r#rrrAror$rrrrrr?s rcCs@g|]8\}}t|trt|ts(|tur|ttttfvr|qSr)r)r issubclassrmrrrr)rZ_kZ_vrrrrcs r)BZ __future__rrrZnumpyr.Z datashaperrrrr rHZtoolzr r Zxarrayr1Zdatashader.glyphs.glyphr ZnumbarrZ)datashader.transfer_functions._cuda_utilsrr ImportErrorZcuda_atomic_nanmmaxr( ExceptionZutilsrrrrr%r7rCrJrXrdrmrrrrrrrrrrrrrrrrrrrrQsetlocalsr__all__rrrrs\        !-&.-Q # -,""%$