a ߙfb>? @s&dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl ZddlZddlZddlZddlZddlmZmZmZmZddlmZz ddlZWneydZYn0ddlZddlmZddl m!Z!ddl"m#Z#m$Z$gdZ%iZ&Gd d d ej'Z(Gd d d ej)Z*Gd ddej+Z,e,Z-Gddde!Z.ddZ/e/Z0ddZ1ej2ddddZ3ddZ4ej2deddZ5dfddZ6dgdd Z7dhd"d#Z8did$d%Z9d&d'Z:dd(d)d*Z;djd,d-ZGd2d3d3ej?j@ZAGd4d5d5ej?jBZCeDdld6d7ZEdmd8d9ZFdnd:d;ZGdodd?ZIdqd@dAZJdBdCZKdrdEdFZLgaMejNdGdHZOdsdIdJZPdtdKdLZQdMdNZRGdOdPdPeSZTeTiZUGdQdRdReVZWdudSdTZXej2dvdUdVZYdwdWdXZZdxddYdZd[Z[dyd\d]Z\dzd^d_Z]d{d`daZ^d|dbdcZ_dS)}z=Functions for accessing, downloading, and caching data files.N)NamedTemporaryFile gettempdirTemporaryDirectorymkdtemp)warn)config)AstropyWarning)find_current_module resolve_name)Confconf download_filedownload_files_in_parallelget_readable_fileobjget_pkg_data_fileobjget_pkg_data_filenameget_pkg_data_contentsget_pkg_data_fileobjsget_pkg_data_filenamesget_pkg_data_pathis_urlis_url_in_cacheget_cached_urlscache_total_sizecache_contentsexport_download_cacheimport_download_cacheimport_file_to_cachecheck_download_cacheclear_download_cache compute_hashget_free_space_in_dircheck_free_space_in_dirget_file_contentsCacheMissingWarning CacheDamagedc@seZdZddZdS)_NonClosingBufferedReadercCs$z |WntyYn0dSNdetach Exceptionselfr-1lib/python3.9/site-packages/astropy/utils/data.py__del__Bs  z!_NonClosingBufferedReader.__del__N__name__ __module__ __qualname__r/r-r-r-r.r&Asr&c@seZdZddZdS)_NonClosingTextIOWrappercCs$z |WntyYn0dSr'r(r+r-r-r.r/Ls  z _NonClosingTextIOWrapper.__del__Nr0r-r-r-r.r4Ksr4c@sveZdZdZeddZeddZeddZejdd d gd Z ed d Z eddZ eddZ ed dZ dS)r z< Configuration parameters for `astropy.utils.data`. zhttp://data.astropy.org/z)Primary URL for astropy remote data site.z$http://www.astropy.org/astropy-data/z(Mirror URL for astropy remote data site.astropyzDefault User-Agent for HTTP request headers. This can be overwritten for a particular call via http_headers option, where available. This only provides the default value when not set by https_headers.g$@z2Time to wait for remote data queries (in seconds).z5astropy.coordinates.name_resolve.name_resolve_timeout)aliasesTz9If False, prevents any attempt to download from Internet.iz%Block size for computing file hashes.z4Number of bytes of remote data to download per step.zzIf True, temporary download files created when the cache is inaccessible will be deleted at the end of the python session.N)r1r2r3__doc___configZ ConfigItemdataurldataurl_mirrordefault_http_user_agentremote_timeoutallow_internetcompute_hash_block_sizedownload_block_size"delete_temporary_downloads_at_exitr-r-r-r.r UsDr c@seZdZdZdS)r$a1 This warning indicates the standard cache directory is not accessible, with the first argument providing the warning message. If args[1] is present, it is a filename indicating the path to a temporary file that was created to store a remote data download in the absence of the cache. N)r1r2r3r7r-r-r-r.r${sr$cCstj|}|jdvS)z Test whether a string is a valid URL for :func:`download_file`. Parameters ---------- string : str The string to test. Returns ------- status : bool String is URL or not. )httphttpsftpZsftpZsshfile)urllibparseurlparseschemelower)stringurlr-r-r.rs rcCs4tj|tj|p2tj|tj|Sr')ospathabspath startswithrealpath)rM parent_pathr-r-r. _is_insidesrRFTc csg}g}|durtj}t|tjr,t|}t|trt|} | rVt||||||d}t |d} | rt|st| | | | n|} t | dszt | } Wn tyt | } Yn0| d} | d| dddkrVddl} z$ddl} | j| d d }|d Wn.tt| jfyB| d|Yn0|d|} n| ddd krJz ddl}Wn0ty|D]}|qtd Yn0zbtddd6}|| ||j|jd d}Wdn1s0Y|d Wn&ty.| d|Yn0|d| ||} n| dddkrz$ddl }|j!| d d}|d WnXty|D]}|qtdYn8ttfy| d|Yn0|d|} |dk}|rz ddl}WntyYnX0t| |jrntddd}| }|||| |t |jd} | | t"| } t#| |d} | dz6| VW|D]}|q|D]}t$|jqn.|D]}|q|D]}t$|jq0dS)a Yield a readable, seekable file-like object from a file or URL. This supports passing filenames, URLs, and readable file-like objects, any of which can be compressed in gzip, bzip2 or lzma (xz) if the appropriate compression libraries are provided by the Python installation. Notes ----- This function is a context manager, and should be used for example as:: with get_readable_fileobj('file.dat') as f: contents = f.read() If a URL is provided and the cache is in use, the provided URL will be the name used in the cache. The contents may already be stored in the cache under this URL provided, they may be downloaded from this URL, or they may be downloaded from one of the locations listed in ``sources``. See `~download_file` for details. Parameters ---------- name_or_obj : str or file-like The filename of the file to access (if given as a string), or the file-like object to access. If a file-like object, it must be opened in binary mode. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. cache : bool or "update", optional Whether to cache the contents of remote URLs. If "update", check the remote URL for a new version but store the result in the cache. show_progress : bool, optional Whether to display a progress bar if the file is downloaded from a remote server. Default is `True`. remote_timeout : float Timeout for remote requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`). sources : list of str, optional If provided, a list of URLs to try to obtain the file from. The result will be stored under the original URL. The original URL will *not* be tried unless it is in this list; this is to prevent long waits for a primary server that is known to be inaccessible at the moment. http_headers : dict or None HTTP request headers to pass into ``urlopen`` if needed. (These headers are ignored if the protocol for the ``name_or_obj``/``sources`` entry is not a remote HTTP URL.) In the default case (None), the headers are ``User-Agent: some_value`` and ``Accept: */*``, where ``some_value`` is set by ``astropy.utils.data.conf.default_http_user_agent``. Returns ------- file : readable file-like N)cache show_progresstimeoutsources http_headersrseekrsrb)fileobjmodesBZhz9This Python installation does not provide the bz2 module.wbF)delete)r^s7zz:This Python installation does not provide the lzma module.Zbinaryencoding)%r r< isinstancerLPathLikefspathstr_is_urlr ioFileIOappendhasattrBytesIOZ read_binaryAttributeErrorreadrYstructgzipZGzipFileOSErrorEOFErrorerrorclosebz2 ImportErrorModuleNotFoundErrorrwriteZBZ2FilenamelzmaZLZMAFiler&r4remove)Z name_or_objrcrSrTr<rVrWZ close_fdsZ delete_fdsrr]Z signaturerprqZ fileobj_newrvfdtmpr{Zneeds_textio_wrapperdatar-r-r.rsV               0              rcOs:t|i|}|WdS1s,0YdS)z Retrieves the contents of a filename or file-like object. See the `get_readable_fileobj` docstring for details on parameters. Returns ------- object The content of the file (as requested by ``encoding``). N)rro)argskwargsfr-r-r.r#s r#ccst||d}tj|r"tdntj|rbt||d}|VWdq1sV0Yn`ttj|||tj|tj |gd*}| d| d|VWdn1s0YdS)a Retrieves a data file from the standard locations for the package and provides the file as a file-like object that reads bytes. Parameters ---------- data_name : str Name/location of the desired data file. One of the following: * The name of a data file included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data/file.dat'`` to get the file in ``astropy/pkgname/data/file.dat``. Double-dots can be used to go up a level. In the same example, use ``'../data/file.dat'`` to get ``astropy/data/file.dat``. * If a matching local file does not exist, the Astropy data server will be queried for the file. * A hash like that produced by `compute_hash` can be requested, prefixed by 'hash/' e.g. 'hash/34c33b3eb0d56eb9462003af249eff28'. The hash will first be searched for locally, and if not found, the Astropy data server will be queried. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. cache : bool If True, the file will be downloaded and saved locally or the already-cached local copy will be accessed. If False, the file-like object will directly access the resource (e.g. if a remote URL is accessed, an object like that from `urllib.request.urlopen` is returned). Returns ------- fileobj : file-like An object with the contents of the data file available via ``read`` function. Can be used as part of a ``with`` statement, automatically closing itself after the ``with`` block. Raises ------ urllib.error.URLError If a remote file cannot be found. OSError If problems occur writing or reading a local file. Examples -------- This will retrieve a data file and its contents for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_fileobj >>> with get_pkg_data_fileobj('data/3d_cd.hdr', ... package='astropy.wcs.tests') as fobj: ... fcontents = fobj.read() ... This next example would download a data file from the astropy data server because the ``allsky/allsky_rosat.fits`` file is not present in the source distribution. It will also save the file locally so the next time it is accessed it won't need to be downloaded.:: >>> from astropy.utils.data import get_pkg_data_fileobj >>> with get_pkg_data_fileobj('allsky/allsky_rosat.fits', ... encoding='binary') as fobj: # doctest: +REMOTE_DATA +IGNORE_OUTPUT ... fcontents = fobj.read() ... Downloading http://data.astropy.org/allsky/allsky_rosat.fits [Done] This does the same thing but does *not* cache it locally:: >>> with get_pkg_data_fileobj('allsky/allsky_rosat.fits', ... encoding='binary', cache=False) as fobj: # doctest: +REMOTE_DATA +IGNORE_OUTPUT ... fcontents = fobj.read() ... Downloading http://data.astropy.org/allsky/allsky_rosat.fits [Done] See Also -------- get_pkg_data_contents : returns the contents of a file or url as a bytes object get_pkg_data_filename : returns a local name for a file containing the data packageDTried to access a data file that's actually a package data directoryrbN)rcrSrVr_r) rrLrMisdirrrisfilerr r9r:rorY) data_namerrcrSdatafnr]r-r-r.rs$g    &  rcCs|durtj}|dr^t|dd}|durXttj|d||tj|tj|gdS|Snftj |}t ||d}tj |rt dn8tj |r|Sttj|d||tj|tj|gdSdS)a Retrieves a data file from the standard locations for the package and provides a local filename for the data. This function is similar to `get_pkg_data_fileobj` but returns the file *name* instead of a readable file-like object. This means that this function must always cache remote files locally, unlike `get_pkg_data_fileobj`. Parameters ---------- data_name : str Name/location of the desired data file. One of the following: * The name of a data file included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data/file.dat'`` to get the file in ``astropy/pkgname/data/file.dat``. Double-dots can be used to go up a level. In the same example, use ``'../data/file.dat'`` to get ``astropy/data/file.dat``. * If a matching local file does not exist, the Astropy data server will be queried for the file. * A hash like that produced by `compute_hash` can be requested, prefixed by 'hash/' e.g. 'hash/34c33b3eb0d56eb9462003af249eff28'. The hash will first be searched for locally, and if not found, the Astropy data server will be queried. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. show_progress : bool, optional Whether to display a progress bar if the file is downloaded from a remote server. Default is `True`. remote_timeout : float Timeout for the requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`). Raises ------ urllib.error.URLError If a remote file cannot be found. OSError If problems occur writing or reading a local file. Returns ------- filename : str A file path on the local file system corresponding to the data requested in ``data_name``. Examples -------- This will retrieve the contents of the data file for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_filename >>> fn = get_pkg_data_filename('data/3d_cd.hdr', ... package='astropy.wcs.tests') >>> with open(fn) as f: ... fcontents = f.read() ... This retrieves a data file by hash either locally or from the astropy data server:: >>> from astropy.utils.data import get_pkg_data_filename >>> fn = get_pkg_data_filename('hash/34c33b3eb0d56eb9462003af249eff28') # doctest: +SKIP >>> with open(fn) as f: ... fcontents = f.read() ... See Also -------- get_pkg_data_contents : returns the contents of a file or url as a bytes object get_pkg_data_fileobj : returns a file-like object with the data Nzhash/T)rSrTrUrVrr)r r<rO _find_hash_fnr r9r:rLrMnormpathrrrrr)rrrTr<ZhashfnZfs_pathrr-r-r.r"s4T        rcCs<t||||d}|}Wdn1s.0Y|S)a Retrieves a data file from the standard locations and returns its contents as a bytes object. Parameters ---------- data_name : str Name/location of the desired data file. One of the following: * The name of a data file included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data/file.dat'`` to get the file in ``astropy/pkgname/data/file.dat``. Double-dots can be used to go up a level. In the same example, use ``'../data/file.dat'`` to get ``astropy/data/file.dat``. * If a matching local file does not exist, the Astropy data server will be queried for the file. * A hash like that produced by `compute_hash` can be requested, prefixed by 'hash/' e.g. 'hash/34c33b3eb0d56eb9462003af249eff28'. The hash will first be searched for locally, and if not found, the Astropy data server will be queried. * A URL to some other file. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. cache : bool If True, the file will be downloaded and saved locally or the already-cached local copy will be accessed. If False, the file-like object will directly access the resource (e.g. if a remote URL is accessed, an object like that from `urllib.request.urlopen` is returned). Returns ------- contents : bytes The complete contents of the file as a bytes object. Raises ------ urllib.error.URLError If a remote file cannot be found. OSError If problems occur writing or reading a local file. See Also -------- get_pkg_data_fileobj : returns a file-like object with the data get_pkg_data_filename : returns a local name for a file containing the data )rrcrSN)rro)rrrcrSr}contentsr-r-r.rs F&r*ccsht||d}tj|r"tdnBtj|r\t|D] }t||r8tj||Vq8ntddS)a Returns the path of all of the data files in a given directory that match a given glob pattern. Parameters ---------- datadir : str Name/location of the desired data files. One of the following: * The name of a directory included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data'`` to get the files in ``astropy/pkgname/data``. * Remote URLs are not currently supported. package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. pattern : str, optional A UNIX-style filename glob pattern to match files. See the `glob` module in the standard library for more information. By default, matches all files. Returns ------- filenames : iterator of str Paths on the local filesystem in *datadir* matching *pattern*. Examples -------- This will retrieve the contents of the data file for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_filenames >>> for fn in get_pkg_data_filenames('data/maps', 'astropy.wcs.tests', ... '*.hdr'): ... with open(fn) as f: ... fcontents = f.read() ... rzDTried to access a data directory that's actually a package data filezPath not foundN) rrLrMrrrrlistdirfnmatchjoin)datadirrpatternrMfilenamer-r-r.rs,    rc csJt|||dD]6}t||d}|VWdq1s:0YqdS)a Returns readable file objects for all of the data files in a given directory that match a given glob pattern. Parameters ---------- datadir : str Name/location of the desired data files. One of the following: * The name of a directory included in the source distribution. The path is relative to the module calling this function. For example, if calling from ``astropy.pkname``, use ``'data'`` to get the files in ``astropy/pkgname/data`` * Remote URLs are not currently supported package : str, optional If specified, look for a file relative to the given package, rather than the default of looking relative to the calling module's package. pattern : str, optional A UNIX-style filename glob pattern to match files. See the `glob` module in the standard library for more information. By default, matches all files. encoding : str, optional When `None` (default), returns a file-like object with a ``read`` method that returns `str` (``unicode``) objects, using `locale.getpreferredencoding` as an encoding. This matches the default behavior of the built-in `open` when no ``mode`` argument is provided. When ``'binary'``, returns a file-like object where its ``read`` method returns `bytes` objects. When another string, it is the name of an encoding, and the file-like object's ``read`` method will return `str` (``unicode``) objects, decoded from binary using the given encoding. Returns ------- fileobjs : iterator of file object File objects for each of the files on the local filesystem in *datadir* matching *pattern*. Examples -------- This will retrieve the contents of the data file for the `astropy.wcs` tests:: >>> from astropy.utils.data import get_pkg_data_filenames >>> for fd in get_pkg_data_fileobjs('data/maps', 'astropy.wcs.tests', ... '*.hdr'): ... fcontents = fd.read() ... )rrrbN)rr)rrrrcfnr}r-r-r.rs : rcCsbt|d@}t}|tj}|r<|||tj}q Wdn1sP0Y|S)am Computes the MD5 hash for a file. The hash for a data file is used for looking up data files in a unique fashion. This is of particular use for tests; a test may require a particular version of a particular file, in which case it can be accessed via hash to get the appropriate version. Typically, if you wish to write a test that requires a particular data file, you will want to submit that file to the astropy data servers, and use e.g. ``get_pkg_data_filename('hash/34c33b3eb0d56eb9462003af249eff28')``, but with the hash for your file in place of the hash in the example. Parameters ---------- localfn : str The path to the file for which the hash should be generated. Returns ------- hash : str The hex digest of the cryptographic hash for the contents of the ``localfn`` file. r\N)openhashlibmd5ror r>update hexdigest)Zlocalfnrhblockr-r-r.r [s   ,r rcGs|durhtdddgd}|dur,tjj|St|dr<|js`d|jvrX|jdd}qf|j}qp|j}nt|}tj |j }tjj|g|R}| dd}t|}tj |j }t ||st d |d |S) aGet path from source-included data directories. Parameters ---------- *path : str Name/location of the desired data file/directory. May be a tuple of strings for ``os.path`` joining. package : str or None, optional, keyword-only If specified, look for a file relative to the given package, rather than the calling module's package. Returns ------- path : str Name/location of the desired data file/directory. Raises ------ ImportError Given package or module is not importable. RuntimeError If the local data file is outside of the package's tree. Nr_zastropy.utils.data contextlib)Zfinddiff __package__.rz2attempted to get a local data file outside of the z tree.)r rLrMrrlrr1 rpartitionr dirname__file__ partitionrR RuntimeError)rrMmoduleZ module_path full_pathZ rootpkgnameZrootpkgZroot_dirr-r-r.r~s(    rr5cCs,t|dD]}t||kr|SqdS)zs Looks for a local file by hash - returns file name if found and a valid file, otherwise returns None. pkgnameN)rvaluesr )rrvr-r-r.rs  rcCsVtj|stdt|j}|rRddlm}|dur>|j }| ||j  |}|S)aB Given a path to a directory, returns the amount of free space on that filesystem. Parameters ---------- path : str The path to a directory. unit : bool or `~astropy.units.Unit` Return the amount of free space as Quantity in the given unit, if provided. Default is `False` for backward-compatibility. Returns ------- free_space : int or `~astropy.units.Quantity` The amount of free space on the partition that the directory is on. If ``unit=False``, it is returned as plain integer (in bytes). zECan only determine free space associated with directories, not files.r)unitsT) rLrMrrrshutilZ disk_usageZfreer5rZbyteZQuantityto)rMunitZ free_spaceur-r-r.r!s   r!cCsPt|t|ddd}||krLddlm}td|d||d||d d S) ax Determines if a given directory has enough space to hold a file of a given size. Parameters ---------- path : str The path to a directory. size : int or `~astropy.units.Quantity` A proposed filesize. If not a Quantity, assume it is in bytes. Raises ------ OSError There is not enough room on the filesystem. rF)rr)human_file_sizezNot enough free space in z to download a z file, only z leftN)r!getattrastropy.utils.consolerrr)rMsizeZspacerr-r-r.r"s  r"c@seZdZddZdS)_ftptlswrappercCs^d|_t|_|j|j|j|j|j|j |j |j d |j }|j|dS)Nr/)ZbusyftplibZFTP_TLSrCZconnecthostportrUZloginuserpasswdZprot_prdirscwd)r,Z_targetr-r-r.inits   z_ftptlswrapper.initN)r1r2r3rr-r-r-r.rsrc@seZdZddZdS)_FTPTLSHandlerc Cst||||||ddS)NF)Z persistent)r)r,rrrrrrUr-r-r. connect_ftpsz_FTPTLSHandler.connect_ftpN)r1r2r3rr-r-r-r.rsrcCsddl}|rtdd|Dni}i}d|vrX||d|dd|dddnd|vshd|vrptd d |vrtdurt|d <|jfi|}|rd |_|j |_ |r|j fi|t j j|d }|rt j t|}n t j |}|S) zT Helper for building a `urllib.request.build_opener` which handles TLS/SSL. rNcss|] }|VqdSr'r-).0itr-r-r. z#_build_urlopener..certfilekeyfilepassword)rrrz_passing 'keyfile' or 'password' in the ssl_context argument requires passing 'certfile' as wellZcafileF)context)ssldictrpop ValueErrorcertifiwhereZcreate_default_contextZcheck_hostnameZ CERT_NONEZ verify_modeZload_cert_chainrErequestZ HTTPSHandlerZ build_openerr)ftp_tls ssl_contextallow_insecurerZ cert_chainZ https_handler urlopenerr-r-r._build_urlopeners4    rc Csddl}t|r|ng}t||dd}tjj||d}z|j||dWStjj y} z| j } t | |j r| j dkrd|d } |s| d | 7} tj | n:| d 7} t | tt||d d}|j||dWYd} ~ SWYd} ~ n d} ~ 00dS) zDHelper for opening a URL while handling TLS/SSL verification issues.rNF)rrr)Zheaders)rUZCERTIFICATE_VERIFY_FAILEDz'Verification of TLS/SSL certificate at a failed: this can mean either the server is misconfigured or your local root CA certificates are out-of-date; in the latter case this can usually be addressed by installing the Python package "certifi" (see the documentation for astropy.utils.data.download_url)z or in both cases you can work around this by passing allow_insecure=True, but only if you understand the implications; the original error was: z%. Re-trying with allow_insecure=True.T)r frozensetitemsrrErZRequestrrtURLErrorreasonrdZSSLErrorrr) source_urlrUrWrrrrrZreqexcrmsgr-r-r. _try_url_openBs4   rc Csddlm} tjs.tjd|dtjd|dur:|}|durFi}|durtj|j dkrzt |||||||ddWStjjy} z$t | j  d rd }nWYd} ~ n d} ~ 00t|||||| d $} | } zt| d }Wntttfy d}Yn0|dur:tt||r:t|}t|||rTtjrTtj}nt}||krrd |}nd |d|}| |||dD}tdtddd}zd}| tj }|r&|!||t"|7}|#|| tj }|dur||krtjd|d|dq|durX||krXtjj$d|d|dddWnLt%ytj&'|j(rzt)|j(Wnt*yYn0Yn0Wdn1s0YWdn1s0YWdn1s0Y|j(S)Nr)ProgressBarOrSpinnerURL z5 was supposed to be downloaded but allow_internet is zP; if this is unexpected check the astropy.cfg file for the option allow_internetrCF)rTrU remote_urlrSrrWrzftp error: error_permT)rUrWrrrzContent-Lengthz Downloading z from )rDzastropy-download--)prefixrazFile was supposed to be z* bytes but server provides more, at least z bytes. Download failed.z bytes but we only got )Zcontent)+rrr r=rErtrrFrGrH_download_file_from_sourcergrrOrinfointKeyErrorr TypeErrorr"r_get_download_cache_locsysstdoutisattyriStringIOrrLgetpidror?rylenrZContentTooShortError BaseExceptionrMexistsrzr|rr)rrTrUrrSrrWrrrreZremoterrdldirZprogress_streamZdlmsgprZ bytes_readrr-r-r.rms             hrc  Cs|durtj}|dur|g}|dur0tjdd}d} |} |rz t|} Wn4ty|} zd}d| d} WYd} ~ n\d} ~ 00|dkrnHt|trtd |d n,tj | t | d } tj | rtj | Si}|D]}z&t|||||||||d }WqWqtjjy} zdt| d rht| jdrh| jjdkrh| jjd|| j_| jj| jjf| j_| ||<WYd} ~ qd} ~ 0tjy} z| ||<WYd} ~ qd} ~ 00q|std|dn8t|dkr||dntjd|||d|r^zt| |d|dk|dWSty\} zd| d|d} WYd} ~ n d} ~ 00| rrtt| |tjrt !|tj |S)aDownloads a URL and optionally caches the result. It returns the filename of a file containing the URL's contents. If ``cache=True`` and the file is present in the cache, just returns the filename; if the file had to be downloaded, add it to the cache. If ``cache="update"`` always download and add it to the cache. The cache is effectively a dictionary mapping URLs to files; by default the file contains the contents of the URL that is its key, but in practice these can be obtained from a mirror (using ``sources``) or imported from the local filesystem (using `~import_file_to_cache` or `~import_download_cache`). Regardless, each file is regarded as representing the contents of a particular URL, and this URL should be used to look them up or otherwise manipulate them. The files in the cache directory are named according to a cryptographic hash of their URLs (currently MD5, so hackers can cause collisions). The modification times on these files normally indicate when they were last downloaded from the Internet. Parameters ---------- remote_url : str The URL of the file to download cache : bool or "update", optional Whether to cache the contents of remote URLs. If "update", always download the remote URL in case there is a new version and store the result in the cache. show_progress : bool, optional Whether to display a progress bar during the download (default is `True`). Regardless of this setting, the progress bar is only displayed when outputting to a terminal. timeout : float, optional Timeout for remote requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`). sources : list of str, optional If provided, a list of URLs to try to obtain the file from. The result will be stored under the original URL. The original URL will *not* be tried unless it is in this list; this is to prevent long waits for a primary server that is known to be inaccessible at the moment. If an empty list is passed, then ``download_file`` will not attempt to connect to the Internet, that is, if the file is not in the cache a KeyError will be raised. pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. http_headers : dict or None HTTP request headers to pass into ``urlopen`` if needed. (These headers are ignored if the protocol for the ``name_or_obj``/``sources`` entry is not a remote HTTP URL.) In the default case (None), the headers are ``User-Agent: some_value`` and ``Accept: */*``, where ``some_value`` is set by ``astropy.utils.data.conf.default_http_user_agent``. ssl_context : dict, optional Keyword arguments to pass to `ssl.create_default_context` when downloading from HTTPS or TLS+FTP sources. This can be used provide alternative paths to root CA certificates. Additionally, if the key ``'certfile'`` and optionally ``'keyfile'`` and ``'password'`` are included, they are passed to `ssl.SSLContext.load_cert_chain`. This can be used for performing SSL/TLS client certificate authentication for servers that require it. allow_insecure : bool, optional Allow downloading files over a TLS/SSL connection even when the server certificate verification failed. When set to `True` the potentially insecure download is allowed to proceed, but an `~astropy.utils.exceptions.AstropyWarning` is issued. If you are frequently getting certificate verification warnings, consider installing or upgrading `certifi`_ package, which provides frequently updated certificates for common root CAs (i.e., a set similar to those used by web browsers). If installed, Astropy will use it automatically. .. _certifi: https://pypi.org/project/certifi/ Returns ------- local_path : str Returns the local path that the file was download to. Raises ------ urllib.error.URLError Whenever there's a problem getting the remote file. KeyError When a file was requested from the cache but is missing and no sources were provided to obtain it from the Internet. Notes ----- Because this function returns a filename, another process could run `clear_download_cache` before you actually open the file, leaving you with a filename that no longer points to a usable file. Nz*/*)z User-AgentZAcceptFz+Cache directory cannot be read or created (z,), providing data in temporary file instead.rz Cache value 'zS' was requested but 'update' is the only recognized string; otherwise use a booleanr)rUrTrSrrrWrrrerrnoz. requested URL: zNo sources listed and file ze not in cache! Please include primary URL in sources if you want it to be included as a valid source.r_rz+Unable to open any source! Exceptions were T)remove_originalreplacerz)Cache directory appears to be read-only (zF), unable to import downloaded file, providing data in temporary file z instead.)"r r<r;rrrrdrgrrLrMr_url_to_dirnamerrNrrErtrrlrrstrerrorrsocketrUrrrPermissionErrorrr$r@_tempfilestodelrk)rrSrTrUrVrrWrrZ missing_cacheurl_keyrrrerrorsrZf_namer-r-r.r si             r cCsBz t|}Wnty YdS0tj|t|d}tj|S)a.Check if a download for ``url_key`` is in the cache. The provided ``url_key`` will be the name used in the cache. The contents may have been downloaded from this URL or from a mirror or they may have been provided by the user. See `~download_file` for details. Parameters ---------- url_key : str The URL retrieved pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. Returns ------- in_cache : bool `True` if a download for ``url_key`` is in the cache, `False` if not or if the cache does not exist at all. See Also -------- cache_contents : obtain a dictionary listing everything in the cache Fr)rrrrLrMrrr)rrrrr-r-r.rs   rcsBd}t|d}t|D]$\}}|tfdd|D7}q|S)z9Return the total size in bytes of all files in the cache.rrc3s$|]}tjtj|VqdSr')rLrMgetsizer)rrzrootr-r.rrz#cache_total_size..)rrLwalksum)rrrrfilesr-rr.rs  rc Cstjj|d`tjj|d,tfi|WdWdS1s\0YWdn1sz0YdS)N temp_config temp_cache)r5rpathsset_temp_configrset_temp_cacher )rr-r-r._do_download_files_in_parallelsr rc sddlm}durtjdur&is8tdtd|rDtj}nt }t t |} |j t fdd| D|d|d } g} |D]} | | | | q| S) a Download multiple files in parallel from the given URLs. Blocks until all files have downloaded. The result is a list of local file paths corresponding to the given urls. The results will be stored in the cache under the values in ``urls`` even if they are obtained from some other location via ``sources``. See `~download_file` for details. Parameters ---------- urls : list of str The URLs to retrieve. cache : bool or "update", optional Whether to use the cache (default is `True`). If "update", always download the remote URLs to see if new data is available and store the result in cache. .. versionchanged:: 4.0 The default was changed to ``"update"`` and setting it to ``False`` will print a Warning and set it to ``"update"`` again, because the function will not work properly without cache. Using ``True`` will work as expected. .. versionchanged:: 3.0 The default was changed to ``True`` and setting it to ``False`` will print a Warning and set it to ``True`` again, because the function will not work properly without cache. show_progress : bool, optional Whether to display a progress bar during the download (default is `True`) timeout : float, optional Timeout for each individual requests in seconds (default is the configurable `astropy.utils.data.Conf.remote_timeout`). sources : dict, optional If provided, for each URL a list of URLs to try to obtain the file from. The result will be stored under the original URL. For any URL in this dictionary, the original URL will *not* be tried unless it is in this list; this is to prevent long waits for a primary server that is known to be inaccessible at the moment. multiprocessing_start_method : str, optional Useful primarily for testing; if in doubt leave it as the default. When using multiprocessing, certain anomalies occur when starting processes with the "spawn" method (the only option on Windows); other anomalies occur with the "fork" method (the default on Linux). pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. Returns ------- paths : list of str The local file paths corresponding to the downloaded URLs. Notes ----- If a URL is unreachable, the downloading will grind to a halt and the exception will propagate upward, but an unpredictable number of files will have been successfully downloaded and will remain in the cache. r_) ProgressBarNzDisabling the cache does not work because of multiprocessing, it will be set to ``"update"``. You may need to manually remove the cached files with clear_download_cache() afterwards.rc s<g|]4}t|d|dtjjjjtjjjjdqS)FN)rrSrTrUrVrrr)rgetr5rrrZ _temp_pathr)rrrSrrVrUr-r. 0s   z.download_files_in_parallel..T)rDZ multiprocessmultiprocessing_start_method)Zconsoler r r<rrrrrirmlistsetmapr rkindex) urlsrSrTrUrVrrr ZprogressZ combined_urlsZcombined_pathsrrKr-r r.rs6M   rcCstdur|ttdkr|t}tj|rLzt|WqztyHYqz0qtj|rzt |WqtyxYq0qdS)Nr) rrrrLrMrr|rrrrrmtree)rr-r-r. _deltempsHs     rc Csz t|}Wn`tyl}zHd}t|jdkr2dn dt|}tt||jj|WYd}~dSd}~00z|durt |nt |rt j |t|}t |nt j ||}t j ||}|drtd|d|t j |\}} |r| d vrt j ||}t j |r&t |n@t|d tjkrftd |rft|} | durft| Wnbty}zHd }t|jdkrdn dt|}tt||jj|WYd}~n d}~00dS) aClears the data file cache by deleting the local file(s). If a URL is provided, it will be the name used in the cache. The contents may have been downloaded from this URL or from a mirror or they may have been provided by the user. See `~download_file` for details. For the purposes of this function, a file can also be identified by a hash of its contents or by the filename under which the data is stored (as returned by `~download_file`, for example). Parameters ---------- hashorurl : str or None If None, the whole cache is cleared. Otherwise, specify a hash for the cached file that is supposed to be deleted, the full path to a file in the cache that should be deleted, or a URL that should be removed from the cache if present. pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. z4Not clearing data cache - cache inaccessible due to r_r: Nz..z2attempted to use clear_download_cache on the path z" outside the data cache directory )rrKz [0-9a-f]+z-Not clearing data from cache - problem arose )rrrrrrgrr$ __class__r1_rmtreerhrLrMrrrelpathrOrsplitrrrZ digest_sizerematchrr) Z hashorurlrrrrestrfilepathZrpdrrr-r-r.r`sL         rc Cszztjtjj|dd}tj|sZzt|Wqvt yVtj|sRYqv0ntj |svt d|d|WSt y}zHd}t |j dkrdn dt |}tt||jj|WYd }~n d }~00d S) aFinds the path to the cache directory and makes them if they don't exist. Parameters ---------- pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. Returns ------- datadir : str The path to the data cache directory. ZdownloadrKzData cache directory z is not a directoryz/Remote data cache could not be accessed due to r_rrN)rLrMrr5rrZ get_cache_dirrmakedirsrrrrrrgrr$rr1)rrrrrr-r-r.rs      rcCst|std|dttj|}|d|d<|ddvrd|drd|ddkrdd|d<tj|}t | d  S) NzMalformed URL: ''r_r)rArBrrrutf-8) rhrrrErFZurlsplitrIZ urlunsplitrrencoder)rKZurlobjZurl_cr-r-r.rs$ rc@seZdZddZdS) ReadOnlyDictcCs tddS)NzThis object is read-only.)r)r,keyvaluer-r-r. __setitem__szReadOnlyDict.__setitem__N)r1r2r3r(r-r-r-r.r%sr%cs(eZdZdZdddfdd ZZS)r%zRecord the URL or file that was a problem. Using clear_download_cache on the .bad_file or .bad_url attribute, whichever is not None, should resolve this particular problem. N)bad_urls bad_filescs:tj|i||dur|ng|_|dur0|ng|_dSr')super__init__r)r*)r,r)r*rrrr-r.r,szCacheDamaged.__init__)r1r2r3r7r, __classcell__r-r-r-r.r%sr%c Cs"t}t}t|d}t|}|D]}tjtj||j}|jdrz|t vrx| || d|jdq(| rt |D]6}|dvrqtj||}| || d|qtj|d}d} tj |s| || d |nht|d d } t| s4| || d | n6t| } |j| krj| || d | d|jtj tj|ds| || dur| d|jdn| d| d|jdq(| || d|dq(Wdn1s0Y|rtd||ddS)aIDo a consistency check on the cache. .. note:: Since v5.0, this function no longer returns anything. Because the cache is shared by all versions of ``astropy`` in all virtualenvs run by your user, possibly concurrently, it could accumulate problems. This could lead to hard-to-debug problems or wasted space. This function detects a number of incorrect conditions, including nonexistent files that are indexed, files that are indexed but in the wrong place, and, if you request it, files whose content does not match the hash that is indexed. This function also returns a list of non-indexed files. A few will be associated with the shelve object; their exact names depend on the backend used but will probably be based on ``urlmap``. The presence of other files probably indicates that something has gone wrong and inaccessible files have accumulated in the cache. These can be removed with :func:`clear_download_cache`, either passing the filename returned here, or with no arguments to empty the entire cache and return it to a reasonable, if empty, state. Parameters ---------- pkgname : str, optional The package name to use to locate the download cache, i.e., for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. Raises ------ `~astropy.utils.data.CacheDamaged` To indicate a problem with the cache contents; the exception contains a ``.bad_files`` attribute containing a set of filenames to allow the user to use :func:`clear_download_cache` to remove the offending items. OSError, RuntimeError To indicate some problem with the cache structure. This may need a full :func:`clear_download_cache` to resolve, or may indicate some kind of misconfiguration. rrmtree-z Cache entry z not scheduled for deletion)rKrzUnexpected file frKNzProblem with URL file fr#rbzMalformed URL: zURL hashes to z but is stored in rzHash z is missing contentsrz with hash zLeft-over non-directory z in cache )r*)rrrLscandirrMrNrrzrOraddis_dirrrr#rhrr%) rr*ZmessagesrrentryrZsfZurlfrKZhashnamer-r-r.rsN)               4rccsdt|||d}z,|VWzt|Wq`ty8Yq`0n$zt|Wnty\Yn00dS)avTemporary directory context manager This will not raise an exception if the temporary directory goes away before it's supposed to be deleted. Specifically, what is deleted will be the directory *name* produced; if no such directory exists, no exception will be raised. It would be safer to delete it only if it's really the same directory - checked by file descriptor - and if it's still called the same thing. But that opens a platform-specific can of worms. It would also be more robust to use ExitStack and TemporaryDirectory, which is more aggressive about removing readonly things. )suffixrdirN)rrrrr)r5rr6r r-r-r._SafeTemporaryDirectory=s  r7c Cstdtjtj|d}zt|tj|dWn:tyJYn*tyrt t d|d|Yn0|durzt||WnDt yYn4t y}z|j t jkrnWYd}~n d}~00Wdn1s0YdS)z.More-atomic rmtree. Ignores missing directory.r/rr6zto-zapzUnable to remove directory z6 because a file in it is in use and you are on WindowsN)rrLrMrrNrenamerFileNotFoundErrorrrr$FileExistsErrorrrr ENOTEMPTY)rMrr rr-r-r.rWs.     rrc Cs<t|d}t|}tj||}tj|d}td|d} tj| d} t|| ttj| dddd} | |Wd n1s0Y|rt || d nVzt | |WnDt yYn4t y} z| jtjkrnWYd } ~ n d } ~ 00Wd n1s0Y|r0t|tj|S) apImport the on-disk file specified by filename to the cache. The provided ``url_key`` will be the name used in the cache. The file should contain the contents of this URL, at least notionally (the URL may be temporarily or permanently unavailable). It is using ``url_key`` that users will request these contents from the cache. See :func:`download_file` for details. If ``url_key`` already exists in the cache, it will be updated to point to these imported contents, and its old contents will be deleted from the cache. Parameters ---------- url_key : str The key to index the file under. This should probably be the URL where the file was located, though if you obtained it from a mirror you should use the URL of the primary location. filename : str The file whose contents you want to import. remove_original : bool Whether to remove the original file (``filename``) once import is complete. pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. replace : boolean, optional Whether or not to replace an existing object in the cache, if one exists. If replacement is not requested but the object exists, silently pass. rrtemp_dirr8rKZwtr#rbNr=)rrrLrMrr7rcopyrryrr9r;rrrr<r|rN) rrrrrZ cache_dirZ cache_dirnameZ local_dirnameZlocal_filenamer>Z temp_filenamerrr-r-r.rrs,%  (  8 rcCstt|dS)a Get the list of URLs in the cache. Especially useful for looking up what files are stored in your cache when you don't have internet access. The listed URLs are the keys programs should use to access the file contents, but those contents may have actually been obtained from a mirror. See `~download_file` for details. Parameters ---------- pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. Returns ------- cached_urls : list List of cached URLs. See Also -------- cache_contents : obtain a dictionary listing everything in the cache r)sortedrkeysrr-r-r.rsrc Csi}zt|d}Wnty(tYS0t|X}|D]B}|jr:ttj||j ddd}tj tj||j d||<q:Wdn1s0Yt |S)aObtain a dict mapping cached URLs to filenames. This dictionary is a read-only snapshot of the state of the cache when this function was called. If other processes are actively working with the cache, it is possible for them to delete files that are listed in this dictionary. Use with some caution if you are working on a system that is busy with many running astropy processes, although the same issues apply to most functions in this module. rrKr#rbrN) rrr_NOTHINGrLr1r3r#rMrrzrNr%)rrXrrr4rKr-r-r.rs    >rcCs~|durt|}t||rdndF}|D]0}t|dg|d}tjj|dd}|||q*Wdn1sp0YdS)aExports the cache contents as a ZIP file. Parameters ---------- filename_or_obj : str or file-like Where to put the created ZIP file. Must be something the zipfile module can write to. urls : iterable of str or None The URLs to include in the exported cache. The default is all URLs currently in the cache. If a URL is included in this list but is not currently in the cache, a KeyError will be raised. To ensure that all are in the cache use `~download_file` or `~download_files_in_parallel`. overwrite : bool, optional If filename_or_obj is a filename that exists, it will only be overwritten if this is True. pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. See Also -------- import_download_cache : import the contents of such a ZIP file import_file_to_cache : import a single file directly NwxT)rSrVrr)Zsafe)rzipfileZipFiler rErFZquotery)filename_or_objrZ overwriterzrrZz_fnr-r-r.rsrc CsBt|d}t}t|D]\}}tj|j}|durL||vrLq$|s^t ||dr^q$t j |t |} ||b} t| d8} | tj} | r| | | tj} qWdn1s0YWdn1s0Yt|| d|dq$Wdn1s0YWdn1s40YdS)a6Imports the contents of a ZIP file into the cache. Each member of the ZIP file should be named by a quoted version of the URL whose contents it stores. These names are decoded with :func:`~urllib.parse.unquote`. Parameters ---------- filename_or_obj : str or file-like Where the stored ZIP file is. Must be something the :mod:`~zipfile` module can read from. urls : set of str or list of str or None The URLs to import from the ZIP file. The default is all URLs in the file. update_cache : bool, optional If True, any entry in the ZIP file will overwrite the value in the cache; if False, leave untouched any entry already in the cache. pkgname : `str`, optional The package name to use to locate the download cache. i.e. for ``pkgname='astropy'`` the default cache location is ``~/.astropy/cache``. See Also -------- export_download_cache : export the contents the cache to of such a ZIP file import_file_to_cache : import a single file directly rXNrr`T)rr)rErFr enumerateZinfolistrErFZunquoterrrLrMrrgrror r?ryr) rGrZ update_cacherrHr iZzfrKZ f_temp_nameZf_zipZf_temprr-r-r.rs"  Jr)NFTNNN)NNT)NTN)NNT)Nr)NrN)r5)F)FNF)NNFNF) TNNFr5NNNF)FTNNr5NNF)r5)r5)rTNNNr5)Nr5)r5)r5)NNN)N)Fr5)r5)r5)NFr5)NFr5)`r7atexitrrr functoolsrrLrirrrrZurllib.requestrEZ urllib.errorZ urllib.parserErZtempfilerrrrwarningsrrrwZastropy.config.pathsr5rr8Zastropy.utils.exceptionsrZastropy.utils.introspectionr r __all__Z_dataurls_to_aliasBufferedReaderr& TextIOWrapperr4ZConfigNamespacer r r$rrhrRcontextmanagerrr#rrrrrr rrr!r"rZ ftpwrapperrZ FTPHandlerr lru_cacherrrr rrr rrregisterrrrrrr%rBrr%rr7rrrrrrr-r-r-r.s        #   p { t L 9 @#> %  * + Z O #  ~  D #  V   C   &