a 'À(bÝã@s4dZddlmZddlZGdd„dƒZddd„ZdS) z¢ Module for processing Sitemaps. Note: The main purpose of this module is to provide support for the SitemapSpider, its API is subject to change without notice. é)ÚurljoinNc@s eZdZdZdd„Zdd„ZdS)ÚSitemapzTClass to parse Sitemap (type=urlset) and Sitemap Index (type=sitemapindex) filescCsRtjjdddd}tjj||d|_|jj}d|vrH|jj dd¡dn||_dS)NTF)ZrecoverZremove_commentsZresolve_entities)ÚparserÚ}é)ÚlxmlZetreeZ XMLParserZ fromstringÚ_rootÚtagÚsplitÚtype)ÚselfZxmltextZxmlpZrt©r ú3lib/python3.9/site-packages/scrapy/utils/sitemap.pyÚ__init__szSitemap.__init__ccsš|j ¡D]Š}i}| ¡D]j}|j}d|vr<| dd¡dn|}|dkrld|jvr„| dg¡ | d¡¡q|jr||j  ¡nd||<qd|vr |Vq dS)NrrÚlinkZhrefZ alternateÚZloc) rZ getchildrenr r ZattribÚ setdefaultÚappendÚgetÚtextÚstrip)r ÚelemÚdZelr Únamer r rÚ__iter__s  zSitemap.__iter__N)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrr r r rr srccsD| ¡D]6}| ¡ ¡ d¡r| dd¡d ¡}t||ƒVqdS)zXReturn an iterator over all sitemap urls contained in the given robots.txt file zsitemap:ú:rN)Ú splitlinesÚlstripÚlowerÚ startswithr rr)Z robots_textZbase_urlÚlineZurlr r rÚsitemap_urls_from_robots(s r%)N)rZ urllib.parserZ lxml.etreerrr%r r r rÚs