You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem: The current object implementation bundling all information about the articles' body called ArticleBody seems overly complex and error-prone, while lacking some features.
Do we need a specific layout format (summary -> sub-headlines -> paragraphs) at all, since some regions like US don't use this kind of layout (see [Discussion]: Most articles only have one section #310).
This would also simplify node extraction.
Support access to the 'raw' text or even lxml node on a per-element base.
I.e. if we iterate over the elements of ArticleBody (alongside the nodes extracted from the HTML document) it should be possible to exact both the cleaned text and the 'raw' original text, also being able to access the lxml.html.HtmlElement would be beneficial but complicated for upcoming serialization approaches.
Write a spyder
Problem: Fundus heavily depends on the information given by publishers to crawl articles, in this case, a map linking articles.
While initially thought of as a feature to crawl publishers 'politely', adding optional functionality to process an entire domain in the form of a spyder and therefore not relying on sitemaps would be an excellent addition to Fundus.
Requirements: The spyder should ...
operate on the same level as HTMLSource yielding HTML for the scrapers to scrape.
This requires a rework of the naming and object hierarchy as there should be a proper interface now replacing HTMLSource which intern was used as an object and interface at the same time.
only operate in cases where no sitemap is provided and also not per default
support functionality to add delays between requests
Expensive: There could be a central request unit organizing request delays and access rules set by the publisher (see /robots.txt) so one would send a deferred request to this unit and expect Optional[HTML] in return.
This would cover all requests in all parts of Fundus.
Serialization
Should we simplify serialization at all, either through utility functions or proper serialization implementations?
Replace PublisherEnum's Enum implementation with a custom one
Python's implementation of the Enum class is a bit messy, especially typing-wise.
Since Fundus' PublisherEnum is solely built around the Enum class, this led to many typing issues but some unexpected behavior as well, and thus the PublisherEnum became notorious for causing problems.
The text was updated successfully, but these errors were encountered:
Rewrite ArticleBody
ArticleBody
seems overly complex and error-prone, while lacking some features.US
don't use this kind of layout (see [Discussion]: Most articles only have one section #310).This would also simplify node extraction.
I.e. if we iterate over the elements of
ArticleBody
(alongside the nodes extracted from the HTML document) it should be possible to exact both the cleaned text and the 'raw' original text, also being able to access thelxml.html.HtmlElement
would be beneficial but complicated for upcoming serialization approaches.Write a spyder
While initially thought of as a feature to crawl publishers 'politely', adding optional functionality to process an entire domain in the form of a spyder and therefore not relying on sitemaps would be an excellent addition to Fundus.
HTMLSource
yieldingHTML
for the scrapers to scrape.This requires a rework of the naming and object hierarchy as there should be a proper interface now replacing
HTMLSource
which intern was used as an object and interface at the same time.Optional[HTML]
in return.This would cover all requests in all parts of Fundus.
SerializationShould we simplify serialization at all, either through utility functions or proper serialization implementations?ReplacePublisherEnum
'sEnum
implementation with a custom onePython's implementation of theEnum
class is a bit messy, especially typing-wise.Since Fundus'
PublisherEnum
is solely built around theEnum
class, this led to many typing issues but some unexpected behavior as well, and thus thePublisherEnum
became notorious for causing problems.The text was updated successfully, but these errors were encountered: