Skip to content

Latest commit

 

History

History
131 lines (98 loc) · 4.06 KB

README.md

File metadata and controls

131 lines (98 loc) · 4.06 KB
Raw

Diffbot

This is a ruby client for the Diffbot API.

Install

Get the latest version from RubyGems:

$ gem install diffbot

Global Options

You can pass some settings to Diffbot like this:

Diffbot.configure do |config|
  config.token = ENV["DIFFBOT_TOKEN"]
  config.instrumentor = ActiveSupport::Notifications
end

The list of supported settings is:

  • token: Your Diffbot API token. This will be used for all requests in which you don't specify it manually (see below).
  • instrumentor: An object that matches the ActiveSupport::Notifications API, which will be used to trace network events. None is used by default.
  • article_defaults: Pass a block to this method to configure the global request settings used for Diffbot::Article requests. See below the options supported.

Articles

In order to fetch an article, do this:

require "diffbot"

article = Diffbot::Article.fetch(article_url, diffbot_token)

# Now you can inspect the result:
article.title
article.author
article.date
article.text
# etc. See below for the full list of available response attributes.

This is a list of all the fields returned by the Diffbot::Article.fetch call:

  • url: The URL of the article.
  • title: The title of the article.
  • author: The author of the article.
  • date: The date in which this article was published.
  • media: A list of media items attached to this article.
  • text: The body of the article. This will be plain text unless you specify the HTML option in the request.
  • tags: A list of tags/keywords extracted from the article.
  • xpath: The XPath at which this article was found in the page.

Options

You can customize your request like this:

article = Diffbot::Article.fetch(article_url, diffbot_token) do |request|
  request.html = true # Return HTML instead of plain text.
  request.dont_strip_ads = true # Leave any inline ads within the article.
  request.tags = true # Generate ads for the article.
  request.comments = true # Extract the comments from the article as well.
  request.summary = true # Return a summary text instead of the full text.
  request.stats = true # Return performance, probabilistic scoring stats.
end

Frontpages

In order to fetch and analyze a front page, do this:

require "diffbot"

frontpage = Diffbot::Frontpage.fetch(url, diffbot_token)

# Results are available in the returned object:
frontpage.title
frontpage.icon
frontpage.items #=> An array of Diffbot::Item instances

The fields you can extract from a Frontpage are:

  • title: The title of the page.
  • icon: The favicon of the page.
  • source_type: What kind of page this is.
  • source_url: The URL of the page.
  • items: The list of Diffbot::Item representing each item on the page.

The instances of Diffbot::Item have the following fields:

  • id: Unique identifier for this item.
  • title: Title of the item.
  • link: Extracted permalink of the item (if applicable).
  • description: innerHTML content of the item.
  • summary: A plain-text summary of the item.
  • pub_date: Date when item was detected on page.
  • type: The type of item, according to Diffbot. One of: IMAGE, LINK, STORY, CHUNK.
  • img: The main image extracted from this item.
  • xroot: XPath of where the item was found on the page.
  • cluster: XPath of the cluster of items where this item was found.
  • stats: An object with the following attributes:
    • spam_score: A Float between 0.0 and 1.0 indicating the probability this item is spam/an advertisement.
    • static_rank: A Float between 1.0 and 5.0 indicating the quality score of the item.
    • fresh: The percentage of the item that has changed compared to the previous crawl.

TODO

  • Implement the Follow API.
  • Add tests for Article and Frontpage requests.
  • Add a Frontpage.crawl method that given the URL of a frontpage, it will fetch the article for each item in the page.

License

This is published under an MIT License, see LICENSE for further details.