Feed Control – FiveFilters.org https://www.fivefilters.org Web articles made accessible Mon, 10 May 2021 10:35:25 +0000 en-US hourly 1 https://wordpress.org/?v=5.7.1 https://www.fivefilters.org/wp-content/uploads/2020/04/cropped-site-logo-round-32x32.png Feed Control – FiveFilters.org https://www.fivefilters.org 32 32 Google News RSS feeds https://www.fivefilters.org/2021/google-news-rss-feeds/ Thu, 06 May 2021 22:21:19 +0000 https://www.fivefilters.org/?p=3885 It's possible to get a variety of RSS feeds from Google News, and these all come from Google itself, so they don't have to be generated with third-party tools like our Feed Creator.

The post Google News RSS feeds appeared first on FiveFilters.org.

]]>
It’s possible to get a variety of RSS feeds from Google News, and they all come from Google itself, so they don’t have to be generated with third-party tools like our Feed Creator.

In this post we’ll show you how to get RSS feeds for top stories, topics, search results, and site-specific feeds. Each section will show you how to use Google News to get the news items you want, and then how to get those same results as an RSS feed.

Top stories

Google’s top stories are at: https://news.google.com/topstories

When you load that page, Google will set a country for you (e.g. US or UK) and show you the top stories for the audience of that country. The URL will change to reflect the language and country.

News for a US audience: https://news.google.com/topstories?hl=en-US&gl=US&ceid=US:en

If you’d like the top stories for a different region or in a different language, find ‘Language & region’ in the left sidebar (towards the bottom) and click it. You’ll then be able to select a different language and region.

Google News language and region
Google News language & region selection

After making your selection, the news items will update and if you look at your browser’s address bar, you’ll notice the URL has changed to refelect your selection. Google uses language codes (lowercase letters) and country codes (uppercase letters) in its URLs:

RSS feeds for top stories

To get RSS feeds for the top stories you want, simply replace ‘topstories‘ in the URL with ‘rss‘:

You can make this URL replacement in your browser’s address bar to view the RSS source.

Google News website and RSS feed side by side.

Copy and paste the RSS feed URL into your favourite news reader to subscribe to it and receive updates.

Topics

Google also provides news for different topics:

You can select from a set of main topics using the links in the left sidebar, or use the topic search to find trending topics or search for topics.

Google News topics
Google News topic selection

RSS feeds for topics

To get RSS feeds for topics you want, replace ‘/topics‘ in the URL with ‘/rss/topics‘:

  • [RSS] Technology
    https://news.google.com/rss/topics/CAAqJggKIiBDQkFTRWdvSUwyMHZNRGRqTVhZU0FtVnVHZ0pIUWlnQVAB?hl=en-US&gl=US&ceid=US%3Aen
  • [RSS] Health
    https://news.google.com/rss/topics/CAAqIQgKIhtDQkFTRGdvSUwyMHZNR3QwTlRFU0FtVnVLQUFQAQ?hl=en-US&gl=US&ceid=US%3Aen
  • [RSS] Entertainment
    https://news.google.com/rss/topics/CAAqJggKIiBDQkFTRWdvSUwyMHZNREpxYW5RU0FtVnVHZ0pWVXlnQVAB?hl=en-US&gl=US&ceid=US%3Aen

You can change the language and region for topics too in the same way as we did for top stories (either using the Google News interface or by modifying the URL and changing the language and country codes).

Advanced search

Google News also lets you get news items based on your search criteria. Click the arrow at the end of the search field to open the advanced search form.

Google News advanced search
Google News advanced search

Examples:

RSS feeds for advanced search

To get RSS feeds for search results, replace ‘/search‘ in the URL with ‘/rss/search‘:

News from a particular site

What if we’re only interested in news from a particular site? If it’s indexed by Google News we can limit results to that site using the same advanced search fields covered in the previous section.

Not all blogs and news sites indexed by the main Google search engine are available in Google News.

Here’s how you’d get recent news items indexed by Google News from The Grayzone website:

Google News advanced search with Website field filled in.
Telling Google News to return news items posted on The Grayzone in the last week

This will result in the following search query: site:thegrayzone.com when:7d

The Grayzone has its own feed, so unless we want to narrow the search down further, we’d be better off using the official feed. And with Google’s hostile attitude to independent news sites we’d advise against relying on Google News feeds for non-corporate news sources whenever possible.

But there are sites indexed by Google News that don’t publish their own feeds. Using Google News’ feed output, you can get RSS feeds for these sites. Reuters is one example. You can get reuters.com news items published in the last hour using this Google News query: site:reuters.com when:1h

(For sites that don’t publish feeds and aren’t on Google News, our Feed Creator service might help.)

It’s also possible to include a path segment in the site: operator to limit results to a specific category. For example, Reuters publishes its technology news at https://www.reuters.com/technology/, so you can limit Google News results to news items from this category by using: site:reuters.com/technology

RSS feeds for specific sites

We’re still using the search endpoint here, so like before, replace ‘/search‘ in the URL with ‘/rss/search‘:

News from a number of specific sites

One useful feature of Google News is being able to combine searches. We can, for example, tell Google News to return news items from two or three specific sites in a single feed.

After you run a search, you can edit the query by hand in the search field to use additional operators.

Let’s say we want news items from two sites: The Grayzone and MintPress News published in the last 7 days, here’s the query to use:

site:thegrayzone.com OR site:mintpressnews.com when:7d

Want another site, let’s say Fairness and Accuracy in Reporting, add that with another ‘OR’:

site:thegrayzone.com OR site:mintpressnews.com OR site:fair.org when:7d
Google News search results for The Grayzone and MintPress News.

RSS feeds for specific sites

As before, we’re still using the search endpoint, so replace ‘/search‘ in the URL with ‘/rss/search‘:

So that’s how you get Google News RSS feeds. If you have any questions, feedback or suggestions, please let us know on our forum.

The post Google News RSS feeds appeared first on FiveFilters.org.

]]>
Using proxy servers for content retrieval https://www.fivefilters.org/2021/proxy-server-support/ Mon, 26 Apr 2021 13:35:40 +0000 https://www.fivefilters.org/?p=3822 We've added proxy support to Feed Control in our latest update. This post will explain what it does, why you might need it, and how you can enable it.

The post Using proxy servers for content retrieval appeared first on FiveFilters.org.

]]>
We’ve added proxy support to Feed Control in our latest update. This post will explain what it does, why you might need it, and how you can enable it.

If you’re a user of our self-hosted Full-Text RSS or Feed Creator software, we’ll be covering how you can enable proxy support in those applications in the next post.

What’s a proxy server?

Proxy servers are used to route HTTP requests (e.g. requests for web pages) through different servers.

When you use our hosted applications (Feed Control, Full-Text RSS or Feed Creator) to fetch content from webpages, those requests go out from our servers in Germany (that’s where we host most of our web services). So when fetching content from example.org, the site will see that someone from Germany is requesting a web page. But it’s also possible to route the same request through a proxy server in the US, or some other country.

Why does it matter where a request originates?

Most of the time, it makes zero difference. A request from Germany will be treated exactly the same as a request from the US. There are situations, however, where it does make a difference.

Geofencing

With the introduction of GDPR in Europe, some sites in the US catering to local communities have decided it’s not worth the hassle to comply with European privacy laws when most of their audience is outside of Europe. They set up geofencing on their sites to refuse access to visitors outside of the US. When you access a site like this from Europe, you’ll often see a message stating that they cannot serve European visitors.

But what happens when someone from the US tries to use our Feed Control, Feed Creator, or Full-Text RSS service with such a site? The request will go out from one of our servers in Germany and will be rejected when it reaches the geofenced site. Regardless of where you live, when you request content via our services, all requests currently look to the target site as if they originate from Germany, because that’s where our servers are based. So certain content accessible to our users in the US won’t be accessible when requested via our services.

Rate limiting

Additionally, there are also sites that will limit the number of requests a single visitor (determined by IP address) can make within a certain timeframe. Such rate limits are usually in place for good reasons. They can prevent malicious activity or excessive requests that can put too much strain on servers. But a sometimes unintended consequence of rate limiting is that requests that would normally be handled fine if made by users directly get rejected when they come from a limited pool of IP addresses belonging to a service acting on behalf of those same users. To the site receiving these requests, it can look like a handful of users making too many requests, rather than a 100 or so users making a reasonable number of requests. You might have experienced something similar if you’ve ever used a VPN service and found yourself unable to load certain sites because of “too many requests”.

How does a proxy server help?

To access sites that enforce geofencing (mostly in our experience US sites that refuse to serve European visitors), we can route requests through US proxy servers. Now the geofenced site sees a request from the US and no longer blocks it.

To handle the rate limiting issue above, a rotating proxy service can be used to distribute requests through a number of different servers, rather than one.

Proxy use in Feed Control

If you use our Feed Control application, we now let you enable proxy use for feeds you add to your account. When enabled, Feed Control will use a rotating proxy service to route requests through different US servers when fetching web content.

The feature is currently only available for two types of feeds in Feed Control:

  • Expanded feeds (when you enable ‘fetch full text’ to have article content retrieved from the source site)
  • Webpage to RSS feeds (feeds built with out Feed Creator application and then added to Feed Control)

In most cases, there will be no need to enable proxy use, so we suggest you try without it first and only enable if you have trouble. You can also contact us via the support link if you need assistance with a feed.

Enabling proxy use in Feed Control

It’s not yet possible to preview feed output with proxy use enabled without adding the feed to your account first (we’ll add support for that in a future update). So if you suspect the content you’re after is not being retrieved because of the issues listed above, you should add your feed in Feed Control’s management console and then enable proxy use.

To do that, follow the steps below:

  1. Log in to your Feed Control account
  2. From the left sidebar select Feeds
  3. Click Add Feed
  4. Paste the feed address into the URL field and click Add Feed
  5. In the Feed Details view that loads, click the Edit button
  6. In the Proxy field, select US Rotating
  7. Click Update Feed
  8. From the actions drop down, select Refresh feed
  9. Click the Feed items tab to see if new items appear (it might take a minute or so for the feed to refresh, so try refreshing the page if you don’t see anything immediately)

We currently limit the number of feeds on which you can enable proxy use based on your plan:

  • Standard – proxy use on up to 10 feeds
  • Plus – proxy use on up to 20 feeds
  • Business – proxy use on up to 50 feeds

If you need more than this, or if you have trouble with any feeds that you’d like us to take a look at, please contact us using the support link in Feed Control.

In the next post we’ll show you how to enable proxy use in our self-hosted software: Full-Text RSS and Feed Creator. We’ll show you how to configure our applications to use the Storm Proxies service, but any other proxy provider should work too.

The post Using proxy servers for content retrieval appeared first on FiveFilters.org.

]]>
Feed Control: JSON output on all paid plans and CDN fixes https://www.fivefilters.org/2021/feed-control-json-output-on-all-paid-plans-and-cdn-fixes/ Thu, 08 Apr 2021 22:07:21 +0000 https://www.fivefilters.org/?p=3813 JSON output is now supported on all paid plans and we've fixed a CDN issue affecting some feeds.

The post Feed Control: JSON output on all paid plans and CDN fixes appeared first on FiveFilters.org.

]]>
JSON output is now supported in Feed Control on all paid plans (previously it was only available on the Plus and Business plans). It can be enabled on the Generate RSS/JSON tab when editing a feed.

Feed Control’s Generate RSS/JSON tab when editing a feed

After it’s been enabled, you can access the JSON URL (hosted on a CDN) using the green JSON link on the same tab:

We’ve also fixed an issue that resulted in the generated RSS and JSON files on the CDN not always getting updated in a timely manner when new items were detected. This only affected a small number of feeds, but if you had feeds that were affected, this should be fixed now.

Let us know if you have any feedback or feature requests related to Feed Control. We have some new features in the works that we hope to be able to share soon.

The post Feed Control: JSON output on all paid plans and CDN fixes appeared first on FiveFilters.org.

]]>
Reuters RSS feeds dead? https://www.fivefilters.org/2021/reuters-rss-feeds/ Tue, 09 Mar 2021 12:50:55 +0000 https://www.fivefilters.org/?p=3616 Reuters officially stopped producing RSS feeds in June 2020, but using a workaround you can still subscribe to their news items.

The post Reuters RSS feeds dead? appeared first on FiveFilters.org.

]]>
Reuters officially stopped producing RSS feeds in June 2020, but there are workarounds available.

Our Feed Creator application can generate RSS feeds from almost any website, including Reuters. We often get requests from people who want to use Feed Creator to generate RSS feeds from the Reuters site, and as long as Reuters continues publishing news stories on its site, you can use it to do just that.

If you’re here for the RSS feeds, we’ve prepared a few for you:

These feeds are hosted by our Feed Control application and we’ll do our best to keep them working, even if there are changes on the Reuters side.

Update April 2021: Due to site changes at Reuters, some of the feeds above stopped working for about a week. We’ve updated them now so they should continue working. The Wire web page is no longer accessible, so that feed will receive no updates unless the situation changes.

Webpage to RSS feed with Feed Creator

If you’re wondering how we produced the Reuters feeds above, we’ve written how-to posts with step-by-step instructions. You can apply the same technique to generate RSS feeds from many other sites.

The post Reuters RSS feeds dead? appeared first on FiveFilters.org.

]]>
Feed Control, Full-Text RSS, Feed Creator: Which to choose? https://www.fivefilters.org/2020/feed-control-comparison/ Mon, 28 Dec 2020 09:31:00 +0000 https://www.fivefilters.org/2020/feed-control-comparison/ We now offer a few products for working with feeds. This post looks at the the differences between them to help you choose the right one.

The post Feed Control, Full-Text RSS, Feed Creator: Which to choose? appeared first on FiveFilters.org.

]]>
Since launching Feed Control, some of you have asked how it compares to Full-Text RSS and Feed Creator. This post will try to answer that.

TLDR: If you’re not a developer, and have no need to run our tools on your own server, choose Feed Control. If you’re a developer, read on.

Our feed products are used in combination with feed reading applications such as Feedly, Newsblur, Fraidycat, and many others, but also by developers who need custom integrations with their own applications and workflows, usually in relation to monitoring and extracting information from blogs and news publications.

We’ll be taking both types of use into account when comparing the products, but because the solutions differ more when evaluated from a developer perspective, we’ll try to focus more on that angle when comparing. If you’re not a developer, you can ignore the parts aimed at developers.

Full-Text RSS

Full-Text RSS is our feed expansion application. It takes a partial feed (e.g. a feed which only contains a short summary of each article) and converts it to a full-text feed by pulling in the full article content for each item.

If you enjoy reading full articles within your news reading application and not having to click into the site itself, Full-Text RSS can help.

If you’re a developer, Full-Text RSS can also be used to extract article content from individual articles. Instead of giving it a feed URL, give it an article URL and it will try to extract the article content and return it along with additional information that might be useful (e.g. language, author).

Full-Text RSS can be used as a hosted service run by us, or bought to run yourself on your own server. We also offer the service via RapidAPI for developers, which is a great way to get up and running integrating it into your own application.

Feed Creator

Feed Creator is our feed creation application. It has two main uses:

  1. Creating feeds from web pages which don’t offer their own
  2. Filtering or merging existing feeds

To create a feed from a web page, you give it the web page URL and some selectors for the content you want extracted. When filtering or merging existing feeds, you give it feed URLs and keywords or URL segments to use as filters.

Feed Creator can be used as a hosted service run by us, or bought to run yourself on your own server.

Feed Control

Feed Control builds on both of the above. You can use it to create full-text feeds and also generate feeds from a web page.

Full-Text RSS and Feed Creator are more bare-bones compared to Feed Control. They have been developed to be fast, efficient, and stateless, with a small server footprint. For developers, they are ideally used as microservices, and can easily be set up on a server and scaled.

While that approach has benefits, keeping those applications small and lean – which we’re committed to doing – does also limit us in what we can implement as part of each service. Feed Control is our effort in making both those tools a little more accessible and adding features many of our users have requested over the years.

Below, we’re going to look at some of the features of Feed Control that are currently not available in Full-Text RSS and Feed Creator. (We have plans for more, which we’ll be covering here in due time.)

Note: Feed Control is currently only available as a hosted service. We have no plans yet to offer a self-hosted version.

User interface for managing feeds

Feed Control

Create an account to store your feeds. View and manage feeds from the admin interface.

Full-Text RSS / Feed Creator

There is no record of the feeds you’ve created.

Faster feed access via a CDN

Feed Control

When you enable RSS or JSON generation for a feed, the resulting file is stored on a content delivery network (CDN) for fast access. This is automatically updated as new items for the feed are pulled in.

Full-Text RSS / Feed Creator

Full-Text RSS and Feed Creator process and generate feeds on an ad-hoc basis, as requests come in (with some caching to increase performance). This approach can result in delays when returning content, especially if the source feed is on a server that’s slow to respond.

Twitter feeds

Feed Control

Monitor and generate feeds from a user’s Twitter timeline. With or without retweets.

Full-Text RSS / Feed Creator

Not available.

Email alerts

Feed Control

Enable email alerts for a feed to receive a notification for each new item detected, or a daily, weekly, or monthly summary.

Full-Text RSS / Feed Creator

Not available.

Webhooks

Webhooks are intended for developers.

Feed Control

Enable webhooks for a feed, and Feed Control will send your application data for each new item. It will send the original HTML; a stripped-down, sanitized version; and plain text.

It’s a great alternative to polling feeds for updates in your application, and also a nice solution for serverless setups. Read more about webhooks in our documentation, including basic code examples.

Full-Text RSS / Feed Creator

Not available.

Generate stripped-down HTML

Feed Control

Feed items that contain a lot of HTML with non-essential elements or styling can be stripped-down further to improve display. When enabling feed generation in Feed Control, you have the option of enabling this.

Full-Text RSS / Feed Creator

Not available.

Filtering

Feed Control

Set up filters to only include items of interest, e.g. ignore a tweet unless it mentions ‘covid’, or alternatively, ignore tweets that do mention ‘covid’. You can filter on the item URL too, e.g. ignoring items unless they contain the segment ‘/news/’ in the URL.

Full-Text RSS / Feed Creator

Filtering is not available in Full-Text RSS, but is available in Feed Creator.

Try for free

We think Feed Control is the best place to start for most users, but you can try all three for free:

Feedback or questions?

If you have any questions or feedback about our tools, or are unsure which one’s right for you, feel free to get in touch on our forum, by email, or Twitter.

The post Feed Control, Full-Text RSS, Feed Creator: Which to choose? appeared first on FiveFilters.org.

]]>
Introducing Feed Control https://www.fivefilters.org/2020/feed-control/ Sat, 19 Dec 2020 11:02:00 +0000 https://www.fivefilters.org/2020/feed-control/ Feed Control is a new, hosted service we're offering here at FiveFilters.org. It can monitor different types of feeds: RSS feeds, Twitter feeds, and webpages. For each feed you can set up filtering, email alerts and webhooks, as well as RSS and JSON generation.

The post Introducing Feed Control appeared first on FiveFilters.org.

]]>
Feed Control is a new, hosted service we’re offering here at FiveFilters.org. It can monitor different types of feeds: RSS feeds, Twitter feeds, and webpages. For each feed you can set up filtering, email alerts and webhooks, as well as RSS and JSON generation.

For regular RSS and webpage-to-RSS feeds, you can also enable feed expansion to include the full article content.

Filtering

Set up filters to only include items of interest, e.g. ignore a tweet unless it mentions ‘covid’, or alternatively, ignore tweets that do mention ‘covid’. You can filter on the item URL too, e.g. ignoring items unless they contain the segment ‘/news/’ in the URL.

When monitoring a Twitter account, it’s also possible to include or exclude retweets.

Email alerts

Enable alerts to be notified by email when new items are detected. Email alerts can also be sent daily, weekly, or monthly, if the feed being monitored produces too many alerts.

Webhooks

Webhooks are intended for developers. Feed Control can send your application item data for each new item. Receive the original HTML; a stripped-down, sanitized version; and plain text.

It’s a great alternative to monitoring feeds in your application, and also a nice solution for serverless setups. Read more about webhooks in our documentation, including basic code examples.

Feed expansion

Transform truncated feeds (those that do not contain the full article content) into full-text feeds by having the article content pulled in for each item.

RSS and JSON generation

After configuring your feed, you can then enable RSS and JSON generation. This allows you to subscribe to your feed in your favourite news reader, or pull in the JSON file in your application. It’s also possible to use the RSS output to integrate further via services like Zapier and IFTTT.

Try it free

You can try Feed Control with a free, 7-day trial give you access to all its features. Get started here.

Please let us know if you have any trouble.

Upcoming changes to the Full-Text RSS free service

With Feed Control now available, we’re going to be limiting the free Full-Text RSS service we provide. These changes do not affect users of the premium, hosted service (users with an access key).

What to expect?

  • If you use a news reader and a feed generated by our free service – feed beginning with ftr.fivefilters.org/makefulltextfeed.php – you will soon start seeing a notice at the end of your feed items pointing you to this blog post and suggesting you switch to Feed Control with feed expansion instead.
  • You might notice less frequent updates as we start rate limiting the free service. This might not affect you at all or as much if you run your news reader on your own computer, but if you use a hosted, cloud-based service, you might notice fewer updates to your feeds.
  • When creating new feeds via the free service, we’ll show you Feed Control as an option.

Why are we doing this?

Managing the free service has simply become more and more of a burden over time. We have never required account signups for using the service, as such, it’s been hard to control usage, leading to excessive strain on our servers.

What are your options?

Users of the free Full-Text RSS service have a few options:

  1. If Feed Control offers what you need, you can signup for an account and move your feeds over.
  2. You can signup for our premium Full-Text RSS hosted plan where you’ll receive an access key to our premium service.
  3. You can buy the self-hosted version of Full-Text RSS to host it yourself.
  4. If your only use is via your own code (and you don’t need a publicly accessible feed URL), our hosted API is also available.

The post Introducing Feed Control appeared first on FiveFilters.org.

]]>