If your site is like many on the internet, you’re probably using a plethora of caching services to speed it up. It’s a known quantifier in Google’s Page Ranking algorithm for results.
One of the most common services used online is Cloudflare. And while they offer many services for free, you might not be aware that some might be harming your most loyal fanbase from receiving your updates.
If you have Cloudflare Security enabled for Bots and crawlers, you might be invertedly preventing RSS Feed Readers from accessing your feeds by placing challenge on the loading page before directing the visitor to the queried URL. You’ve probably seen this before if Cloudflare or Google has ever asked if your a human and might have asked your to solve a puzzle or answer a question.
These “challenges” can’t be seen from the RSS Feed reader crawler and so no new information is indexed for the user using it.
How to Fix It?
The only current solution to this issue is adding the custom user-agent the feed reader uses for crawling to bypass the bot detection. This means the site owner will have to add each user-agent known for feed readers to hopefully bypass the bot detection service. And this doesn’t come with out its own issues, it might allow misidentified malicious users pretending to be a feed reader from accessing your site. This detection deterrent helps prevent ddos attacks.
