Content theft, also known as web scraping, is becoming more and more common on the internet. Many website owners get headaches when all the effort they put into their content is stolen, and the copied article sometimes even ranks higher than the original. In this article, Lucid Gen shares some tips to prevent web scraping that will frustrate content thieves and make them want to give up.
Read this before you decide
When your content first gets stolen, you will probably be eager to find a way to stop it completely. But you should know that stopping it “completely” is impossible. Read through the sections below to understand the nature of the problem and choose the right course of action.
What is web scraping?
Web scraping is the process of collecting and extracting data from a website for the purposes of whoever runs the tool. It is often used for legitimate reasons such as gathering information, but many people abuse it to steal other websites’ content.
Web scraping bots do this nonstop, every day, hour, and minute, to find the latest content and deliver it to their users quickly. These bots are hard for regular website owners to spot, but don’t worry, Lucid Gen will show you how to identify them.
Why do they steal your website content?
There are many reasons, but the most common one is “wanting the reward without the work”. They want traffic but are too lazy to write articles and can’t afford good content writers, so they take a shortcut and steal other people’s work.
The second type learned from “gurus” who sell courses on making money with “auto blogging”. To put it bluntly: create a website, install a few plugins that copy articles from other websites, then wait for AdSense money. If you come across one of these courses, just walk away. It’s not that easy, my friend.
There are other types too. But rest assured, it usually doesn’t end well for them. Google rates websites that copy other websites’ content very low, and they may not even get approved for AdSense. Copying for a long time also becomes a habit, so they never learn to create anything and will always be one step behind you.
How do they steal?
From Lucid Gen’s research, there are three common types of content theft today. I will list them from the easiest to prevent to the hardest.
Steal content using RSS
Web scraping bots read your RSS feeds to discover the latest posts. They then visit each new article, extract its content, and copy it to their own server. RSS feeds usually look like this:
https://domain.com/feed(WordPress uses this one)https://domain.com/feedshttps://domain.com/rss
This type of content theft is easy to prevent. Just delay your RSS updates and block the IPs of web scraping bots, and you’ll be fine. Lucid Gen explains the details below.
Steal content using HTML
Web scraping bots visit your blog page and article categories, then analyze the HTML to discover new articles.
With this type, the bots don’t access your RSS feeds, so they are harder to detect. But there are still ways to deal with them, which Lucid Gen explains in the prevention section.
Stealing content manually
In this case, the thieves are more “hardworking” and copy and paste by hand. No tool can prevent this. The only way is to be more “hardworking” than the thief: add more internal links, put watermarks on your images, use your own name more often…
Editing the text, changing names, and deleting links is easy for them (sometimes they even forget), but recreating images that match the content without your watermark is much harder.
Potential benefits of content theft
Lucid Gen wants to share a different perspective on this issue. In many cases, having your content stolen actually benefits the original website.
Google rates your website highly
If other websites copy your content, it’s a sign that your content is good and valuable to users. Google can usually tell which article is the original and which one is the copy. So if they steal your content but don’t take your ranking position, don’t worry too much.
Keep an eye on it for a while and let them copy until you see signs that they are competing with you for rankings, then take action. Imagine 30-50% of their website is copied from you. Taking action once and wiping out that whole 30-50% is much more satisfying than removing just 1 or 2 articles, right?
Get backlinks from competitors
Some sites copy your content but leave your internal links intact. Maybe they forgot to delete them, or they left them on purpose as a courtesy. Either way, you just got some extra backlinks.
If they don’t link back to you, try asking them to add a link to your original article. If they agree, you get a backlink. If they don’t, it’s time to take action.
Should you use a copy protection plugin?
In Lucid Gen’s opinion, no. Here is why:
- 99% of your traffic is regular users, and only 1% are manual thieves. Don’t make life harder for the 99% because of that 1%. Sometimes people need to copy text or images from your website for good reasons, such as studying or helping a friend.
- Lucid Gen also finds it annoying to visit sites that block copying or right-clicking. It feels like the owners of those websites are being a bit selfish.
- A browser extension like Enable Copy can easily turn off copy protection. Regular users may not know this, but professional content thieves certainly do.
- It makes your website heavier.
So don’t bother installing plugins that block copying. They don’t stop thieves, they only annoy your users.
How to prevent web scraping
After reading the section above, you probably understand how content theft works. Choose the solutions that match how seriously it affects your website, or better yet, use all of them together.
Lucid Gen can’t help you solve this problem “completely”. But if you apply all the measures below, content theft on your website should drop noticeably. If I find new effective measures, I will add them to this article.
Block the IPs of web scraping bots
For this method, you need to use Wordfence Premium. Wordfence will log the IP, Hostname, and User-agent of every visitor to your website, so you can filter out web scraping bots and block them.
Step 1: Set up Live Traffic. Go to Wordfence > Tools and configure it as follows.
- Amount of Live Traffic data to store: 500-5000, depending on your website traffic. You can choose a number equal to about 1/4 of your traffic.
- Maximum days to keep Live Traffic data: 7-14 days.
- Traffic logging mode: ALL TRAFFIC.
Step 2: Filter out the scraping bots to block them. Click Show Advanced Filters > select URL > contains > feed to see which bots have accessed your RSS feed.
Here is how to recognize web scraping bots:
- The User-agent is usually marked as a bot. Some content theft tools can fake a human User-agent. That case is a bit trickier, and I will cover it below.
- They visit your website at very regular intervals, for example every 5, 10, 15, 20, or 25 minutes.
- The Hostname or User-agent contains words like feed, content, newspaper…
Note to avoid confusion with friendly scraping bots:
- Google’s bot will have a Hostname containing the suffix
googlebot.com,google.com. - Bots from sites where you created bookmarks or backlinks usually include that site’s name or domain. Keep track of where you created bookmarks or backlinks so you can compare.
Now just click the BLOCK IP button to block these bots. Also note their characteristics, such as IP range, Hostname, and User-agent, for the advanced steps below.
Step 3: Add blocking rules for bots with known characteristics. Go to Wordfence > Blocking > Custom Pattern and configure it as follows.
Note: Fill in only one of IP Address Range, Hostname, or User-agent for each rule. If you fill in all three, a visitor is only blocked when it matches all three.
- Block Reason: Any general name that’s easy to remember works, for example “Web scraping bot”.
- IP Address Range: Content theft tools often change IPs, so block the whole range by replacing the last number with 0/24. For example, if you blocked 192.168.200.200, the bot may switch to 192.168.200.201 to keep stealing, so block 192.168.200.0/24 instead.
- Hostname and User-agent: Enter
*keyword*. For example, if a bot’s Hostname or User-agent often contains the word “newspaper”, enter*newspaper*. The two asterisks mean it will be blocked no matter what comes before or after the keyword.
So what about web scraping bots with a Hostname and User-agent that look like regular visitors?
- You know which websites are stealing your content, right? Find their IP addresses and block the entire IP range. WordPress content scraping plugins running on those sites will be stuck. Check from time to time whether those websites have moved to new servers, and add the new IP ranges to your block rules. Thieves won’t buy lots of servers just to steal your content. That money would be better spent hiring writers.
- Look at how often they visit. As mentioned above, web scraping bots visit at regular intervals of 5-25 minutes. If you find an IP visiting that often, block its IP range. In Live Traffic, if you suspect an IP, click SEE RECENT TRAFFIC to check whether all of its traffic looks like a bot.
Delay RSS updates
Lucid Gen finds this method simple but effective against RSS-based content theft. The goal is to make the thief’s copy get indexed after yours, so Google knows it was copied from you.
Insert this code into your theme’s functions.php file. Change the number and unit to the RSS delay you want. In this example, the delay is 12 hours. If your website gets indexed slowly, you can increase it to a few days.
//Delay RSS Feed by LucidGen.com
function publish_later_on_feed($where) {
global $wpdb;
if ( is_feed() ) {
$now = gmdate('Y-m-d H:i:s');
$wait = '12'; // integer
$device = 'HOUR'; //MINUTE, HOUR, DAY, WEEK, MONTH, YEAR
$where .= " AND TIMESTAMPDIFF($device, $wpdb->posts.post_date_gmt, '$now') > $wait ";
}
return $where;
}
add_filter('posts_where', 'publish_later_on_feed');
Lucid Gen knows you may be wondering, “Wouldn’t it be better to delay it by a few months or turn off RSS altogether?” (I used to think so too).
But you shouldn’t. As the saying goes, don’t wake a sleeping dog. Let the thief keep using a simple method, and stopping them stays simple too. If they can’t access your RSS feed, or notice too many new articles missing from it, they may get suspicious and switch to a more advanced method, which only makes your job harder.
Remember, this method aims to get sites that copy your content via RSS to index after you.
Shorten content in RSS
This method is outdated, since web scraping bots now visit the article itself instead of only reading the RSS feed. Still, it doesn’t hurt to set it up. Go to Settings > Reading and select Excerpt.
Randomize HTML classes on content pages
I heard about this method from experts in online groups, but I haven’t managed to do it myself. Randomizing HTML classes isn’t hard, but making the CSS follow those random classes is. It’s a strong solution against HTML-based content theft. Facebook and Google, for example, use random classes.
Add more internal links to the content
This one is easy. When writing an article, add internal links related to the main content. The main purpose is to let readers find more information in other articles that support the main one. The secondary purpose is to lower the quality of the content once it’s stolen.
Thieves usually delete your internal links. Imagine parts of the original article telling readers to check another article for more details, but the stolen copy doesn’t have those links. Readers will be annoyed and may realize the content is stolen. On top of that, the copied article lacks the internal links that support the original, so it can’t perform as well in SEO.
Add a watermark (logo) to images
If you look closely, you will see that all images on Lucid Gen have a watermark. If a thief uses your original images, they are basically promoting your website for you. It’s very easy to do. I use a tool to add my logo to photos in bulk before uploading them.
Important: “don’t put the logo in a corner”. Thieves will cover it with a bigger logo of their own or simply crop it out. Place it so it doesn’t bother users much but is hard for thieves to hide. You can do it the way Lucid Gen does, which is similar to what image sites like Shutterstock and Freepik do.
Use DMCA and report violating sites
Many people ask why bother with DMCA, since thieves keep stealing anyway, and you give DMCA a free backlink. It sounds reasonable, but it doesn’t convince me.
- You give DMCA a backlink: The DMCA badge links to your certificate page on dmca.com so others can verify your protection status. If you don’t want a followed outbound link, you can add rel=”nofollow” to the badge code. The badge still works and still points to your certificate page.
- Your content still gets stolen: That’s true, DMCA only helps you when you file a report.
- You don’t need DMCA to report to Google: Not quite. In some cases, Google will ask for more evidence. The simplest thing to send is your DMCA link. I once reported 180 URLs copying Lucid Gen’s content in a single complaint, and Google asked for more evidence. I sent them the DMCA link, and those 180 URLs were then removed from the search results. DMCA can help in other ways too (such as reporting to hosting providers abroad). At the very least, it shows Google that your articles were protected earlier than the copies, and if the other side has no DMCA at all, they lose.
You should use DMCA. The free plan is fine, and upgrading to Pro is even better (Pro shows thieves and Google that you take this seriously, and it comes with a few extra features).
I won’t go into DMCA in detail here. Just visit dmca.com to create an account > get the DMCA code > go back to WordPress > Appearance > Widgets to add the DMCA code to your website footer.
Finally, if you are determined to report the thieves, see how to submit a DMCA copyright infringement report to Google and follow the step-by-step instructions. However, your complaint won’t always be approved. It depends on what the Google team asks for and whether your answers meet their requirements.
Ask Google to index new content right after publishing
This is important. Let Google know about your new article as soon as you publish it. Open Google Search Console > paste the new article’s URL into the inspection bar > inspect the URL > click Request Indexing.
Use your name instead of "I" and "me"
Use your own name or your website’s brand name more often instead of pronouns like “I” and “me”. If the thief forgets to edit it, readers will realize the content was taken from your website.
Conclusion
If you apply all of the techniques above, Lucid Gen believes content theft on your website will drop significantly. If you know other good methods, please share them with Lucid Gen in the comments.
