r/Blogging • u/ModwildTV • 12d ago
Question Persistent, debilitating automated traffic
I'm wondering whether any other publishers are seeing anything similar, because I'm drowning in automated traffic, and there's no end (or help) in sight.
Since August 31, TV Fanatic has been hit by persistent waves of abnormal traffic that are heavily concentrated on our homepage. The pattern is remarkably consistent: huge desktop-heavy surges, mostly direct traffic, very short sessions (10 seconds), often one or two actions, arriving across constantly changing (apparently residential) IPs, networks, and even countries.
It comes in distinct waves throughout the day and night. During the quiet periods, our normal readership reappears.
Then another wave arrives.
The business impact has been devastating. Our traffic numbers look enormous while actual readership of current articles has collapsed relative to those numbers, and our advertising RPM has fallen dramatically. Mediavine has confirmed that the traffic change is specific to our site and began around August 31.
I'm not looking for answers here. Instead, I'm trying to find out whether we're truly alone. And I even have visitor verification on, but my host only sees humans. I know they are not. I've been studying my traffic for over a decade. This bot behavior.
Has anyone else seen a sudden increase in direct, desktop-heavy traffic, unusual homepage activity, rhythmic traffic surges, or a major RPM decline that doesn't make sense compared with your actual readership?
Even if you dismissed it as ordinary bot traffic at first, I'd really like to hear what you're seeing. 🙏
2
u/RankingInTheMeadow 10d ago
I’m at the beginning of my blogging journey for my current business, so I was very skeptical when there was a spike in traffic and then ongoing waves of traffic, too early on (days into beginning to publish). Definitely bots and a slew of them! Beyond rhythmic, it was same time, same wave, and same device types…
In addition to the standard bot protection, I added a few custom firewall rules and that seemed to clean things up A TON! So I’d say maybe consider adding a few custom firewall rules and see how that goes. Hope it helps and your RPM recovers!
2
u/ModwildTV 10d ago
Welcome to the rollercoaster!! Of blogging AND bots!
All of my bot mitigations had previously worked. I'm now about 97% certain they have chosen my site as their training ground to break through barriers. My host finally got the message after I sent enough data to teach a class on bot fingerprints. They have turned on Precursor at the enterprise level. And they are shocked it's still not enough to battle what's attacking me. They even have Cloudflare on the case now.
Honestly, I don't know that I will be able to get rid of them. Every switch we pull, the circumvent. Cut them off by location, they move. By fingerprint, they change them. Everything I see indicates a training ground because I own a 20 year old site with complexities and over 100k posts.
Well, I'm doing my best to slam the door in their faces, hoping that what my host and Cloudflare learns helps stop them before they harass others like they have me. Wish me luck!!
Now that my host is helping, I'm breathing a little easier. But my RPM is at $2. It was $23 in August, and that was a bad month for us.
2
u/RankingInTheMeadow 10d ago
Thank you 😂 Part of my old career responsibilities was managing the website and blog for an ecommerce brand, so while I no longer have a team, I love building something that’s “mine”, even if it comes with tons of shenanigans!
I’m sure you probably have, but one thing that helped me was to enforce no AI bot training. If your host and Cloudflare haven’t done this yet, it might be worth a try.
Edit: Oh my 🥹 Not $23 was a bad month and now it’s at $2…
2
u/ModwildTV 10d ago
At this point, there isn't a block I haven't tried. These new bad actors are the real deal. I have to assume it's AI based and learning like crazy. Sadly, they aren't the kind that will obey the rules. They're nefarious and couldn't care less about who's hunting them. In fact, it seems to make them stronger. Scary stuff, actually.
1
u/RankingInTheMeadow 10d ago
I’m so sorry to hear that! I hope between your host and platform it gets sorted though 🤞🏾
And if I happen to come across anything that might be even remotely useful, I’ll circle back and drop it here.
I have a weekly standing task (already) dedicated to log monitoring and bot mitigation 🫠
1
u/ModwildTV 10d ago
Same because I truly believe I'm where they're going their skills. And if I don't stop them they may all be coming for the rest of you. And when I say they're are pervasive, they are with me 25/7 by the hundreds. By the end of the day, I have thousands of homepage hits at under 10 seconds. They're really working it!
2
u/Flaky-Eggplant-3479 8d ago
Pre-AI we used to call this "the internet saying hi". Kinda the same thing now but different scale; agents are beginning to become the primary consumers of content on the internet. Its already true for some types of content (e.g. documentation) and i think it only increases from here
2
u/ModwildTV 8d ago
The level and complexity of the traffic is extraordinary. And it beats Cloudflare's Precursor, so they're actively investigating it. Maybe my trouble can eventually help others. 🤞🏼🤞🏼🤞🏼
2
u/Flaky-Eggplant-3479 8d ago
it would be nice if the owners of the bots respected standard protocols like robots.txt or agents.txt. then you could have an optimized hot path for content that you wanted to be digested that way
1
u/ModwildTV 8d ago
They respect nothing. Hell, if they hung around and let ads load, I'd let them using my site for training. 🤣 Kidding!! Kind of. 🤭
1
u/Flaky-Eggplant-3479 8d ago
This probably isn't super helpful but on my actual applications, I use an agent that reviews traffic logs on a schedule and updates a ruleset at the load balancer. My blog itself is low traffic so its kind of whatever. It's using a combo of Mercury 2.5 and GLM model families so its quite cheap
1
u/jethropickering 12d ago
Sounds like a negative ctr campaign?
2
u/ModwildTV 12d ago
That's an interesting thought. I considered deliberate SEO manipulation, but most of this traffic is direct/no-referrer and overwhelmingly targets the homepage rather than arriving from Google search results. We've also seen it systematically traverse very old/archive URLs. So far, I haven't found evidence tying it to search-result clicks, although I can't rule out some kind of deliberate manipulation. That would be my luck.
1
u/lordchickenburger 12d ago
cloudflare is partnered with browserbase a web scraping company. All their blocks are fucking useless. fuck cloudflare
1
u/jensmalmgren 8d ago
Is it Muse doing this?
1
u/lordchickenburger 8d ago
Yes im seing muse.ai referalls
1
u/jensmalmgren 8d ago
Recenty came across this about Muse.
1
u/lordchickenburger 8d ago
Yes the internet will be flooded with ai agents in the next decade. Webmcp is already underway for agents to navigate sites. Agents will be first class citizen of the internet.
1
u/jensmalmgren 8d ago
Ad-based revenue models will die.
1
u/lordchickenburger 8d ago
Its already happening openai and google ai overviews are hijacking you to their shitty chat app and puting their ads there. They steal from publishers and give them a final fuck you by siphoning ads to their platform
3
u/baddaywithacamera 12d ago
Are you behind Cloudflare?