We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

i've been using self-hosted matomo for a couple of weeks after moving away from ga4. keeping the data on my own server was the main reason for switching.

getting everything set up took a bit of fiddling, especially bot filtering and event tracking. the traffic numbers are close to what ga4 reports, but i'm not sure the conversion data is quite as complete yet

i might keep both running for a while and compare the results, mainly to see which organic landing pages actually bring in signups

for anyone who's used matomo for a while, have you found any meaningful differences in the data, or does it mostly come down to which interface you prefer?

I'd like to analyze log data, either real or dummy data. Can someone share a link to a data source for this?

Crossposting but would love feedback.

I built a small robots.txt checker for my own sites, and last night I got curious and pointed it at 122 sites I actually use, the big ones everyone knows. It reads the file the way a crawler does (longest matching user-agent group wins, otherwise the wildcard) and tells me which of 19 AI bots are blocked.

The thing that surprised me most: six news sites block ClaudeBot and let GPTBot through. Washington Post, Guardian, Verge, WSJ, Wired, Ars Technica. Only one site anywhere did it the other way round, and it was Khan Academy of all places. I assume that's licensing deals showing up in a text file, but I don't actually know.

Most of the dev tool companies don't block anything. Cloudflare, Supabase, Netlify, Vercel, the React and Vue docs, even Anthropic's own site. The social networks and the newspapers block nearly everything, and so do Amazon and Yelp.

Bytespider is the most blocked bot on the list, ahead of GPTBot and ClaudeBot, which I didn't expect.

llms.txt is a developer thing so far. About 47 of the sites have a real one (I had to check the body, because a bunch of sites return their 404 page with a 200), and nearly all of those are companies that sell to developers. None of the newspapers I checked have one at all.

Usual caveat: robots.txt is a polite request and the server logs are what actually happened. I put the per-site results up as a json file if anyone wants to check a site they manage, allthepossibles.com/data/ai-crawlers-2026-10-19.json. The checker's mine, so take my framing with a grain of salt; the file is just what it saw.

so ahrefs put out numbers this year that only 38% of pages cited in an AI overview also rank top 10 for that query. it was 76% a year before. read that and thought fine, we're a big site, we're in the 38%.

we're not. few thousand pages, top 3 for most of our category's questions, and the overview above them cites us in almost none. pulled a Google AI answers report through GEO MarketMap on the 40 questions we track (paid, took about eight minutes) and the sources table was the bit that hurt. youtube was in most of the overviews on our questions. forums in none. us in 5 of 40. our top 3 page sitting right there under an overview built from a video.

GEO MarketMap wrote a page for one of the gaps, structured so it can actually be quoted, which is more than the AI visibility tracker we had did, that one counted citations and left me to guess. i have no idea yet whether a text page can even get into an overview that's resting on youtube for that question. nobody seems to have an answer for that part.
rough edges since someone will ask. the free site check reads about a dozen pages, not the whole site. and the right hand panel lags the chat by ten seconds or so, which made me think it broke.

anyone got a page into an overview where youtube already owned it? or is the honest answer "make a video"

Hello Experts !

I am looking for technical advice and what else data required to for the SEO guys other than we have at this platform, I would like to explain what is included in it.
it took me an year to build and here what I achieved so far.

Live Backlinks 2T
Global Keywords 12.5B
Live Pages Indexed 197.3B
LLM Prompts 338.4M
Backlink Domains 708.6M
Google & Bing Keywords 12.5B
Total Prompts 338.4 M
Google AIO Prompts 312.7 M
ChatGPT Prompts 25.7 M
Whois Domains 288.9 Million
Business Listings262.2 Million

This project started when I was too much annoyed by semrush and ahrefs bloating and charging me for nothing, even I did not use their rank trackers but they kept charging me $250+ just to add 5+ projects and see whats inside the pages, as well as their DR, DA, AS scores were fake so I decided to do it for me and created SERPSpur which tool almost an year and now its working and polished next generation giant like saas and offers everything we need for technical SEO including Onpage , offpage, AI search etc, and a lot more coming soon if you suggest anything to add.

Additional metric : I introduced TAI score as a proprietary of SERPSpur which detects fake traffic of backlinks/domains you are going to buy backlinks from. so no one can fool you now with bot traffic and sell the backlinks for $350 where they are manipulating the google search and artificially increasing traffic, getting redirect links and gaining dr etc, not I tried to put a fullstop on all of this.

Please review and let me know how it works for you and what else is important for the technical stuff.