Has Stardot been purged from Google search?

[lurkio]

Loads of links to Stardot conversation threads seem to have been removed from Google search results, possibly as a side-effect of the recent forum software update.

For example, the following Google search now returns zero hits, when previously it returned links to multiple Stardot forum threads:

• site:stardot.org.uk GAC quill

Has Google been told to stop crawling the forums?

[danielj]

Nope. It hasn't!

[lurkio]

Nope. It hasn't!

I wonder what's going on then? Stardot convos used to be top hits for lots of different Acorn-related Google searches. But not any more..!

[BigEd]

Nope. It hasn't!

One of the top hits for site:stardot.org.uk is
"Board index - stardot.org.uk - Accept privacy policy"

I wonder if robots should not have to jump through that hoop? Or is this a red herring?

[Elminster]

Surely Google web crawling robots accepting a privacy policy is a bit of an Oxymoron.

[danielj]

Lemme see what I can do....(it shouldn't have had to).

[danielj]

It was a bug in that privacy policy extension, which I've now had to disable simply as the new version completely broke everything and I don't have time to look at it at the moment!

d.

[danielj]

Working again. Google should be indexing (I hope)


d.

[lurkio]

Working again. Google should be indexing (I hope)

Thanks. So the solution you implemented was to disable the privacy policy extension completely?

I'm probably missing something, but what was it about the extension that made Google delete all its previous Stardot indexes?

[BigEd]

(I suppose we might need to wait a week or more for a re-index?)

[danielj]

Thanks. So the solution you implemented was to disable the privacy policy extension completely?

I'm probably missing something, but what was it about the extension that made Google delete all its previous Stardot indexes?

No, it was to update the version - as people had pointed out to the extension author that it was blocking bots. I think this new version should let them index, but I'll keep an eye on it. Of course, my first effort to update the extension broke stardot

d.

[roland]

I think they are already indexing again: https://www.google.com/search?q=atom+2k18

[BigEd]

I'm not so sure. This is my test.

Edit: ah, it's already revisited the index page of the forum - 36 mins ago - so with a bit of luck it will re-trawl the whole archive in due course. (Total posts 202796...)

Edit: a little over an hour after that first hit, there's now one thread in the results. 14673 threads to go!

Edit: another hour, and a second thread turns up. This is so exciting!

Edit: T+20 and we see 10 threads. Is the pace slowing...

[cmorley]

Does the forum now enforce SSL? Perhaps the http index has been binned if those URLs are now invalid/redirect and it needs to reindex the site with https URLs...

[danielj]

It does enforce SSL, but that goes from the main index of the forum (http://stardot.org.uk/forums/). The issue here was the privacy policy thing - it was asking googlebot to agree to the privacy policy, which of course it couldn't.

d.