Page 1 of 1

Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:10 am
by lurkio
Loads of links to Stardot conversation threads seem to have been removed from Google search results, possibly as a side-effect of the recent forum software update.

For example, the following Google search now returns zero hits, when previously it returned links to multiple Stardot forum threads:

Has Google been told to stop crawling the forums?

:?:

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:12 am
by danielj
Nope. It hasn't!

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:13 am
by lurkio
danielj wrote:
Wed Jun 13, 2018 9:12 am
Nope. It hasn't!
I wonder what's going on then? Stardot convos used to be top hits for lots of different Acorn-related Google searches. But not any more..!

:(

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:15 am
by BigEd
danielj wrote:
Wed Jun 13, 2018 9:12 am
Nope. It hasn't!
One of the top hits for site:stardot.org.uk is
"Board index - stardot.org.uk - Accept privacy policy"

I wonder if robots should not have to jump through that hoop? Or is this a red herring?

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:17 am
by Elminster
Surely Google web crawling robots accepting a privacy policy is a bit of an Oxymoron. :evil:

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:19 am
by danielj
Lemme see what I can do....(it shouldn't have had to).

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 9:50 am
by danielj
It was a bug in that privacy policy extension, which I've now had to disable simply as the new version completely broke everything and I don't have time to look at it at the moment!

d.

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 10:01 am
by danielj
Working again. Google should be indexing (I hope) :)


d.

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 10:46 am
by lurkio
danielj wrote:
Wed Jun 13, 2018 10:01 am
Working again. Google should be indexing (I hope) :)
Thanks. So the solution you implemented was to disable the privacy policy extension completely?

I'm probably missing something, but what was it about the extension that made Google delete all its previous Stardot indexes?

:?:

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 11:41 am
by BigEd
(I suppose we might need to wait a week or more for a re-index?)

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 11:43 am
by danielj
lurkio wrote:
Wed Jun 13, 2018 10:46 am
Thanks. So the solution you implemented was to disable the privacy policy extension completely?

I'm probably missing something, but what was it about the extension that made Google delete all its previous Stardot indexes?

:?:
No, it was to update the version - as people had pointed out to the extension author that it was blocking bots. I think this new version should let them index, but I'll keep an eye on it. Of course, my first effort to update the extension broke stardot :D

d.

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 12:16 pm
by roland
I think they are already indexing again: https://www.google.com/search?q=atom+2k18

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 12:24 pm
by BigEd
I'm not so sure. This is my test.

Edit: ah, it's already revisited the index page of the forum - 36 mins ago - so with a bit of luck it will re-trawl the whole archive in due course. (Total posts 202796...)

Edit: a little over an hour after that first hit, there's now one thread in the results. 14673 threads to go!

Edit: another hour, and a second thread turns up. This is so exciting!

Edit: T+20 and we see 10 threads. Is the pace slowing...

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 12:33 pm
by cmorley
Does the forum now enforce SSL? Perhaps the http index has been binned if those URLs are now invalid/redirect and it needs to reindex the site with https URLs...

Re: Has Stardot been purged from Google search?

Posted: Wed Jun 13, 2018 12:59 pm
by danielj
It does enforce SSL, but that goes from the main index of the forum (http://stardot.org.uk/forums/). The issue here was the privacy policy thing - it was asking googlebot to agree to the privacy policy, which of course it couldn't. :)

d.