Has Stardot been purged from Google search?

feedback, comments and suggestions pertaining to the *. forums
Post Reply
User avatar
lurkio
Posts: 1572
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Has Stardot been purged from Google search?

Post by lurkio » Wed Jun 13, 2018 9:10 am

Loads of links to Stardot conversation threads seem to have been removed from Google search results, possibly as a side-effect of the recent forum software update.

For example, the following Google search now returns zero hits, when previously it returned links to multiple Stardot forum threads:

Has Google been told to stop crawling the forums?

:?:

User avatar
danielj
Posts: 6187
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: Has Stardot been purged from Google search?

Post by danielj » Wed Jun 13, 2018 9:12 am

Nope. It hasn't!

User avatar
lurkio
Posts: 1572
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Has Stardot been purged from Google search?

Post by lurkio » Wed Jun 13, 2018 9:13 am

danielj wrote:
Wed Jun 13, 2018 9:12 am
Nope. It hasn't!
I wonder what's going on then? Stardot convos used to be top hits for lots of different Acorn-related Google searches. But not any more..!

:(

User avatar
BigEd
Posts: 1831
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Has Stardot been purged from Google search?

Post by BigEd » Wed Jun 13, 2018 9:15 am

danielj wrote:
Wed Jun 13, 2018 9:12 am
Nope. It hasn't!
One of the top hits for site:stardot.org.uk is
"Board index - stardot.org.uk - Accept privacy policy"

I wonder if robots should not have to jump through that hoop? Or is this a red herring?

User avatar
Elminster
Posts: 2040
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Has Stardot been purged from Google search?

Post by Elminster » Wed Jun 13, 2018 9:17 am

Surely Google web crawling robots accepting a privacy policy is a bit of an Oxymoron. :evil:

User avatar
danielj
Posts: 6187
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: Has Stardot been purged from Google search?

Post by danielj » Wed Jun 13, 2018 9:19 am

Lemme see what I can do....(it shouldn't have had to).

User avatar
danielj
Posts: 6187
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: Has Stardot been purged from Google search?

Post by danielj » Wed Jun 13, 2018 9:50 am

It was a bug in that privacy policy extension, which I've now had to disable simply as the new version completely broke everything and I don't have time to look at it at the moment!

d.

User avatar
danielj
Posts: 6187
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: Has Stardot been purged from Google search?

Post by danielj » Wed Jun 13, 2018 10:01 am

Working again. Google should be indexing (I hope) :)


d.

User avatar
lurkio
Posts: 1572
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Has Stardot been purged from Google search?

Post by lurkio » Wed Jun 13, 2018 10:46 am

danielj wrote:
Wed Jun 13, 2018 10:01 am
Working again. Google should be indexing (I hope) :)
Thanks. So the solution you implemented was to disable the privacy policy extension completely?

I'm probably missing something, but what was it about the extension that made Google delete all its previous Stardot indexes?

:?:

User avatar
BigEd
Posts: 1831
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Has Stardot been purged from Google search?

Post by BigEd » Wed Jun 13, 2018 11:41 am

(I suppose we might need to wait a week or more for a re-index?)

User avatar
danielj
Posts: 6187
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: Has Stardot been purged from Google search?

Post by danielj » Wed Jun 13, 2018 11:43 am

lurkio wrote:
Wed Jun 13, 2018 10:46 am
Thanks. So the solution you implemented was to disable the privacy policy extension completely?

I'm probably missing something, but what was it about the extension that made Google delete all its previous Stardot indexes?

:?:
No, it was to update the version - as people had pointed out to the extension author that it was blocking bots. I think this new version should let them index, but I'll keep an eye on it. Of course, my first effort to update the extension broke stardot :D

d.

User avatar
roland
Posts: 2896
Joined: Thu Aug 29, 2013 8:29 pm
Location: Born (NL)
Contact:

Re: Has Stardot been purged from Google search?

Post by roland » Wed Jun 13, 2018 12:16 pm

I think they are already indexing again: https://www.google.com/search?q=atom+2k18
256K + 6502 Inside
MAN WOMAN :shock:

User avatar
BigEd
Posts: 1831
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Has Stardot been purged from Google search?

Post by BigEd » Wed Jun 13, 2018 12:24 pm

I'm not so sure. This is my test.

Edit: ah, it's already revisited the index page of the forum - 36 mins ago - so with a bit of luck it will re-trawl the whole archive in due course. (Total posts 202796...)

Edit: a little over an hour after that first hit, there's now one thread in the results. 14673 threads to go!

Edit: another hour, and a second thread turns up. This is so exciting!

Edit: T+20 and we see 10 threads. Is the pace slowing...
Last edited by BigEd on Thu Jun 14, 2018 8:05 am, edited 3 times in total.

cmorley
Posts: 465
Joined: Sat Jul 30, 2016 7:11 pm
Location: Oxford
Contact:

Re: Has Stardot been purged from Google search?

Post by cmorley » Wed Jun 13, 2018 12:33 pm

Does the forum now enforce SSL? Perhaps the http index has been binned if those URLs are now invalid/redirect and it needs to reindex the site with https URLs...

User avatar
danielj
Posts: 6187
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: Has Stardot been purged from Google search?

Post by danielj » Wed Jun 13, 2018 12:59 pm

It does enforce SSL, but that goes from the main index of the forum (http://stardot.org.uk/forums/). The issue here was the privacy policy thing - it was asking googlebot to agree to the privacy policy, which of course it couldn't. :)

d.

Post Reply