Tech Blog: Update 6th July 2026

In this tech blog update, crawling spiders. Sounds like something out of a horror movie.

You might not know that there are readily available tools to ‘crawl’ any website out there to see a complete site map. One of these is called “screaming frog” – snazzy name huh. This will crawl any website you point it at and show you up to 500 results for that site.

Private, much?

While I was looking for examples of legal compliance documents, I ran screaming frog to a popular swingers/sex website. Expecting to see the basic site map, the crawler returned 500 results of which over 80% were members profiles. I was totally stunned and of course realised that privacy in this type of industry requires so many considerations in order to ensure trust, not to win it.

Switched ON by default?

I know on that particular site that the member can select whether their profiles are visible to search engines – and to be frank I doubt that the user finds the setting to disable that for a long while, if at all (how many times have you read Terms and Conditions?) – surely this should be switched OFF by default?

Well, it will be for us.

So my tech brief for the programmers will be that members profiles don’t have an obvious web address, e.g. ‘members/sam’ but to generate obfuscated web addresses in place of those e.g. ‘/u/23p98dfli’ so that they are difficult to identify. As well as using ‘no-follow’ to tell search engines not to go there. Top of the list though is to have everything behind a reliable membership “wall”. I could also consider code that measures the rate at which page are being “browsed” – a human might look at twenty profiles in an hour. A crawler would scrape twenty a second – so there’s most likely a way to write code that spots this in a milisecond and stops it.

To use a previous analogy – this is turning out to be a massive onion.

So, thought filed – I’m going to make sure my architectural brief (for the clever programmers) includes this in my ‘want’ list for Circles.

Today I’m going to finish the Investors Pack. It’s not the kind of ‘admin’ I’m a fan of, but I need to make sure that Circles has the best chance possible of getting the investment needed to make sure it is robust, future-proof and scalable.

Another ‘footer’ link then, might have to make the site wider.