Earth Notes: On Website Technicals (2026-10)
Updated 2026-10-25 10:41 GMT.By Damon Hart-Davis.

: RSS Bad Bots
A further small experimental tightening of the screw for /rss/.*rss requests: any bad bot (anywhere in the list of greedy pollers) will get a 429 during skipHours.
% egrep ' /rss/.[^ ]*rss .*TPA' /var/log/apache2/other_vhosts_access.log | tail [22/Oct/2026:09:51:15 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0" [22/Oct/2026:11:58:07 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0" [22/Oct/2026:14:18:14 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0" [22/Oct/2026:16:12:41 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0" [22/Oct/2026:17:57:59 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0" [22/Oct/2026:20:06:49 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0" [22/Oct/2026:22:35:39 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0" [23/Oct/2026:00:01:01 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0" [23/Oct/2026:01:52:26 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0" [23/Oct/2026:03:56:28 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0"
There has been a slight improvement (halving polling overnight) for one:
% egrep ' /rss/.[^ ]*rss .*MuckRackFeedParser' /var/log/apache2/other_vhosts_access.log | tail [22/Oct/2026:19:33:43 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [22/Oct/2026:20:27:07 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [22/Oct/2026:21:22:20 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [22/Oct/2026:22:16:48 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [23/Oct/2026:00:04:57 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [23/Oct/2026:01:57:50 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [23/Oct/2026:03:57:17 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [23/Oct/2026:05:58:22 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [23/Oct/2026:08:10:22 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com" [23/Oct/2026:09:03:30 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
: FressRSS
I reported:
Do you now have the retry interval capped at 1h maximum? If so that is way too short, eg for monthly podcast episodes. I suggest a cap of no less than a day, which won't hideously break things even for very badly configured feeds.
For example there is a FreshRSS/1.30.0 instance currently polling me hourly on a feed that I have deliberately kept unchanged for several months as part of a larger attempt to reduce compute/bandwidth/carbon waste. The normal cadence of that feed is roughly monthly, but it doesn't matter at all for that feed or my related ones if someone waits a day or three to hear an update. The Cache-Control (and Retry-After) headers are set to many hours to steer polls to the middle of the day, and skipHours is set to request avoid polling my off-grid powered server overnight in part since I only update feeds during the day.
Very quickly someone came back to me to say the cap is 48h, which is good.
The response headers currently being seen by that client will be:
HTTP/1.1 429 Too Many Requests Date: Sun, 25 Oct 2026 10:35:27 GMT Server: Apache Retry-After: 3420 Vary: Accept-Encoding,Referer Upgrade: h2 Connection: Upgrade, close Last-Modified: Wed, 04 Mar 2026 10:09:21 GMT Accept-Ranges: bytes Content-Length: 564 X-Frame-Options: DENY Content-Type: text/html Content-Encoding: gzip
: Bigger Bad Bot
I am watching in my logs a very persistent bot pulling about 1 page per second, claiming to be
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36, which when blocked at a firewall comes in from a new address within seconds, often in a different country, always apparently from a hosting company.
This is well-resourced content theft by the smell of it.
: RSS Bad Bots
# Greedy podcast feed pullers: keys are MD5 hashed User-Agent.
# A non-empty txt map lookup of the %{md5:%{HTTP:User-Agent}} means bad!
# Built: 2026-10-19T11:04+00:00 (UTC)
# MAXHITSPERUAPERDAY: 10
# MAXUAS: 25
#----------------
# request-count User-Agent
# MD5hash approx-hits-per-day
#----------------
# 202 Spotify/1.0
4582d9bdbcef42af27d89da91c6eb804 202
# 142 Google-Podcast
8dea568b39db0451edd6b30f29238eaf 142
# 59 iTMS
97f76eb7e02c5ff923e1198ff1c288cd 59
# 47 Amazon Music Podc
d69be2563c9f1929edf2906d41809aea 47
# 46 atomstr/0.9.13
fed9820e6b1b919d848e718e0bf49e41 46
# 37 axios/1.6.8
b534882134248c9a5957e0c011a37037 37
# 35 Mozilla/5.0 (Maci
dbb1533fc27ea415389d2e9c70c8e051 35
# 35 Mozilla/5.0 Apple
fac378ac3b3d3886829021b3309d4fd1 35
# 31 MuckRackFeedParse
62b46fff1cf5f8af7b4b37a2f783b57a 31
# 27 Mozilla/5.0 (Wind
6b9a00393fb1607b0ada13520f814ab5 27
# 20 itms
2e7f714a929b3f52f3c094710819a99a 20
# 11 TPA/1.0.0
86e71fcf5ba78d28f18270f7f83256bb 11
: Pasteurisation Tweaks
In script/heat-battery-target.sh
I have just pushed out the maximum interval between pasteurisation runs, eg from the grid even in the absence of 'green' (low-carbon-intensity) hours, from 30 days to 33 days to slightly reduce demand, avoid any multiple of 7 days, and still keep such last-resort runs to roughly monthly.
: Missing 10W
When the router is off-grid (~10W) and the MBA shows that it is getting ~20W, the solar controller shows ~40W being drawn, ie ~10W goes missing.
I have checked/swapped/measured various points in the system, and do not seem to be losing more than ~1–2W in the wiring to upstairs, and nothing obvious in the 12V switched cigar-lighter circuit at my desk. Even swapping out the final USB-C cable for a fatter one seems nugatory. Maybe the 12V car charger really is turning ~8W into heat, though it does not seem hot enough for that...
I will recheck my wiring run at leisure to make sure that resistance has been minimised by use of cores etc.
: AlDente Tweaks
Apparently the scheduler bug I reported 3 years ago is about to be fixed.
Since I am trying to keep my MBA entirely off-grid while at home, though may not succeed as the days get shorter, yesterday and today I
simplified my AlDente setup
by disabling half the schedule rules (leaving 3). Now all that happens is being forced into lowpowermode
as peak time starts (but not out afterwards, so leaving it low), and halving the charge limit during peak time to deter charging.
If I want out of lowpowermode I will adjust it manually; easy.
Seeing AlDente-reported MBA power consumption under 3W in light evening use; down to ~2.3W at times.
Importing
This week we started to import from the grid overall:
Last October had net electricity emissions of 11kgCO2e:
Net electricity utility energy kgCO2e over last 12 calendar months (before ): -64
Net kgCO2e for each of the last 12 full calendar months: 11 27 42 52 28 -2 -22 -40 -52 -60 -40 -8
Possible HTML output micro-optimisation
Looking at minified HTML output it seems that almost any paragraph close at EOL (and preceding and trailing whitespace) could be safely stripped, such as with a
regex. An exception is a paragraph wrapped in *</p> *$<ins>.
: Battery Reserve
Every time I have tried Enphase ToU tariff stuff it has been very broken, but I would like another go, in case it still exists and is better now!
My aim is to avoid putting on the grid in winter in the peak demand window our typical cooking load of 1kWh+ between ~ and ~.
I think that to achieve this, all year or from October to February, during the day accumulate a reserve of ~xkWh of battery charge. If the battery has >xkWh stored, or after (to midnight nominally), then the battery can cover loads automatically in the normal way. x is somewhere between 1 and 3; mid-winter generation is ~1.5kWh/d.
: PV generation chart
: removeMacAI
I do not want to let 12GB be absorbed by features that I do not want to use. (The 12GB is visible when examining About this Mac ... Storage ... macOS.)
% df -h . Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk3s5 926Gi 737Gi 164Gi 82% 3.6M 1.7G 0% /System/Volumes/Data % brew install omlahore/tap/removemacai % removemacai % df -h . Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk3s5 926Gi 736Gi 165Gi 82% 3.6M 1.7G 0% /System/Volumes/Data % removemacai status RemoveMacAI 0.2.5 ยท macOS 27.0 Features Siri and Siri AI off (locked) ChatGPT and other AI extensions off (locked) Writing Tools off (locked) Genmoji off (locked) Image Playground off (locked) Mail summaries and smart replies off (locked) Notification summaries off (locked) Messages summaries off (locked) Safari summaries off (locked) Notes transcription summaries off (locked) Inline text predictions off (locked) Spatial Photos off (locked) Photos Clean Up off (locked) Xcode predictive code completion off (locked) Models on disk Apple Intelligence foundation models none Image and Genmoji models none Spatial Photos models none Photos Clean Up models none Xcode code completion models none Total 0 MB macOS removes deleted model files itself, so System Settings can count them for a while. Apple Intelligence is off. Undo with: removemacai revert
: going...
% df -h . Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/disk3s5 926Gi 728Gi 172Gi 81% 3.6M 1.8G 0% /System/Volumes/Data
: Variable Eddi Export Margin
It is (now) possible to vary the Eddi's Export Margin via the app, ie remotely, but MyEnergi will not even informally publish that part of the API for the rest of us.
If I had the facility, rather than the current fixed 300W, I think I would lower the margin to 100W, or as low as reasonably possible (maybe 50W), to play nicely with the Enphase battery:
- During a pasteurisation cycle, to maximise the chance of it completing.
- When the grid is super-green (or pricing negative), ie has an abundance of low-carbon generation.
- When the heat battery is cold and the grid carbon intensity not high/red.
I might also raise the Export Margin above 300W when the heat battery is nearly full to aim to just fill it by peak time.
Google-Podcast reported
I filled in the
Google Search Central Overcrawling Report form for Google-Podcast, for its ~140 polls per day of an RSS feed that is unchanged since January, and where an initial 429 response is followed by another five requests that each also receive 429.
Completely ignores 429 status and Retry-After and repolls *immediately* 5 times after the first 429. The RSS podcast feed it polls has not changed since January. This is an absurd waste of energy and carbon budget.www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:03 +0000] "HEAD /rss/podcast.rss HTTP/1.1" 429 5204 "-" "Google-Podcast" www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:04 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5768 "-" "Google-Podcast" www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:10 +0000] "HEAD /rss/podcast.rss HTTP/1.1" 429 5204 "-" "Google-Podcast" www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:10 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5768 "-" "Google-Podcast" www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:21 +0000] "HEAD /rss/podcast.rss HTTP/1.1" 429 5204 "-" "Google-Podcast" www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:21 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5768 "-" "Google-Podcast"
See also Googlebot Still Rogue...
References
- [eden2026DOI] How to get a DOI for your blog posts
- [pathak2026ewaste] Operations Research-Based Formalisation and Design of Sustainable Reverse Logistics for E-Waste Supply Chains
(Count: 2)