Earth Notes: On Website Technicals (2026-10)

Updated 2026-10-25 10:41 GMT.
By Damon Hart-Davis.
Tech updates: Export Margin, RemoveMacAI, gen chart, reserve, AlDente simplified, 10W, pasteurisation, bad bots...
tools 800w JA
Observing the flip over into net consumption this month...

: RSS Bad Bots

A further small experimental tightening of the screw for /rss/.*rss requests: any bad bot (anywhere in the list of greedy pollers) will get a 429 during skipHours.

% egrep ' /rss/.[^ ]*rss .*TPA' /var/log/apache2/other_vhosts_access.log | tail
[22/Oct/2026:09:51:15 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0"
[22/Oct/2026:11:58:07 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0"
[22/Oct/2026:14:18:14 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0"
[22/Oct/2026:16:12:41 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0"
[22/Oct/2026:17:57:59 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0"
[22/Oct/2026:20:06:49 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "TPA/1.0.0"
[22/Oct/2026:22:35:39 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0"
[23/Oct/2026:00:01:01 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0"
[23/Oct/2026:01:52:26 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0"
[23/Oct/2026:03:56:28 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5586 "-" "TPA/1.0.0"

There has been a slight improvement (halving polling overnight) for one:

% egrep ' /rss/.[^ ]*rss .*MuckRackFeedParser' /var/log/apache2/other_vhosts_access.log | tail
[22/Oct/2026:19:33:43 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[22/Oct/2026:20:27:07 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[22/Oct/2026:21:22:20 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[22/Oct/2026:22:16:48 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[23/Oct/2026:00:04:57 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[23/Oct/2026:01:57:50 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[23/Oct/2026:03:57:17 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[23/Oct/2026:05:58:22 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5418 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[23/Oct/2026:08:10:22 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"
[23/Oct/2026:09:03:30 +0000] "GET /rss/podcast.rss HTTP/1.1" 304 4941 "-" "MuckRackFeedParser/1.0 +https://muckrack.com"

: FressRSS

I reported:

Do you now have the retry interval capped at 1h maximum? If so that is way too short, eg for monthly podcast episodes. I suggest a cap of no less than a day, which won't hideously break things even for very badly configured feeds.

For example there is a FreshRSS/1.30.0 instance currently polling me hourly on a feed that I have deliberately kept unchanged for several months as part of a larger attempt to reduce compute/bandwidth/carbon waste. The normal cadence of that feed is roughly monthly, but it doesn't matter at all for that feed or my related ones if someone waits a day or three to hear an update. The Cache-Control (and Retry-After) headers are set to many hours to steer polls to the middle of the day, and skipHours is set to request avoid polling my off-grid powered server overnight in part since I only update feeds during the day.

Very quickly someone came back to me to say the cap is 48h, which is good.

The response headers currently being seen by that client will be:

  HTTP/1.1 429 Too Many Requests
  Date: Sun, 25 Oct 2026 10:35:27 GMT
  Server: Apache
  Retry-After: 3420
  Vary: Accept-Encoding,Referer
  Upgrade: h2
  Connection: Upgrade, close
  Last-Modified: Wed, 04 Mar 2026 10:09:21 GMT
  Accept-Ranges: bytes
  Content-Length: 564
  X-Frame-Options: DENY
  Content-Type: text/html
  Content-Encoding: gzip

: Bigger Bad Bot

I am watching in my logs a very persistent bot pulling about 1 page per second, claiming to be Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36, which when blocked at a firewall comes in from a new address within seconds, often in a different country, always apparently from a hosting company.

This is well-resourced content theft by the smell of it.

: RSS Bad Bots

# Greedy podcast feed pullers: keys are MD5 hashed User-Agent.
# A non-empty txt map lookup of the %{md5:%{HTTP:User-Agent}} means bad!
# Built: 2026-10-19T11:04+00:00 (UTC)
# MAXHITSPERUAPERDAY: 10
# MAXUAS: 25
#----------------
# request-count User-Agent
# MD5hash approx-hits-per-day
#----------------
# 202 Spotify/1.0
4582d9bdbcef42af27d89da91c6eb804 202
# 142 Google-Podcast
8dea568b39db0451edd6b30f29238eaf 142
# 59 iTMS
97f76eb7e02c5ff923e1198ff1c288cd 59
# 47 Amazon Music Podc
d69be2563c9f1929edf2906d41809aea 47
# 46 atomstr/0.9.13
fed9820e6b1b919d848e718e0bf49e41 46
# 37 axios/1.6.8
b534882134248c9a5957e0c011a37037 37
# 35 Mozilla/5.0 (Maci
dbb1533fc27ea415389d2e9c70c8e051 35
# 35 Mozilla/5.0 Apple
fac378ac3b3d3886829021b3309d4fd1 35
# 31 MuckRackFeedParse
62b46fff1cf5f8af7b4b37a2f783b57a 31
# 27 Mozilla/5.0 (Wind
6b9a00393fb1607b0ada13520f814ab5 27
# 20 itms
2e7f714a929b3f52f3c094710819a99a 20
# 11 TPA/1.0.0
86e71fcf5ba78d28f18270f7f83256bb 11

: Pasteurisation Tweaks

In script/heat-battery-target.sh I have just pushed out the maximum interval between pasteurisation runs, eg from the grid even in the absence of 'green' (low-carbon-intensity) hours, from 30 days to 33 days to slightly reduce demand, avoid any multiple of 7 days, and still keep such last-resort runs to roughly monthly.

: Missing 10W

When the router is off-grid (~10W) and the MBA shows that it is getting ~20W, the solar controller shows ~40W being drawn, ie ~10W goes missing.

I have checked/swapped/measured various points in the system, and do not seem to be losing more than ~1–2W in the wiring to upstairs, and nothing obvious in the 12V switched cigar-lighter circuit at my desk. Even swapping out the final USB-C cable for a fatter one seems nugatory. Maybe the 12V car charger really is turning ~8W into heat, though it does not seem hot enough for that...

I will recheck my wiring run at leisure to make sure that resistance has been minimised by use of cores etc.

: AlDente Tweaks

Apparently the scheduler bug I reported 3 years ago is about to be fixed.

Since I am trying to keep my MBA entirely off-grid while at home, though may not succeed as the days get shorter, yesterday and today I simplified my AlDente setup by disabling half the schedule rules (leaving 3). Now all that happens is being forced into lowpowermode as peak time starts (but not out afterwards, so leaving it low), and halving the charge limit during peak time to deter charging.

If I want out of lowpowermode I will adjust it manually; easy.

Seeing AlDente-reported MBA power consumption under 3W in light evening use; down to ~2.3W at times.

Importing

This week we started to import from the grid overall:

At 16WW over 7 days from 2026-10-05 to 2026-10-12: electricity import 1.1kWh/d (net 0.3kWh/d).
At 16WW over 14 days from 2026-09-28 to 2026-10-12: electricity import 0.9kWh/d (net -0.5kWh/d).
At 16WW over 28 days from 2026-09-14 to 2026-10-12: electricity import 0.8kWh/d (net -1.5kWh/d).
At 16WW over 371 days from 2025-10-06 to 2026-10-12: electricity import 3.1kWh/d (net -2.3kWh/d).

Last October had net electricity emissions of 11kgCO2e:

Net electricity utility energy kgCO2e over last 12 calendar months (before ): -64

Net kgCO2e for each of the last 12 full calendar months: 11 27 42 52 28 -2 -22 -40 -52 -60 -40 -8

Possible HTML output micro-optimisation

Looking at minified HTML output it seems that almost any paragraph close at EOL (and preceding and trailing whitespace) could be safely stripped, such as with a *</p> *$ regex. An exception is a paragraph wrapped in <ins>.

: Battery Reserve

Every time I have tried Enphase ToU tariff stuff it has been very broken, but I would like another go, in case it still exists and is better now!

My aim is to avoid putting on the grid in winter in the peak demand window our typical cooking load of 1kWh+ between ~ and ~.

I think that to achieve this, all year or from October to February, during the day accumulate a reserve of ~xkWh of battery charge. If the battery has >xkWh stored, or after (to midnight nominally), then the battery can cover loads automatically in the normal way. x is somewhere between 1 and 3; mid-winter generation is ~1.5kWh/d.

: PV generation chart

Added today, charts for gen vs net grid flow.

bucketed
PV generation (shown as negative) vs net grid flows (negative means export) for with very high generation; export peak was kept away from solar noon.

: removeMacAI

I do not want to let 12GB be absorbed by features that I do not want to use. (The 12GB is visible when examining About this Mac ... Storage ... macOS.)

% df -h .
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5     926Gi   737Gi   164Gi    82%    3.6M  1.7G    0%   /System/Volumes/Data
% brew install omlahore/tap/removemacai
% removemacai
% df -h .
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5     926Gi   736Gi   165Gi    82%    3.6M  1.7G    0%   /System/Volumes/Data
% removemacai status
RemoveMacAI 0.2.5  ยท  macOS 27.0

Features
  Siri and Siri AI                        off (locked)
  ChatGPT and other AI extensions         off (locked)
  Writing Tools                           off (locked)
  Genmoji                                 off (locked)
  Image Playground                        off (locked)
  Mail summaries and smart replies        off (locked)
  Notification summaries                  off (locked)
  Messages summaries                      off (locked)
  Safari summaries                        off (locked)
  Notes transcription summaries           off (locked)
  Inline text predictions                 off (locked)
  Spatial Photos                          off (locked)
  Photos Clean Up                         off (locked)
  Xcode predictive code completion        off (locked)

Models on disk
  Apple Intelligence foundation models    none
  Image and Genmoji models                none
  Spatial Photos models                   none
  Photos Clean Up models                  none
  Xcode code completion models            none
  Total                                   0 MB
  macOS removes deleted model files itself, so System Settings can count them for a while.

Apple Intelligence is off. Undo with: removemacai revert

: going...

% df -h .
Filesystem      Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s5   926Gi   728Gi   172Gi    81%    3.6M  1.8G    0%   /System/Volumes/Data

: Variable Eddi Export Margin

It is (now) possible to vary the Eddi's Export Margin via the app, ie remotely, but MyEnergi will not even informally publish that part of the API for the rest of us.

If I had the facility, rather than the current fixed 300W, I think I would lower the margin to 100W, or as low as reasonably possible (maybe 50W), to play nicely with the Enphase battery:

  1. During a pasteurisation cycle, to maximise the chance of it completing.
  2. When the grid is super-green (or pricing negative), ie has an abundance of low-carbon generation.
  3. When the heat battery is cold and the grid carbon intensity not high/red.

I might also raise the Export Margin above 300W when the heat battery is nearly full to aim to just fill it by peak time.

Google-Podcast reported

I filled in the Google Search Central Overcrawling Report form for Google-Podcast, for its ~140 polls per day of an RSS feed that is unchanged since January, and where an initial 429 response is followed by another five requests that each also receive 429.

Completely ignores 429 status and Retry-After and repolls *immediately* 5 times after the first 429. The RSS podcast feed it polls has not changed since January. This is an absurd waste of energy and carbon budget.
www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:03 +0000] "HEAD /rss/podcast.rss HTTP/1.1" 429 5204 "-" "Google-Podcast"
www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:04 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5768 "-" "Google-Podcast"
www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:10 +0000] "HEAD /rss/podcast.rss HTTP/1.1" 429 5204 "-" "Google-Podcast"
www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:10 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5768 "-" "Google-Podcast"
www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:21 +0000] "HEAD /rss/podcast.rss HTTP/1.1" 429 5204 "-" "Google-Podcast"
www.earth.org.uk:443 173.194.94.68 - - [05/Oct/2026:19:07:21 +0000] "GET /rss/podcast.rss HTTP/1.1" 429 5768 "-" "Google-Podcast"

See also Googlebot Still Rogue...

References

(Count: 2)

~1656 words.