Earth Notes: On Website Technicals (2024-11)
Updated 2024-12-03 02:45 GMT.By Damon Hart-Davis.
2024-11-28: Footer Copyright Year
At long last the site copyright footer date is no longer hardwired into build script, forcing a script edit and rebuild of all pages in December/January when there is not a lot of sun around. But nor is it now automatically the current year.
In common with a couple of other scripts the year is pulled in from a file, so editing that will have any subsequent page builds use the new value, lazily. No forced rebuilds of all pages just for the copyright year, after this one! That is also more in the spirit of copyright law, ie there should be some human input by me to justify extending the range.
2024-11-27: Heat Battery Script Adjustment
I am slowly drifting the heat battery scripts in the right direction, at least to get the graphs looking closer in the first instance.
It will probably be a while before the Thermino gets reconnected, and it will probably be longer before I can force the heat pump for DHW, but I do have the DHW cylinder immersion heater working via the Eddi, so I have one lever for dynamic response to weather conditions.
I introduced the notion of energy system 'soft parameters' today, at
data/consolidated/softparams.txt
, mirroring a system I use when building the EOU site. This should gather many useful parameters in one place, make it easier to display and document them, especially those useful share across multiple areas of control.
(The gas supply was capped today, so we are fossil fuel free at 16WW, other than the odd candle I think. The heat pump should be running tomorrow.)
2024-11-24: Heat Battery Script Optimisation
I aim to poll the NESO Carbon Intensity API far less often now, basically only if using it might make the difference between doing a top-up of heat storage from grid or not.
To that end the script has been tweaked and re-arranged a little. I expect something up to a 4-fold reduction in polling of the API.
The P/p/Q/q flags will now appear much less often in the log, see this before-and-after pair of log entries:
2024-11-24T16:41Z MT 0 GI 112 IT 0 F WFHpqbvxN ES 1 2024-11-24T16:51Z MT 0 GI 116 IT 0 F WFHbvxN ES 1
A very quick check suggest that only ~4% of super-green samples happen during 'H' peak demand hours (and sampling happens more often at green times):
% egrep ' F [A-Za-z]*G' data/heatBattery/log/live/2024????.log | egrep ' F [A-Za-z]*H' | wc -l 681 % egrep ' F [A-Za-z]*G' data/heatBattery/log/live/2024????.log | egrep -v ' F [A-Za-z]*H' | wc -l 15701
2024-11-11: Energy Series Optimisation
As the number of energy series data streams increases, some trimming in various places to reduce bloat and to boost speed seems worthwhile.
Make series depend only on current year sources files
All energy series outputs such as consolidated CSV and MIDI files depend on all the input data sets, even though in almost all cases only the most recent file for each source data set changes.
% ls -al .work/tmp/consolidationdeps.mk 24833 Nov 4 11:27 .work/tmp/consolidationdeps.mk % wc -l !$ wc -l .work/tmp/consolidationdeps.mk 312 .work/tmp/consolidationdeps.mk
A sample from the middle of that dependencies file looks is:
... data/.flags/consolidated.flag: data/.private/YearlyMeterReadings.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2016.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2017.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2018.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2019.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2020.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2021.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2022.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2023.csv data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2024.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2008.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2009.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2010.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2011.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2012.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2013.csv ...
Limiting to only data files with the current year in their name will save make
a little time scanning file details and will still catch most stuff until the next unconditional rebuild of all series when any change is detected, so often daily anyway! No named script dependencies are excluded.
A one-line change in the makefile
gets us to:
% wc -l .work/tmp/consolidationdeps.mk 165 .work/tmp/consolidationdeps.mk
The previous sample reduces to:
... data/.flags/consolidated.flag: data/WW-PV-offgrid/EC2024.csv data/.flags/consolidated.flag: data/WW-PV-roof/E2024.csv ...
Energy series page inserts top few only
Traces of all energy series have been inserted into relevant desktop page footers until now, albeit only with graphs at year cadence. A typical page size with things set up this way:
% ls -alS saving-electricity-2024.html* 81330 saving-electricity-2024.html 12035 saving-electricity-2024.htmlgz 9877 saving-electricity-2024.htmlbr
Reducing such inserts to only be for the most important series (gas
, imp
, gen
, geno
) in this case roughly halves the overall uncompressed size:
% ls -alS saving-electricity-2024.html* 41947 saving-electricity-2024.html 9414 saving-electricity-2024.htmlgz 7749 saving-electricity-2024.htmlbr
Although the effects on compressed size are less dramatic, this change saves time building each page and decoding it in a browser!
New series
I was provoked into the optimisation when realising that I would probably find two more synthetic series helpful:
- divDHW
- all diverted solar PV DHW to immersion (resistive/CoP1) heating via the Eddi
- immDHW
- all DHW to immersion (resistive/CoP1) heating via the Eddi
There was initially some problem with the coverage being reported for at least divDHW
, but this was tracked down to buggy conversion scripts from daily Eddi data.
Looking at some of the intermediate data files, it is event that values not yet being presented for h2x
were not being handled well, as zeros:
% tail data/consolidated/energy/std/h2d/D/h2d-D.csv data/consolidated/energy/std/h1d/D/h1d-D.csv ==> data/consolidated/energy/std/h2d/D/h2d-D.csv <== 2024-10-23,Eddi,1, 2024-10-24,Eddi,1, 2024-10-25,Eddi,1, 2024-10-26,Eddi,1, 2024-10-27,Eddi,1, 2024-10-28,Eddi,1, 2024-10-29,Eddi,1, 2024-10-30,Eddi,1, 2024-10-31,Eddi,1, #script,"script/consolidate/energy/std/h2d/D/Eddi/h2d-D-Eddi.sh" ==> data/consolidated/energy/std/h1d/D/h1d-D.csv <== 2024-10-23,Eddi,1,0.033 2024-10-24,Eddi,1,0.005 2024-10-25,Eddi,1,0 2024-10-26,Eddi,1,0 2024-10-27,Eddi,1,0.001 2024-10-28,Eddi,1,0 2024-10-29,Eddi,1,0 2024-10-30,Eddi,1,0 2024-10-31,Eddi,1,0.001
WAVE improvements
Using the
WAVE Firefox plugin I have made some improvements/fixes to the energy series tables (replacing empty th
items with td
), which may improve accessibility eg for those using screen readers. I do not think there are any bad effects from the changes.
2024-11-10: Additional Energy Data Streams
There is new data to be captured for the reconfigured heat-pump space heating and DHW system. The next set of these, which I will be starting to collect today, will be the Eddi second heater output and the heat pump H4 boundary flow.
- h2
- Eddi heater 2 output total
- h2d
- Eddi heater 2 output diversion only
- h2b
- Eddi heater 2 output boost only
- hph4
- Eddi second current clamp for heat pump H4 boundary consumption
Firstly, I have modified the data sources CSV
:
--- data/consolidated/config_data_sources.csv (revision 59843) +++ data/consolidated/config_data_sources.csv (working copy) @@ -1,7 +1,7 @@ # List of data sources, one per line. Not all sources report all variables. # Expected variables listed after each source. # 'synth' must be last if present. -Eddi,"Myenergi Eddi solar PV diverter",imp,exp,h1d,h1b,h1 +Eddi,"Myenergi Eddi solar PV diverter",imp,exp,h1d,h1b,h1,h2d,h2b,h2,hph4 Enphase,"Enphase AC-coupled battery system",imp,exp,gen,con,battc,battd Loop,"Loop gas meter system",gas SSMPPT,"Morningstar SunSaver SS-MPPT-15L off-grid solar controller",geno
Next I will create updated versions of the existing Eddi data capture script
script/myenergi/eddiDaySummary.sh
to capture the new values
h2d
, h2b
, ectp3
(for hph4
), appending the new values to each daily record so as not to disturb any existing uses of those records.
On the Eddi I tried to configure CT2
to be Monitor
(it is currently None
) but I am unable to, I suspect because no clamp is electrically connected at the moment.
I may wish to do a trial connection and configuration before the heat-pump installation.
Then I created the first of the scripts to create the new variables, editing the new hph4-M-Eddi.sh
monthly script to extract the new (last) field for hph4
:
% svn --parents mkdir script/consolidate/energy/std/hph4/M/Eddi % svn cp script/consolidate/energy/std/imp/M/Eddi/imp-M-Eddi.sh script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh
The diff looks like:
% svn diff script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh --- script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh (revision 59843) +++ script/consolidate/energy/std/hph4/M/Eddi/hph4-M-Eddi.sh (working copy) @@ -1,10 +1,10 @@ #!/bin/sh -# Extract to stdout normalised (archived, not live) eddi grid import records. +# Extract to stdout normalised (archived, not live) eddi hp H4 records. # Returns with status 0 in case of success. # The output record/line format is: # -#YYYY-MM,device,coverage,imp,comment +#YYYY-MM,device,coverage,hph4,comment # List of candidiate archived data files, in order. CANDS="`ls -1 data/eddi/log/20????.daily.csv`" @@ -15,12 +15,16 @@ #2022-03-02,24,0,0,6.047,0.032 #2022-03-03,24,0,0,3.706,0.321 +##Eddi daily stats summary in kWh; partial if h lt 24 +##UTCISOdate,h,h1d,h1b,imp,exp,h2d,h2b,hph4 +#2024-11-10,19,0,0.51,4.949,0.01,0,0,0 + if [ "" = "$CANDS" ]; then echo "ERROR: no 20????.daily.csv files found." 1>&2; exit 1; fi # Create header which ensures non-zero length also. -echo "#YYYY-MM,device,coverage,imp,comment" +echo "#YYYY-MM,device,coverage,hph4,comment" # List input files. for f in $CANDS; do @@ -36,13 +40,13 @@ MM="`echo $f | sed -n -e 's/^.*\/20[0-9][0-9]\([01][0-9]\).daily.csv$/\1/p'`" DATE="$YYYY-$MM" # Coverage is as min 28 days, 24h/d. - cie="`awk < $f -F, '/^20/ {++d;h+=$2;simp+=$5} + cie="`awk < $f -F, '/^20/ {++d;h+=$2;shph4+=$10} END { # Coverage is (imperfectly) day fraction for each day captured, # further reduced pro-rata if fewer than 28 days seen. df = h / (24*d); # Day fraction. if(d < 28) { df *= d/28; } - print df","simp + print df","shph4 }'`" echo "$DATE,$DEVICE,$cie" done
All four new variables are added to config_highbad.csv
, and the new hph4
is added to the generated CSVs, etc:
--- data/consolidated/config_variables.csv (revision 59843) +++ data/consolidated/config_variables.csv (working copy) @@ -16,3 +16,4 @@ h1b,"DHW heater 1 boost from grid" h1,"DHW heater 1 all" heat,"DHW heater 1 all plus gas" +hph4,"heat pump H4 consumption"
It turns out that an entirely-zero dataset (ie where the maximum is zero) had a divide-by-zero bug while normalising HTML bar charts, until just now...
Last is to create the extraction/generation scripts for
h2
h2b
h2d
from their h1
counterparts in much the same way, and then update the relevant synthetic derived variables:
heat,Eddi.h1+Eddi.h2+Eddi.hph4+meter.gas conexDHW,meter.con-Eddi.h1-Eddi.h2
I also encountered and fixed a minor bug in the daily h1d
code.
2024-11-09: Bad Expiry
I noticed that some status graphs were being cached too long in my browser. My attempt to boldly extend cacheing in winter (months 11, 12, 1, 2) seemed to be applying too widely. So I commented out as below:
# Set expiry. # May depend on variables, so should come late in the config! <IfModule mod_expires.c> ExpiresActive On # Crank up the cacheing for selected resources... # Default ~1 day to optimise Cache-Control max-age to 5 digits HPACK. ExpiresDefault "access plus 92222 seconds" # DHD20241109: this conditional seems to be overriding lots of stuff below. #<If "%{TIME_MON} -lt 3 || %{TIME_MON} -gt 10"> # # Winter ~11 days to reduce load a little. # ExpiresDefault "access plus 922222 seconds" #</If> ...specific cases by <Location>...