Earth Notes: On Data Collections and GraphsUpdated 2020-08-05 19:48 GMT.
Some of the data is marked up with HTML5 microdata schema.org/Dataset. The intent is to "start with the basics" as suggested by Google, including 'DataDownload' and 'license' information where possible. (This mark-up is checked with Google's Structured Data Testing Tool.)
See available datasets for others.
This page also references circuit diagrams/schematics.
I've collected much data for our house at 16 Willingham Way (16WW) referred to in the various documents on this site. Much of that data is structured and machine-readable.
The bulk of the data can be found here and is browsable, in simple text formats such as CSV. Common compression techniques have been applied for larger/non-live data sets.
- 16WW bulk energy and related data
- 16WW domestic energy and related data, including time series, Kingston-upon-Thames, London, UK
- domestic, energy
- date created
- date published
- date modified
- temporal coverage
- spatial coverage
- UK centre 51.406696N,-0.288789E elevation 16m
- canonical URL
- this descriptive text with markup
- this dataset is licensed under CC0, ie it is effectively public domain; if you make use of this data, attribution is welcome but not obligatory
Other available datasets (eg shards of this overall 16WW dataset).
Unstructured and Other Data
SAP and Air-Leakage
To set a baseline and understand better what needed fixing, I had SAP, air-leakage and other tests done in early 2009.
Tests/results/certificates by Tophouse 2009-03-30 for us were:
- Air test results
- SAP Energy Performance Certificate
- SAP calculations and inputs
- SAP 2005 worksheet
- RDSAP (Reduced-Data SAP) draft EPC and data input
- NHER results, calculations and data
- NHER draft results
- Building regs checks DER and TER calculations
Grid-Tie PV Generation
2008-02-25: the first phase/round of PV, 1.29kWp (west-facing, 6 x Sanyo HIT 215Wp hybrid crystalline/thinfilm panels with SMA Sunny Boy 1100 inverter), was installed.
2009-02-10: the second phase was installed, taking the system to 3.87kWp; same panels and inverters for each of the two new east-facing strings as for the first phase.
2010-04-26: the third phase (west-facing, same panels, SMA Sunny Boy 1200 inverter) of 1.29kWp was installed, taking the whole system to 5.16kWp. New tariff and generation meter for this last part.
Rob Ferber of ElectronVault wrote to me looking for intra-day generation data (with 10 minute sampling intervals). I had at that time exposed very little because of data-cleanup issues. (Such as bizarre time offsets in the Sunny Beam apparently attempting to correct to German civil time even though I keep it on UTC, serial numbers, corrupt data, slight drift of Sunny Beam from true UTC, etc.)
I capture these intraday samples manually after dusk about twice per month. Sometimes also I capture data on particularly 'interesting' days, of late using the SunnyBeamTool c/o Michaël Peeters on my SheevaPlug (and now, as of 2014, Raspberry Pi B+) ARM Linux system. I didn't do this continuously because I have not been able to stop the Sunny Beam apparently drawing considerable power via the USB port, and because the SheevaPlug didn't have a 'spare' USB port: I manually disconnected the USB I/O board temporarily to connect the Sunny Beam. (As of 2014-07 experimentally I have the Sunny Beam plugged in permanently, and take 5-minute samples during daylight hours.)
Rob produced lovely tweaked graphs from my raw data up to September 2012. Thanks, Rob: great job!
Prior to Rob's analysis I had made public but not announced these intraday samples from SMA's Windows application, with no explicit time correction: 2008-03-24 2008-04-01 2008-04-09 2008-04-17 2008-05-08 2008-09-13 2008-09-14 2008-09-19 2008-10-11 2008-12-13 2009-03-15 2009-04-08 2009-05-11 2009-05-24 2009-06-29
The entire raw data set is now available, including new samples as I take them, though with many caveats (beware timestamp wobbles in particular).
Here is an extract of the numbers from the PVGIS estimate not knowing anything about our local trees, etc!
Solar radiation database used: PVGIS-classic
Nominal power of the PV system: 5.2 kW (crystalline silicon)
Estimated losses due to temperature: 7.3% (using local ambient temperature)
Estimated loss due to angular reflectance effects: 4.1%
Other losses (cables, inverter etc.): 14.0%
Combined PV system losses: 23.5%
Fixed system: inclination=23°, orientation=-90° Month Ed Em Hd Hm Jan 2.78 86.0 0.73 22.6 Feb 5.36 150 1.34 37.4 Mar 8.83 274 2.16 66.8 Apr 14.40 433 3.57 107 May 17.80 552 4.51 140 Jun 18.20 545 4.65 140 Jul 18.20 564 4.69 145 Aug 15.60 483 4.02 125 Sep 10.70 322 2.72 81.7 Oct 6.57 204 1.67 51.7 Nov 3.43 103 0.89 26.8 Dec 1.94 60.1 0.52 16.2 Yearly average 10.3 315 2.63 80.0 Total for year 3780 960
Ed: Average daily electricity production from the given system (kWh)
Em: Average monthly electricity production from the given system (kWh)
Hd: Average daily sum of global irradiation per square meter received by the modules of the given system (kWh/m2)
Hm: Average sum of global irradiation per square meter received by the modules of the given system (kWh/m2)
Note also that adding just 10% (~500Wp) vertically mounted south-facing PV would add about 1/3rd to mid-winter generation from the main array (ie ~1.6kWh/d actual from the roof and ~0.5kWh/d from the new array predicted.)
Nominal power of the PV system: 0.5 kW (crystalline silicon)
Estimated losses due to temperature and low irradiance: 6.1% (using local ambient temperature)
Estimated loss due to angular reflectance effects: 4.3%
Other losses (cables, inverter etc.): 14.0%
Combined PV system losses: 22.7%
Fixed system: inclination=90°, orientation=0° Month Ed Em Hd Hm Jan 0.58 18.0 1.40 43.5 Feb 0.79 22.2 1.94 54.2 Mar 1.16 36.0 2.89 89.7 Apr 1.25 37.6 3.25 97.4 May 1.06 32.9 2.87 89.1 Jun 1.01 30.4 2.81 84.2 Jul 1.04 32.2 2.88 89.2 Aug 1.06 32.9 2.87 88.9 Sep 1.17 35.0 3.05 91.5 Oct 0.96 29.8 2.44 75.5 Nov 0.73 22.0 1.80 53.9 Dec 0.53 16.4 1.27 39.3 Yearly average 0.946 28.8 2.46 74.7 Total for year 345 896
Off-grid Battery Voltage/State
(See the gnuplot commands for live voltage plotting.)
Meter ReadingsMonthly meter readings and comments.
Gross Electricity Consumption
Footprint From Electricity and Gas
Heating Degree Day Data
Partial monthly summary of local HDD12 (base temperature of 12°C) data suitable for gnuplot to digest.
See annual data too.
Water Mains Inlet Temperature
See 16WW water mains inlet temperatures for an idea of how much extra "lift" is required by the DHW system in winter, for example.
Also in the context of thermal discomfort.
Local Temperature (etc) Monitoring
See a graph of local internal temperature at 16WW collected from an OpenTRV unit tethered to the Web server.
For the 16WW OpenTRV devices see:
- ambient temperature (larger).
- sensor battery voltage and off-grid main battery temperature.
- relative humidity.
- temperature target (not live).
- temperature setback (not live).
- valve % open (not live).
- valve % cumulative movement (not live).
- ambient light levels and deltas (not live).
and also see a 2016-09-22 snapshot.
See the OpenTRV/16WW public data set. Note that data formats have evolved over time, but are all plain text, in various common compressed archive formats for speed of download, etc.
iButton Temperature Monitoring
The take-home message from the analysis is that it believes that we have our living room at the average UK 19°C and that our living room temperature varies between about 16°C and 21°C generally in a sawtooth pattern. (Note that 18°C in the morning before school seems to suffice.)
Note that from early evening 2012-12-19 I lowered the radiator flow temperature from about 60°C to about 45°C, independently of this, to see if the i30 PRC/eTRV could work with a lower heating rate. The reduced flow temperature seems to help reduce overshoot both in the target bedroom and in the living room, though the motivation was to try to test at typical heat-pump temperatures.
Note that we don't leave our house thermostat in the living room alone; we keep it lowish unless we feel cold and/or we want to force the heating on for the children's bedrooms before they go up. The latter action would be unnecessary if we had zoning with each bedroom able call for heat independently and those wasteful and soporific peaks >19°C could probably be trimmed. In the interim we should probably make sure that we get the thermostat down to ~18°C as the children go to bed to minimise overshoot.
A friend has generously provided his raw CSV data and chart too (from somewhat east of London), for comparison. In his case there is evidence of much tighter temperature control, at a similar mean level to ours, even with lower external temperatures.
Temperature and Relative Humidity Data from Thermal Discomfort Study
At the end of 2012 I participated in a UCL Energy Institute/LoLo study on responses to thermal discomfort, in return for access to the raw data collected by them on temperature and relative humidity at 16WW, which overlaps with some of my iButton mission 4 data collection.
Three draft plots (based on the sensors at 4 heights in each room):
- temperature and RH% in each room vs outside
- RH% against height in each room
- temperature against height in each room
were made available first, right at the end of the year. (The visible codes are: C04 Bedroom, C05 Living Room, C06 Kitchen.)
See the underlying raw data (note the room codes above on the filenames) in Mac .csv file format (with lines CR terminated, not CRLF or LF), c/o Stephanie Gauthier (then) of UCL whose project this was for, ZIPped.
Air Quality Egg (AQE)
I did not have much joy attempting to measure "air quality" with the AQE Air Quality Egg. These things are hard to do well and cheap at the same time.
- Off-grid PV system schematic (Ki-CAD) including various snapshots image/other formats.
- (archive) 2011-07-25: off-grid (lead-acid and LiFePO4/LFP) power system circuit diagram .SHX, PDF.
All historic logging data from the k8055, mainly off-grid numbers, is available under
data/k8055/YYYY-MM.gz, and a log of system changes is at the end of
the historic stats.
Lead-Acid Voltage/SoC Chart
Here is Solar John's SOC chart reproduced with permission, along with his note: "it provides only a rough estimate of the actual SOC":
|% Charge||Charging||At Rest||Discharging|
See also, for example Battery State-Of-Charge Chart For Voltage & Specific Gravity and Batteries Maintenance 101.
Lead-Acid Battery State of Charge vs Voltage (Richard Perez, Home Power #36, Aug/Sep 1993) notes that "At 32°F (0°C), the effect of temperature becomes pronounced enough to distinctly change not only the battery voltage vs. SOC profile, but also its useful Ampere-hour capacity. The discharge voltage curves may be depressed by as much as 0.5VDC from those shown on the graph. Charge voltages will be elevated by as much as 0.5VDC for a cold 12 Volt lead-acid battery." (This chimes with, for example, Victron's -24mV/°C threshold adjustment from a 20°C baseline.)
FactoidsA few random facts that are useful for rules of thumb, etc:
- A person exhales ~1kg/d of CO2 (~0.5m^3).
- A continuous 1W UK mains electricity draw costs ~1£/y circa 2012.
- Our portable dehumidifier (2011 vintage) consumes ~1kWh to remove 1kg of water from the air (and is a CoP>1 heater). It may have been managing to extract a peak of ~2l/2kg per hour, eg helping to dry out a recently-plastered room (2015-12-09).
- Q10=2 applies to max absolute humidity vs air temperature (ie equilibrium absolute humidity approximately doubles (and so %RH halves for a fixed g/m^3 of water) for each 10K (or °C) increase in temperature).
- Heat capacity of water ~116kWh/t for a delta-T of 100K (~10t/MWh), ~90kWh/t for a delta-T of 80K, ~60kWh/t for a delta-T of 50K (eg from 90°C down to 40°C in a DHW tank), or 2.3kWh for 200l DHW tank falling from 55°C to 45°C or mains at 10°C preheated to 20°C in winter (10K) and note tank losses ~1kWh/d ballpark. Compare with well under 30kWh/t for lead-acid batteries (16kWh/t for AGM @ typical 50% DoD), 100kWh--200kWh/t for Lithium-chemistry batteries.
- Approx lighting levels for office and better lit parts of home 400lux, dimly lit living areas of home 80lux.
- Fieldlines thread: ... the consensus was that a good cyclist can achieve 150 watts hour after hour, a professional cyclist can do 200 watts all day, and at Olympic level they can mange 250 watts for an hour or two but still keep up the average 200 watts. A 'normal' adult can maybe sustain 100W for a reasonable time, and note that resting heat output is also 100W, presumably leading to nearly 200W total output.
- Hours of sun per year USA vs Europe: SW US 3500+, UK south coast 1800--2600, north Scotland <1200.
- "Ordinary" radiator output into 20°C room relative to 70°C flow: 30°C/13.5%, 40°C/32.1%, 50°C/53.0%, 60°C/75.8%.
GB Grid Intensity
Code used to collect and compute GB grid intensity numbers, live and historical, published under an BSD-style free/open licence on GitHub: reutils.
20110124: archival initial snapshot of Eclipse project:
- GZIPped tar image (348405 bytes): code/reutils-1.0.0-SNAPSHOT.tgz
- ZIP file (366206 bytes): code/reutils-1.0.0-SNAPSHOT.zip
Just for giggles, note that there's nothing secret about the makefile used to build this site, though note that for tidiness not all of the supporting scripts are in public view. If anyone really cares I can make some or all of them visible too.