Second Test Report (V1) Further Notes Addendum ================== DHD20170329 addendum to report DHD20170221 DOCUMENT HISTORY ADDENDUM CREATED: "DRAFT" Damon Hart-Davis 20170329 REVIEWED: Mark Hill ISSUED 20170406 Static Factors -------------- The black box approach was chosen for the protocol for the following reasons: * The end user will care about energy (thus money and carbon) saved, not the exact mechanism involved, and that is what the protocol seeks to directly measure, in terms of heating fuel savings (normalised by heating degree days). * To allow products other than OpenTRV TRV1 (eg from third parties), and future versions of OpenTRV, to use this methodology. * To minimise the number of calibration chains required, in particular keeping the energy-saving equipment itself which is designed for consumer use, out of those chains. Under this protocol only the supply meter and weather data need be traceable. * To acknowledge the fact that homes are not controlled environments and that energy efficiency mechanisms must work in that uncontrolled environment. In regard to concerns about effects on the results of occupancy, holidays, homes as uncontrolled environments etc: * In this test run the fact that TRV1 uses occupancy as a key driver is not in itself directly relevant by the protocol's design, even though TRV1 is intended to automatically deal well (automatically save extra energy) with longer-than-usual vacancy, eg holiday absences. * More specifically, because this test gathered data from multiple rounds of installation with initial control periods throughout the heating seasons, as discussed in the addendum in relation to gas calorific value, it is unlikely that there has been significant unexpected bias by holidays. In future test runs this variation on control period/sampling may be made more systematic, or bias might be otherwise explicitly tested for to potentially allow correction. Data Filtering -------------- For this test an initial 58 homes (social housing) across north-west London were selected. Approximiately half of those turned out to have credit (pre-pay) meters fitted which prevented use of our selected method to track heating fuel usage which requires a mechanical gas meter. Difficulties with data gathered via our equipment (used purely to establish control/on-control periods) and via the "Loop" gas meter trackers, resulted in large numbers of homes being eliminated for having no matching data or too little to reliably perform the specified linear regression on. (To use in the calculations there must be simultaneously HDD data, fuel use data, and knowledge of the valves' control/normal state.) Also, homes where the heating use/control appeared to be particularly poor (as evidenced by the R^2 value) were removed from the data set. The table below summarises the remaining steps in the data funnel leading to the small number of homes in the final result. The 'out part' corresponds to the output filename (as seen in the test report) and thus filtering stage. Note in particular that in line with the protocol, one home that reported using secondary (electric) heating but that otherwise survived all the data quality tests/filters was manually removed from the data set. That home's data would have implied OpenTRV's efficacy as higher than really the case. See out part 30 below. Please refer the the body of the test plan for more information on the filters applied. Ultimately the code base including the filters are published under an open source licence on GitHub.com for inspection also. --------+----+-------------------------------------------------- Cohort | | Content of surviving homes cohort Size |Out | and (homes) |Part| key reasons for reduction in cohort size --------+----+-------------------------------------------------- 58 | | Initial full trial cohort. --------+----+-------------------------------------------------- 33 | 10 | Essentially all homes with some Loop | | (gas consumption) data. | | Nearly half the homes had prepayment meters | | with which Loop could not be used. --------+----+-------------------------------------------------- 17 | 20 | This cohort has sufficient days' data | | and exceeds an R^2 floor; | | a combination of problems with some fuel meters | | (such as not showing any consumption at all) | | and indication of very poor heating systems or | | poor use of those heating systems (very low R^2) | | excluded about half of the previous cohort. --------+----+-------------------------------------------------- 8 | 30 | This is the first step using data from the valves | | and the surviving cohort had usable valve logs; | | severe reliability issues in our mobile data | | backhaul eliminated about half the homes here. | | Later revisions of the firmware improved | | reliability for the 2016/2017 heating season. | | | | Also home "N" is manually excluded from this | | point onwards (by removing the Loop/valve link) | | because it reported using secondary heating, | | as required by the protocol. Without this, | | overall efficacy/savings would be reported as | | about 10% higher. --------+----+-------------------------------------------------- 4 | 31 | The surviving cohort has sufficient data | | in both control and normal operation modes | | that matches with fuel consumption data. --------+----+-------------------------------------------------- Visualisation and Data Exploration ---------------------------------- See http://www.earth.org.uk/img/OpenTRV/20132014-savings-chart.png for an example of how an earlier incarnation of OpenTRV (pre TRV1) caused heating fuel slope (kWh/HDD) to drop while installed, as evidenced by the fuel demand dropping away from the HDD curve. Looking in more detail at household "L" that passed all the filtering to be included in the final result set, and which appears 'typical' in many regards (reasonable R^2 and a typical TRV count of three), and has a reasonable number of control and non-control (ie normal) days (16 finally used) in its data set: * From 31_segmentedStatsOut.csv: "L",0.78175366,14.245473,0.3992715,16,1.195788 ie slope of 0.78kWh/HDD (and R^2 of 0.4) and efficacy of 1.2 (ie energy saving of about 20%) when in normal mode, vs control mode. * From 30_presegmentedStatsOut.csv: L,18,32,0 ie 18 control days and 32 normal (non-control) days available from valve log data. * From 20_basicFilteredStatsOut.csv: "L",1.663113,5.4242167,0.8635711,270, ie a slope of 1.66kWh/HDD (and R^2 of 0.86) overall with 270 days of gas meter (heating fuel consumption) data available. Hardware for the latest version of the firmware (RC3b) was installed 2017/12/29 and the initial control period ran to 2017/01/14 inclusive. Normal operation ran from 2017/01/15 up to 2017/02/15 inclusive, the end of the control/normal data available when this test run computation was performed. Note that the last available Loop (ie heating fuel) data at the time of this test run covered until the end of January, and because it did not wrap over midnight the final day (the 31st) was not used in the calculations. So the per-day data available for analysis for "L" is: date,status,kWh,HDD 20161220,Disabled,20.119873,10.2 20161229,Disabled,25.77002,13.2 20161230,Disabled,22.630127,14.1 20161231,Disabled,20.75,9.5 20170101,Disabled,19.169922,9.4 20170102,Disabled,22.320068,12.7 20170103,Disabled,23.26001,13.6 20170104,Disabled,27.97998,10.8 20170105,Disabled,28.29004,14.0 20170106,Disabled,24.509766,12.9 20170107,Disabled,19.810059,6.2 20170108,Disabled,17.28003,6.8 20170109,Disabled,17.610107,7.7 20170110,Disabled,20.429932,9.1 20170111,Disabled,22.310059,7.1 20170112,Disabled,24.52002,10.9 20170113,Disabled,26.409912,12.6 20170114,Disabled,26.70996,13.0 20170115,Enabled,23.890137,9.8 20170116,Enabled,22.629883,10.0 20170117,Enabled,23.580078,12.9 20170118,Enabled,26.71997,14.5 20170119,Enabled,27.340088,13.8 20170120,Enabled,22.319824,13.5 20170121,Enabled,26.090088,15.2 20170122,Enabled,29.22998,15.4 20170123,Enabled,27.659912,15.0 20170124,Enabled,21.380127,12.5 20170125,Enabled,23.570068,14.5 20170126,Enabled,27.659912,16.1 20170127,Enabled,17.29004,12.1 20170128,Enabled,24.52002,8.5 20170129,Enabled,22.319824,9.3 20170130,Enabled,18.860107,7.9 20170131,Enabled,null,8.2 20170201,Enabled,null,5.2 20170202,Enabled,null,5.2 20170203,Enabled,null,7.4 20170204,Enabled,null,10.4 20170205,Enabled,null,11.0 20170206,Enabled,null,11.5 20170207,Enabled,null,8.6 20170208,Enabled,null,11.1 20170209,Enabled,null,13.5 20170210,Enabled,null,14.0 20170211,Enabled,null,13.8 20170212,Enabled,null,12.3 20170213,Enabled,null,9.5 20170214,Enabled,null,8.4 20170215,Enabled,null,6.7 Note the lack of kWh data from 20170131 inclusive, and note the spurious initial value from system provisioning on the 20161220, see for more discussion in the note "20170224DataFilteringNote.txt". In the speadsheet: 20170221Test2ReportV1notes.addendum-further-notes.s1.ods the Java calculations are replicated, eg using SLOPE() to compute the kWh/HDD slope by linear regression. (With the 20161220 value omitted 'Ideal' the computed efficacy is 1.181970 rather than 1.195788 'Actual'.) House "S" shows a larger nominal OpenTRV efficacy (1.58) and had 52 non-control days total. Key data, including two control periods are: date,status,kWh,HDD 20161020,Disabled,44.20996,4.6 20161021,Disabled,41.97998,5.2 20161022,Disabled,69.31006,6.3 20161023,Disabled,54.54004,6.0 20161024,Disabled,40.42993,4.4 20161025,Disabled,64.86011,4.5 20161026,Disabled,33.659912,3.4 20161027,Disabled,40.76001,4.0 20161028,Disabled,37.32007,1.9 20161029,Disabled,41.32007,2.3 20161030,Disabled,40.089844,5.2 20161031,Disabled,21.77002,3.4 20161101,Disabled,50.090088,6.5 20161102,Disabled,59.54004,9.3 20161103,Disabled,43.649902,8.9 20161104,Disabled,55.19995,7.5 20161105,Disabled,64.09009,10.2 20161106,Disabled,112.3999,11.1 20161107,Enabled,65.63989,9.8 20161108,Enabled,125.510254,12.4 20161109,Enabled,86.85986,9.2 20161110,Enabled,76.08008,8.9 20161111,Enabled,91.18994,9.2 20161113,Enabled,80.52002,7.7 20161114,Enabled,61.200195,5.7 20161115,Enabled,53.54004,1.8 20161116,Enabled,35.97998,3.6 20161117,Enabled,62.649902,6.7 20161118,Enabled,35.759766,10.6 20161119,Enabled,111.63037,10.8 20161120,Enabled,89.52002,8.7 20161121,Enabled,56.31006,6.5 20161122,Enabled,81.75,5.9 20161123,Enabled,100.95996,7.8 20161124,Enabled,83.07959,6.1 20161125,Enabled,31.660156,8.1 20161126,Enabled,129.28027,10.1 20161127,Enabled,55.759766,8.6 20161128,Enabled,84.08008,9.4 20161129,Enabled,94.08008,13.8 20161130,Enabled,82.08008,15.3 20161201,Enabled,134.27979,13.7 20161202,Enabled,69.97998,9.5 20161203,Enabled,105.95996,9.8 20161204,Enabled,133.28027,12.3 20161205,Enabled,85.07959,12.5 20161206,Enabled,127.180176,8.8 20161207,Enabled,43.089844,4.8 20161208,Enabled,14.0,3.9 20161209,Enabled,85.07031,3.5 20161210,Enabled,54.429688,4.0 20161211,Enabled,31.100098,9.1 20161212,Enabled,57.97998,8.2 20161213,Enabled,59.75,4.9 20161214,Enabled,46.20996,6.3 20161215,Enabled,67.41992,6.8 20161216,Enabled,91.08008,6.0 20161217,Enabled,84.740234,9.2 20161218,Enabled,105.62988,8.8 20161219,Enabled,69.200195,9.9 20161220,Enabled,84.739746,10.2 20161221,Enabled,53.54004,6.2 20161222,Enabled,96.740234,9.3 20161223,Enabled,97.84961,6.9 20161224,Enabled,71.97998,7.1 20161225,Enabled,91.8501,3.7 20161226,Enabled,70.09033,8.6 20161227,Enabled,97.069824,13.2 20161228,Enabled,109.95996,13.2 20170111,Disabled,60.30957,7.1 20170112,Disabled,101.07031,10.9 20170113,Disabled,70.87012,12.6 20170114,Disabled,188.82031,13.0 20170115,Disabled,77.299805,9.8 20170116,Disabled,79.299805,10.0 20170117,Disabled,93.08008,12.9 20170118,Disabled,121.62012,14.5 20170119,Disabled,107.07031,13.8 20170120,Disabled,116.62988,13.5 20170121,Disabled,132.2793,15.2 20170122,Disabled,117.850586,15.4 20170123,Disabled,139.16992,15.0 20170124,Disabled,95.62988,12.5 20170125,Disabled,125.62012,14.5 20170126,Disabled,119.62012,16.1 20170127,Disabled,140.05957,12.1 In the speadsheet: 20170221Test2ReportV1notes.addendum-further-notes.s2.ods and the graph: 20170221Test2ReportV1notes.addendum-further-notes.s2.png it is evident that the bar representing the slope of the middle (normal-mode) time period is lower than for the control periods either side. Note that efficacy > 1 is true (lower kWh/HDD slope in non-control periods) even slicing and dicing differently than the protocol suggests, in this case chopping into large contigious blocks rather than lumping all of the control/normal days into two blocks, ie the test is reasonably robust.