Second Test Report (V1) ================== DHD20170221 DOCUMENT HISTORY CREATED: "DRAFT" Damon Hart-Davis 20170221 REVIEWED: Mark Hill 20170221 ISSUED "V1": update 20170221 with corrections after review MH20170221 For device: OpenTRV "TRV1" Trial/data set name: ES1 201603--201701 (second data set) Household description: single family households, social housing Household location (weather station): north London UK (EGLL) Operation restrictions conditions from test plan met: yes Analytics code release: https://github.com/opentrv/OpenTRV-Java/releases/tag/ETV-20170220-RC2 Valve code releases: TRV1.5-rv1-gE-20160919, 20161115-RC2b, 20161211-RC3b https://github.com/opentrv/OpenTRV-Arduino-V0p2 https://github.com/opentrv/OTRadioLink https://github.com/opentrv/OTAESGCM Calibration traceability of data inputs for primary verified measure: 1) For a verified calculation sample from METAR data to Heating Degree Days for EGLL weather station by degreedays.net (and thus degreedays.net HDD traceability) please see: http://www.earth.org.uk/OpenTRV/stds/ETV/20160518-sample-degree-day-calculation-degreedays-dot-net-for-EGLL.xlsx (Worked calculations showing feed-through from METAR data for EGLL on three separate dates in different parts of the heating season: 2015/10/01, 2016/01/01, 2016/03/01.) 2) For a verified sample of tracking of Loop gas meter reading to gas supply meter reading (and thus Loop energy meter reading calibration traceability via the supply meter calibration chain) please see: http://www.earth.org.uk/OpenTRV/stds/ETV/20160819LoopErrorSample.txt (Sample showed an in-situ measurement error of <0.1% over 7 months.) Also see: http://www.earth.org.uk/OpenTRV/stds/ETV/20170215TraceabilityNotes.txt Control vs Normal ----------------- For the TRV1 code release, each device acts as its own control by having an initial period after being installed of up to 254 days (14--21 typical) where the majority of the energy-saving features (the automatic temperature setback) are disabled. Some minor features remain enabled, such as the widened temperature regulation deadband around the user-set temperature when the room is dark, because it is primarily there to reduce disturbance eg of sleep or TV watching. (The wide deadband will potentially continue to improve energy efficiency by letting temperatures drift lower before calling for heat, and by calling for heat in slightly longer more efficient bursts than usual.) Results and Notes ----------------- Result of full program run on 2016/03--2017/01 Loop data, EGLL HDD data, plus control/normal stats segmentation using valve log data is shown below. Note that due to problems with the trial, primarily the first below, much less data than desirable has been collected still: 1) A number of issues with the radio relaying of stats over the cellular (2G) network meant that for many units data stopped being relayed after a short while, and attempting to restart the radio relay units was disconcerting some of the householders, so was not pursued. In part this seems to have been bug in the 2G card itself. Newer firmware in our radio relay units has mitigated this issue, but other issues such as failure of the Loop meter have narrowed the available data set. 2) A longer initial control period (>= 14 days rather than 8) should have been chosen and the valve firmware has now been amended to allow this. Later releases have this longer control. 3) The trial would ideally have started earlier in the heating season to allow more data to be gathered before the heating season ended. 4) More households than expected of the original 60 had pre-pay meters for which it was not possible to monitor gas usage with Loop. This shows the discarding of poor data and the reduction to a small final data set of three households of the original 60 (of which ~33 had any data at all). 4 households have adequate data but one of them reports using some secondary heating on the coldest days as their central heating was not proving adequate, and so this data (apparently showing very high efficacy for OpenTRV) was excluded. More detail is shown below. All these results have been manually anonymised. Note that in the results below for the 'significant' households that passed filtering the following short names have been used: T Household with a set of 3 OpenTRV valves. F Household with a full set of 4 OpenTRV valves. S Household with a single OpenTRV radiator. N Household reporting some use of secondary heating, ultimately excluded. L Household with a set of 3 OpenTRV valves with all data from RC3b release. Data Run Output With Sensitivity Analysis ----------------------------------------- Initial runs were made with no households excluded other than by the built-in filtering for poor data. This contains matched data up to 2017/01/11; the final data run contains slightly more. In particular household "N" was left in for this analysis; it shows abnormally high apparent efficiency gains possibly because the primary (monitored) central heating was satisfying a significantly reduced fraction of total heat demand. The runs were at the target 15.5C baseline and 1C either side, ie 14.5C and 16.5C, to look for sensitivity in reported results to (small) changes in baseline. tail ETV-prepared-data-out-14.5 ==> ETV-prepared-data-out/31_segmentedStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "T",4.2068853,26.963495,0.0996482,12,1.7867157 "F",3.6974223,67.94995,0.35522425,13,0.7251288 "S",4.963076,42.084282,0.2865219,51,1.1699458 "N",2.8377986,66.48517,0.20239213,8,2.7224467 ==> ETV-prepared-data-out/90_multihouseholdSummaryStatsOut.csv <== allHouseholdsCount,finalHouseholdsCount,normalDayCount,RsqMean,RsqSD,SlopeMean,SlopeSD,EfficacyMean,EfficacySD 33,4,84,0.23594663,0.095509395,3.9262955,0.7731192,1.6010593,0.7491814 tail ETV-prepared-data-out-15.5 ==> ETV-prepared-data-out/31_segmentedStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "T",4.2458296,22.638836,0.10528531,12,1.7546113 "F",3.6974223,64.25253,0.35522422,13,0.7251288 "S",4.9495926,37.263317,0.28598693,51,1.1363724 "N",2.8377988,63.647366,0.20239218,8,2.671603 ==> ETV-prepared-data-out/90_multihouseholdSummaryStatsOut.csv <== allHouseholdsCount,finalHouseholdsCount,normalDayCount,RsqMean,RsqSD,SlopeMean,SlopeSD,EfficacyMean,EfficacySD 33,4,84,0.23722216,0.09343842,3.9326608,0.77238023,1.5719289,0.73304766 tail ETV-prepared-data-out-16.5 ==> ETV-prepared-data-out/31_segmentedStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "T",4.2455077,18.395458,0.10579917,12,1.7478405 "F",3.6974223,60.55511,0.35522422,13,0.7251288 "S",4.9495926,32.313725,0.28598693,51,1.1113709 "N",2.837799,60.809563,0.20239219,8,2.6248496 ==> ETV-prepared-data-out/90_multihouseholdSummaryStatsOut.csv <== allHouseholdsCount,finalHouseholdsCount,normalDayCount,RsqMean,RsqSD,SlopeMean,SlopeSD,EfficacyMean,EfficacySD 33,4,84,0.23735063,0.09325711,3.9325805,0.7723475,1.5522975,0.7188931 The most eye-catching change vs base temperature change is "baseload", though that is not significant to the test's calculations and never shows implausible (such as negative) values. The R^2 value is relatively stable vs base temperature, ie to 2SF. The slope (kWh/HDD) value is stable vs base temperature, ie to 2SF. The efficacy value is a little less stable, showing evidence of a slight negative relationship with base temperature, though ~2%/C in 3 cases (and zero in the forth) for small base/bias errors, much smaller than the nominal ~8%/C available in energy savings, and with external temperatures ranging more than 20C during the study interval. There is certainly scope for characterising this better over a larger data set, and possibly allowing individual household base temperatures to be computed where the data set is large enough, but the fixed base temperature seems unlikely to be a significant source of error for this data. Data Run Output Explanation --------------------------- 1) 10_basicStatsOut.csv This section lists all the households for which there is any energy consumption data after performing an heating efficiency (kWh/HDD) run over all available data without regard to the OpenTRV valves being in control or normal mode. 2) 20_basicFilteredStatsOut.csv This section lists those households that survive this initial filter for inadequate data or very poor R^2 suggesting very poor heating management. There is no visibility of control/normal operation at this stage. The key filters are: a) Being able to compute the HDD metrics at all which implies a very minimal number of data points and a non-flat energy consumption. b) Sufficient days' coverage: a minimum of 12 (twelve) to nominally allow 6 (six) for control mode with energy-saving features largely or entirely disabled, and 6 (six) for normal mode. c) A floor R^2 to eliminate very poor data: 0.15 for per-day data. This is much lower than would be expected for heating systems with even reasonable control and is intended only to remove extreme noisy outliers. These are enforced in the Java code by predicate: uk.org.opentrv.ETV.filter.CommonSimpleResultFilters.goodDailyDataResults 3) 30_presegmentedStatsOut.csv This incorporates the status segmentation data from the valve logs to split each household's data into days that are control (energy-saving features disabled), normal (enabled) or "don't use". The "don't use" category is for example for days where no majority of valves is in the same enabled/disabled state. The data shown in this output section downs the number of each type of day by household. This status segmentation data can be provided manually, but for TRV1 the basic status for each valve as class: OTLogActivityParse.ValveLogParseResult is extracted by: OTLogActivityParse.parseTRV1ValveLog() which notes which days (always local timezone midnight-to-midnight) have any log data at all, in which of those there is evidence of calling for heat (from the valve being open), and in which of those the automated setback value is being reported, and where is is non-zero. The last of these is taken to indicate the energy saving features enabled. (For future code an explicit stats value may be sent with the enabled/disabled state to remove ambiguity.) In the routine: StatusSegmentation.segmentActivity() combines all valve log data above to produce the result for a single household. "This can use days where any device in the household is calling for heat AND reporting its energy saving status (enabled or disabled). Within those days, any that are marked as having energy saving features disabled for the majority of devices can be regarded as control days; where the majority have energy saving features enabled are normal days." This may understate valve efficacy by eliminating days from the normal set where noting needed to call for heat. 4) 31_segmentedStatsOut.csv This shows the per-house analysis of all those that pass further screening, in particular that they have the minimum acceptable number of both control daya and normal days, separately, as enforced by predicate: uk.org.opentrv.ETV.filter.CommonSimpleResultFilters.enoughControlAndNormal The data shows the 'normal' days energy efficiency (kWh/HDD slope), baseload (kWh/day), R^2, 'normal mode' days. The final field is the efficacy with the kWh/HDD of control days divided by that for normal days, so values > 1 indicate actual energy savings (ie improved heating efficiency) with the energy-saving features turned on. 5) 90_multihouseholdSummaryStatsOut.csv This is the final result of the analysis, across all households remaining at the previous stage, and includes the count of households in the first and final stages, and 'normal mode' days aggregate, and mean and (population) standard-deviation for each of R^2 and slope (kWh/HDD) in normal mode, and for OpenTRV TRV1 efficacy. Final Data Run -------------- The results of running the analysis on the data set are below. The data sets fully overlap from 2016/03 to the end of 2017/01. (The gas kWh data runs until 2017/01/31, the other data into 2017/02.) The household numbers are manually anonymised. This set excludes house N beyond the initial "10_"+"20_" qualification steps per protocol because household N had reported some use of secondary heating. ==> ETV-prepared-data-out/10_basicStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "T",6.396126,11.10129,0.7594415,336, "0002",0.0,0.0,NaN,336, "0003",0.0,0.0,NaN,2, "0004",2.0317907,-1.679537,0.1455493,122, "0005",0.0,0.0,NaN,145, "0006",-1.8764359E-4,0.0047127097,6.7323225E-4,315, "0007",5.7441745,1.9635121,0.59602344,293, "0008",7.1308983E-4,0.006406744,7.2655326E-4,313, "0009",8.068876,5.0032454,0.7142549,157, "0010",6.8730965,0.7575834,0.6485401,72, "F",6.4864955,23.57184,0.65438306,303, "0011",0.0,0.0,NaN,32, "0012",1.5977722,8.093293,0.37530658,76, "0013",1.7509801,15.115561,0.17595467,49, "5987" "0014",5.502946,4.2327795,0.80286276,298, "0015",-0.041431993,1.2646971,0.021207383,298, "0016",0.2521144,12.532636,0.011345407,94, "0017",4.6236567,5.5930495,0.7603363,287, "0018",0.97071934,37.049965,0.011139902,4, "0019",3.8158543,9.227616,0.54065645,178, "0020",3.848014,3.2572298,0.7604772,126, "0021",0.016483473,0.07369595,0.022475852,68, "0022",1.463423,5.812638,0.22673942,277, "S",7.9222536,9.379069,0.81110597,277, "0023",-0.254681,1.1785526,0.14781575,125, "0024",0.23994349,0.52968127,0.10490016,89, "L",1.663113,5.4242167,0.8635711,270, "0025",0.0,0.0,NaN,220, "0026",1.7649931,4.9175944,0.70401925,215, "N",7.5541997,16.884703,0.8164522,255, "0028",2.5040345,13.56231,0.13277346,255, "0029",2.5896924,2.9834023,0.54983693,131, ==> ETV-prepared-data-out/20_basicFilteredStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "T",6.396126,11.10129,0.7594415,336, "0007",5.7441745,1.9635121,0.59602344,293, "0009",8.068876,5.0032454,0.7142549,157, "0010",6.8730965,0.7575834,0.6485401,72, "F",6.4864955,23.57184,0.65438306,303, "0012",1.5977722,8.093293,0.37530658,76, "0013",1.7509801,15.115561,0.17595467,49, "0014",5.502946,4.2327795,0.80286276,298, "0017",4.6236567,5.5930495,0.7603363,287, "0019",3.8158543,9.227616,0.54065645,178, "0020",3.848014,3.2572298,0.7604772,126, "0022",1.463423,5.812638,0.22673942,277, "S",7.9222536,9.379069,0.81110597,277, "L",1.663113,5.4242167,0.8635711,270, "0026",1.7649931,4.9175944,0.70401925,215, "N",7.5541997,16.884703,0.8164522,255, "0029",2.5896924,2.9834023,0.54983693,131, ==> ETV-prepared-data-out/30_presegmentedStatsOut.csv <== houseID,controlDays,normalDays,dontuseDays T,32,12,0 0007,9,2,0 F,8,22,2 0014,1,30,0 0017,0,0,0 S,42,52,0 L,18,32,0 0026,3,2,0 ==> ETV-prepared-data-out/31_segmentedStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "T",4.2458296,22.638836,0.10528531,12,1.7546113 "F",3.6974223,64.25253,0.35522422,13,0.7251288 "S",4.951003,37.351414,0.2859154,52,1.5804358 "L",0.78175366,14.245473,0.3992715,16,1.195788 ==> ETV-prepared-data-out/90_multihouseholdSummaryStatsOut.csv <== allHouseholdsCount,finalHouseholdsCount,normalDayCount,RsqMean,RsqSD,SlopeMean,SlopeSD,EfficacyMean,EfficacySD 33,4,93,0.2864241,0.112115465,3.419002,1.5861326,1.313991,0.3955596 Analysis and Conclusion ----------------------- There is less data available for this snapshot than desirable; this trial's data collection runs until the end of March and some households have only relatively recently been upgraded for more reliable comms. (The input data includes approximately 3.5 million OpenTRV log entries, and over half a million gas heating fuel log records.) The household-final R^2 values (in 31_segmentedStatsOut.csv), which are taken from the non-control (energy saving enabled) element of the data, are significantly lower than in the overall household heating performance data (20_basicFilteredStatsOut.csv), likely in part because the number of 'normal' days for which data are available is still relatively small. Again, this metric should improve by the end of the heating season in March. Though note that all the R^2 values are low enough (well below 0.9) to indicate that the households' management of their heating could be significantly improved, especially household "F". Household "F" has an overall R^2 of 0.65 which is likely an indication that heating control is very poor[DDN], and notes from installation imply some possible problems with the heating system in general. So although "F" is left as part of the final result, and indicates that OpenTRV has NOT improved efficiency for that household, it is possible that that household would benefit from other larger fixes, or that unrelated changes in use or behaviour were occurring. Note that household "S" has only a single OpenTRV fitted, though to the (very large) radiator in the living room, and so space-heat energy efficiency for the household as a whole could be relatively muted as a consequence, and could depress the final combined figure. The OpenTRV efficacy figure for "T" is higher than normally expected, but not by any means impossible if relatively few rooms are actively occupied for example. Confidence in this figure must be tempered by the low R^2 value. Caveats aside, the overall EfficacyMean figure is greater than 1, indicating that OpenTRV has improved the energy efficiency of space heating. The value implies an overall saving of 31%. Conclusion: it appears that OpenTRV is meeting its target for 30% (or above) energy efficiency improvement, in the absence of OpenTRV boiler control. References ---------- [DDN] http://www.degreedays.net/regression-analysis