Test Report DRAFT ================= DHD20160904 For device: OpenTRV "TRV1" Trial/data set name: ES1 2016H1 Household description: single family households, social housing Household location (weather station): north London UK (EGLL) Operation restrictions conditions from test plan met: yes Analytics code release: https://github.com/opentrv/OpenTRV-Java/releases/tag/ETV-20160904-RC1 Valve code release: https://github.com/DamonHD/OpenTRV-Archive-20160813/releases/tag/DORM1-1.0.4-20160315 https://github.com/opentrv/OTRadioLink/releases/tag/DORM1-1.0.4-20160315 https://github.com/opentrv/OTAESGCM/releases/tag/DORM1-1.0.4-20160315 Calibration traceability of data inputs for primary verified measure: 1) For a verified calculation sample from METAR data to Heating Degree Days for EGLL weather station by degreedays.net (and thus degreedays.net HDD traceability) please see: http://www.earth.org.uk/OpenTRV/stds/ETV/20160518-sample-degree-day-calculation-degreedays-dot-net-for-EGLL.xlsx (Worked calculations showing feed-through from METAR data for EGLL on three separate dates in different parts of the heating season: 2015/10/01, 2016/01/01, 2016/03/01.) 2) For a verified sample of tracking of Loop gas meter reading to gas supply meter reading (and thus Loop energy meter reading calibration traceability via the supply meter calibration chain) please see: http://www.earth.org.uk/OpenTRV/stds/ETV/20160819LoopErrorSample.txt (Sample showed an in-situ measurement error of <0.1% over 7 months.) Control vs Normal ----------------- For the TRV1 code release, each device acts as its own control by having an initial period after being installed of up to 254 hours where the majority of the energy-saving features (the automatic temperature setback) are disabled. Some minor features remain enabled, such as the widened temperature regulation deadband around the user-set temperature when the room is dark, because it is primarily there to reduce disturbance eg of sleep or TV watching. (The wide deadband will potentially continue to improve energy efficiency by letting temperatures drift lower before calling for heat, and by calling for heat in slightly longer more efficient bursts than usual.) For some valves in the ES1 trial this was set to 192h (8 days), for others 254h 10.5d). Results and Notes ----------------- Result of full program run on 2016H1 EGLL HDD data and all available Loop data for the ES1 households, plus control/normal stats segmentation using valve log data is shown below. Note that due to problems with the trial, primarily the first below, not enough data was collected to give a meaningful result this time: 1) A number of issues with the radio relaying of stats over the cellular (2G) network meant that for many units data stopped bing relayed after a short while, and attempting to restart the radio relay units was disconcerting some of the householders, so was not pursued. In part this seems to have been bug in the 2G card itself. Newer firmware in our radio relay units should have mitigated this issue. 2) A longer initial control period (>= 14 days rather than 8) should have been chosen and the valve firmware has now been amended to allow this. 3) The trial would ideally have started earlier in the heating season to allow more data to be gathered before the heating season ended. 4) More households than expected of the original 60 had pre-pay meters for which it was not possible to monitor gas usage with Loop. This shows the discarding of poor data and the reduction to a small (and indeed inadequate) final data set of two households of the original 60 (of which ~33 had any data at all), with only 51 days' normal operation. The efficacy result in this case appears to indicate that the OpenTRV TRV1 actually increases heat demand for a given external temperature, but there are two few houses and days' operation for this result to be regarded as meaningful. All these results have been manually anonymised. Data Run Output --------------- ==> 20160904-ETV-prepared-data-out/10_basicStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "0001",5.330478,12.457156,0.7404024,122, "0002",0.0,0.0,NaN,122, "0003",0.0,0.0,NaN,2, "0004",2.1069944,-2.1184983,0.13011934,101, "0005",0.0,0.0,NaN,100, "0006",0.0017260361,0.0011776201,0.014440049,101, "0007",5.421212,-0.12177912,0.6333143,100, "0008",0.008996182,-0.008193092,0.017441792,99, "0009",8.655888,1.6624336,0.6682982,99, "0010",6.8730965,0.7575834,0.6485401,72, "0011",6.058634,26.66937,0.6872283,93, "0012",0.0,0.0,NaN,32, "0013",1.5977722,8.093293,0.37530658,76, "0014",1.7509801,15.115561,0.17595467,49, "0015" "0016",6.4025903,0.6371882,0.74515855,84, "0017",-0.10733571,1.5359908,0.06382664,84, "0018",0.18331076,13.032505,0.006014747,80, "0019",3.5778897,5.755468,0.50982165,73, "0020",0.97071934,37.049965,0.011139902,4, "0021",4.1175313,8.1984825,0.5807201,70, "0022",4.1286397,1.9523131,0.7605321,69, "0023",0.019338386,0.049035538,0.04133039,64, "0024",2.6722965,1.5939214,0.5145182,63, "0025",5.598001,10.684185,0.49589765,63, "0026",-0.08976547,0.49837723,0.14986117,55, "0027",0.029748304,1.2001798,0.0020280986,57, "0028",2.1908882,4.021558,0.6970245,56, "0029",0.0,0.0,NaN,56, "0030",1.2611406,5.596489,0.22003976,34, "0031",9.394523,10.837236,0.34934887,41, "0032",-0.21766476,7.4427094,0.011378305,39, "0033",1.8332185,3.0867794,0.60528654,30, ==> 20160904-ETV-prepared-data-out/20_basicFilteredStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "0001",5.330478,12.457156,0.7404024,122, "0007",5.421212,-0.12177912,0.6333143,100, "0009",8.655888,1.6624336,0.6682982,99, "0010",6.8730965,0.7575834,0.6485401,72, "0011",6.058634,26.66937,0.6872283,93, "0013",1.5977722,8.093293,0.37530658,76, "0014",1.7509801,15.115561,0.17595467,49, "0016",6.4025903,0.6371882,0.74515855,84, "0019",3.5778897,5.755468,0.50982165,73, "0021",4.1175313,8.1984825,0.5807201,70, "0022",4.1286397,1.9523131,0.7605321,69, "0024",2.6722965,1.5939214,0.5145182,63, "0025",5.598001,10.684185,0.49589765,63, "0028",2.1908882,4.021558,0.6970245,56, "0030",1.2611406,5.596489,0.22003976,34, "0031",9.394523,10.837236,0.34934887,41, "0033",1.8332185,3.0867794,0.60528654,30, ==> 20160904-ETV-prepared-data-out/30_presegmentedStatsOut.csv <== houseID,controlDays,normalDays,dontuseDays 0030,3,2,3 0033,0,0,0 0021,0,0,0 0011,0,0,0 0025,1,0,1 0019,0,0,0 0031,7,0,7 0024,8,42,8 0016,0,0,0 0009,0,0,0 0014,0,0,0 0022,8,9,8 0010,0,0,0 0028,0,0,0 0007,0,0,0 0013,0,0,0 ==> 20160904-ETV-prepared-data-out/31_segmentedStatsOut.csv <== "house ID","slope energy/HDD","baseload energy","R^2","n","efficiency gain if computed" "0022",4.5706663,-5.3188424,0.41156936,9,0.35477635 "0024",2.3153985,1.4308916,0.32259336,42,0.92488915 ==> 20160904-ETV-prepared-data-out/90_multihouseholdSummaryStatsOut.csv <== allHouseholdsCount,finalHouseholdsCount,normalDayCount,RsqMean,RsqSD,SlopeMean,SlopeSD,EfficacyMean,EfficacySD 33,2,51,0.36708134,0.044487998,3.4430323,1.1276339,0.63983274,0.2850564 Data Run Output Expanation -------------------------- 1) 20160904-ETV-prepared-data-out/10_basicStatsOut.csv This section lists all the households for which there is any energy consumption data after performing an heating efficiency (kWh/HDD) run over all available data without regard to the OpenTRV valves being in control or normal mode. 2) 20160904-ETV-prepared-data-out/10_basicStatsOut.csv This section lists those households that survive this initial filter for inadequate data or very poor R^2 suggesting very poor heating management. There is no visibility of control/normal operation at this stage. The key filters are: a) Being able to compute the HDD metrics at all which implies a very minimal number of data points and a non-flat energy consumption. b) Suffient days' coverage: a minimum of 12 (twelve) to nominally allow 6 (six) for control mode with energy-saving features largely or entirely diabled, and 6 (six) for normal mode. c) A floor R^2 to eliminate very poor data: 0.15 for per-day data. This is much lower than would be expected for heating systems with even reasonable control and is intended only to remove extreme noisy outliers. These are enforced in the Java code by predicate: uk.org.opentrv.ETV.filter.CommonSimpleResultFilters.goodDailyDataResults 3) 20160904-ETV-prepared-data-out/30_presegmentedStatsOut.csv This incorporates the status sgementation data from the valve logs to split each household's data into days that are control (energy-saving features disabled), normal (enabled) or "don't use". The "don't use" category is for example for days where no majority of valves is in the same enabled/disabled state. The data shown in this output section downs the number of each type of day by household. This status segmentation data can be provided manually, but for TRV1 the basic status for each valve as class: OTLogActivityParse.ValveLogParseResult is extracted by: OTLogActivityParse.parseTRV1ValveLog() which notes which days (always local timezone midnight-to-midnight) have any log data at all, in which of those there is evidence of calling for heat (from the valve being open), and in which of those the automated setback value is being reported, and where is is non-zero. The last of these is taken to indicate the energy saving features enabled. (For future code an explicit stats value may be sent with the enabled/disabled state to remove ambiguity.) In the routine: StatusSegmentation.segmentActivity() combines all valve log data above to produce the result for a single household. "This can use days where any device in the household is calling for heat AND reporting its energy saving status (enabled or disabled). Within those days, any that are marked as having energy saving features disabled for the majority of devices can be regarded as control days; where the majority have energy saving features enabled are normal days." This may understate valve efficacy by eliminating days from the nomral set where noting needed to call for heat. 4) 20160904-ETV-prepared-data-out/31_segmentedStatsOut.csv This shows the per-house analysis of all those that pass further screening, in particular that they have the minimum acceptable number of both control daya and normal days, separately, as enforced by predicate: uk.org.opentrv.ETV.filter.CommonSimpleResultFilters.enoughControlAndNormal The data shows the 'normal' days energy efficiency (kWh/HDD slope), baseload (kWh/day), R^2, 'normal mode' days. The final field is the efficacy with the kWh/HDD of control days divided by that for normal days, so values > 1 indicate actual energy savings (ie improved heating efficiency) with the energy-saving features turned on. 5) 20160904-ETV-prepared-data-out/90_multihouseholdSummaryStatsOut.csv This is the final result of the analysis, across all households remaining at the previous stage, and includes the count of households in the first and final stages, and 'normal mode' days aggregate, and mean and (population) standard-deviation for each of R^2 and slope (kWh/HDD) in normal mode, and for OpenTRV TRV1 efficacy. 33,2,51,0.36708134,0.044487998,3.4430323,1.1276339,0.63983274,0.2850564 The results inidicate that only 33 (thirty-three) households had any data, and only 2 (two) made it though to the final stage, with between those final two only 51 days of 'normal mode' data, so not enough data to be meaningful. (In this case the low R^2 is also an indication that the data is poor.) The efficacy mean < 1 would, if there were enough data, indicate that the OpenTRV TRV1 device reduces heating efficiency cf the previously installed mechanical TRVs and does not meet its claim.