Test Plan Key Elements ***DRAFT*** ====================== 0. Version Version Date Updated By Comments -------+--------+--------------------------+------------------------------- V0.1 |20160428|Damon Hart-Davis, OpenTRV |Plain-text initial draft. V0.2 |20160512|Damon Hart-Davis, OpenTRV |Updated, with NPL and PM comments. This document shall be kept under version control, both using an underlying version control system such as SVN or git, and with explicit timestamps and revision numbers for significant updates. V1.0 will be the first non-draft revision. This particular document may not leave draft, and will have its content used elsewhere. 1. SCOPE This is not a complete test plan. This enumerates key elements and their substance for the full test plan. Superset of elements of main document addressed in this document: 2 TEST DESIGN 2.1 Test site 2.1.1 Types of test sites 2.1.2 Addresses 2.1.3 Descriptions 2.2 Tests 2.2.1 Test methods 2.2.2 Test staff 2.2.3 Test schedule 2.2.4 Test equipment 2.2.5 Type and number of samples 2.2.6 Operation conditions 2.2.7 Operation measurements 2.2.8 Technology maintenance 2.2.9 Health, safety and wastes 3 ANALYSIS AND MEASUREMENTS 3.1 Analytical laboratory 3.2 Analytical and measurement parameters and methods 3.3 Analytical and measurement performance requirements 3.4 Preservation and storage of samples 3.5 Data management 3.6 Data storage, transfer and control 4 QUALITY ASSURANCE 4.1 Test plan review 4.2 Performance control – analysis and measurements 4.3 Test system control 4.4 Data integrity check procedures 4.5 Test system audits 4.6 Test report review 5 TEST REPORT APPENDIX 3 IN-HOUSE TEST METHODS APPENDIX 4 IN-HOUSE ANALYTICAL METHODS & MEASUREMENTS 2. Elements Preamble ot location TBD in doc: Testing will be as far as possible according to chapter 5 of ISO 17025: • Personnel involved in testing • Testing premises and environmental conditions; • Test and calibration methods and method validation • Equipment • Measurement traceability • Sampling, and handling of test and calibration items • Quality control • Result reporting MAIN DOC 2.1 Test site Data is gathered from multiple sites/households in real time or with a local timestamped data logger. This data is transported securely to a central server for storage an analysis. The households' approximate locations are recorded geographically in terms of the closest reliable weather station such as "Heathrow Airport / EGLL" for computing heating degree days, and optionally with some further precision such as "north London, UK", but for privacy/security reasons full addresses or lat/lon is not part of the main data set. The data collected will include that gathered and reported by the heat control devices, as well as energy consumption/supply metering. Some details of the data collection sites may also be recorded such as "small family home with X bedrooms" but again not enough to de-anonymise the data subjects. The test analysis can be performed anywhere as no physical tests or equipment rig are required. Formally the tests may be regarded as being performed at the offices of the tester. MAIN DOC 2.2 Tests The primary test consists of analytics on the collected data to show any change in kWh of space heating energy/fuel per heating-degree day. If the devices under test are saving significant space-heat energy as claimed then it should be possible to see a reduction in the kWh/HDD using a linear regression method. This method is usually able to largely eliminate the effects of other uses of the same fuel (eg gas for cooking as hot water as well as space heat). This is primarily a per-property (ie per-household metric), but it may be possible to analyse the data Optionally any automatic temperature setbacks and other energy saving actions explicitly reported by the devices (or implicitly deducable) may be converted into an expected saving and compared with the primary measured saving above. Optionally any reported occupancy may be compared with temperature and heating profiles to estimate bounds on energy saving availability, and compared with the primary measured saving above.. MAIN DOC 2.2.1 Test methods Primary test. Separate the test data for each device/household into those where automatic setbacks are enabled or disabled. For whole-household analysis used data points only where a clear majority (usually at least 2:1) of devices are in the same enabled/disabled mode; discard the rest for the primary analysis. MAIN DOC 2.2.2 Test staff The named technical lead for the ETV testing will be responsible performing the test, though may delegate work as required. MAIN DOC 2.2.3 Test schedule The test schedule will be as specified in the project plan as far as possible. Note that a preliminary round of testing with available data to that date will be collected as a dry run of the test plan and allow any minor fixes in methods, reporting, etc, and to help clarify how abnormal outliers may be identified (if any). A full run against all the test data gathered by the conclusion of the test period, including the data that was used in the preliminary round, will be performed. A copy of the data used will be versioned, frozen and archived. The full data set will be available for audit providing data subject confidentiality can be preserved. As full as possible a set will be made publicly available, eg partially anonymised and/or consolidated, and seperately the analytics may be run on this derived data to show that the outcomes are close and to act as a test case for subsequent development of standards and devices. MAIN DOC 2.2.4 Test equipment The test equipment will be a set of analytics based on tools such as Excel and languages such as Java and Python on Linux or another Unix-like operating system and should be reasonably portable and easy to understand for practitoners in the relevant fields (energy efficiency and simple numerical analysis). Note also the gas meter, Loop, METAR temperature data and derived HDD. MAIN DOC 2.2.5 Type and number of samples Testing should run for a minimum of two weeks, with an initial minimum of one week in some or all properties consisting of a baseline with most smart setback features disabled. In outline temperature, setback and other reading should be approximately 1--15 times per hour, heat fuel consumption hourly--daily, HDD daily. A test data set should be gathered over a period from a minimum of approximately two weeks upwards. Note that some samples may be missed due to (for example) radio transmission failures, and sample values that are not changing may be sent less frequently to make best use of transmission bandwidth. I1) Some basic descriptive data on each household should be provided such as rough geographical location (eg "South east London, UK, 3 bedroom detached house, solid wall 1930s, family of 4, social housing, some fuel poverty issues") but not in form that allows (re-)identification of the household or occupants. I2) Timestamped room air temperature readings gathered regularly with a target precision and absolute accuracy of better than 1C at least hourly, and permantly stored for analysis, and associated with a given heat source controller and zone. a) This is typically the T|C16 measure for OpenTRV devices, in units (and default precision) of 1/16 degrees C, sampled every minute and typically transmitted wirelessly every 4 or 8 minutes depending on other data to be sent. b) Temperature measurement can be taken at the heating device control, eg in the OpenTRV valve for C3/C4, or at a separate sensor unit in the same room / heating zone for C2, but in any case the relationship between the heat controller (eg radiator valve) and the sensor must be clear, such as coming from the same unit with a unique ID or tied by unique IDs of controller and sensor. (It is possible that users will physically move separate sensors to the wrong zone by accident at least temporarily.) Temperature measurements can be taken at the controller and one or more separate sensors, but in this case a primary sensor should be designated, or an algoritihm for synthesising such primary temperature value from the sensors. I3) Heating controller actions and auxiliary data such as detected occupancy hould be timestamped and permanently recorded against the controller ID, eg setbacks applied, actuator operations/values (such as valve open %). I4) Heating Degree Data for the local area for each household gathered at least daily and generally no less than weekly, with appropriate timestamps to match up with (I2), with a demonstrated link back to METAR or equivalent, eg using historically matched points. a) Unless there is a good reason otherwise, a default baseline HDD temperature of 15.5C will be used. I5) Space heating energy consumption will be measured by reference to the household supply meter (traceable back to calibration standards), with before and after checks of any auxiliary reading devices (eg to transmit the data live and in fine grain) against this meter. This meter could be domestic gas or electricity or heat supply. This should be sampled and timestamped at least weekly, preferably daily or hourly. At least 8 distict daily samples are required in each period of operation (eg full vs reduced/control). a) It must be possible to know when specific heat controllers have reduced or full automated savings engaged, eg based on time or on data sent by the devices such as an explicit status flag/datum or inferrable from others. MAIN DOC 2.2.6 Operation conditions As in the protocol: primarily domestic dwellings with 1--4 occupants in the UK with radiator-based central heating (and no significant secondary space heating) where the heating fuel/energy use can be monitored by calibrated metering, along with Heating Degree Days (HDD) based on local METAR or equivalent data. MAIN DOC 2.2.7 Operation measurements WHAT: physical properties being measured (ng m^3 and kWh, METAR temps and HDD) HOW: supply meter and Loop, HDD from degree days (and calibration/verification) MAIN DOC 2.2.8 Technology maintenance Not applicable. MAIN DOC 2.2.9 Health, safety and wastes Not applicable. MAIN DOC 3 ANALYSIS AND MEASUREMENTS TODO: link to Appendix 3 and Appendix 4. MAIN DOC 3.4 Preservation and storage of samples MAIN DOC 3.5 Data management MAIN DOC 3.6 Data storage, transfer and control Integrity with authentication to server. Archival storage with checksums thereafter. MAIN DOC 4.5 Test system audits According to the test body's procedures, regular checks, calibrations and audits are carried out on the various systems used. Eg Loop against gas meter. HDD against base METAR data, checking by separate director that procedures are being followed. MAIN DOC APPENDIX 3 IN-HOUSE TEST METHODS A3.1 Verified Primary Measure Matched pairs of HDD samples (typically 1 day or 1 week interval samples) and kWh consumption (eg derived from gas m^3 * 11.1kWh/m^3) over matching contiguous intervals with automated setbacks enabled or not uniformly within each interval must be collated. Linear interpolation can be used to extract values for matching timestamps if not perfectly aligned (ie with a mismatch of much less than 50% typically), or for closely-matched intervals (75% or better and much less than 1 day) it is likely acceptable to simply ignore the mismatch given relatively slow changes in environmental and house temperatures and setpoints. If the test team is notified of a holiday or other substantial absence or other significant change in heating regime lasting more than 2 or 3 days (if heating is turned off manually, or for example, the boiler fails), then the data for the relevant period plus optionally data for a day or two either side may be excluded from the computations. (If the heating is left running normally and the heating controls detect this and automatically set back further, for example, this data should not be excluded.) Baseload values (intercept) should be recorded, but unless clearly hugely abnormal are not reason to reject a data set, and play no further part in efficiency calculations. (Note that slightly-negtive baseload values are not unusual based on reasonable amounts of noise in the input data. R^2 values should be recorded but unless clearly hugely abnormal are not reason to reject a data set, and play no further part in efficiency calculations, except that values over ~0.9 are indicators of well-controlled heating. R^2 value below 0.5 may be cause for exclusion of the data. Multiple data input formats are acceptable, but where possible should be plain-text (preferably line-per-record CSV), preferably self-describing for a human operator, eg with units, and with all timestamps UTC based if possible unless otherwise stated. Part of the test process is to be able to load and parse data and convert to a standard internal representation before applying analytics, ie the same analytics are applied regardless of input source and format. Example gas consumption data (raw weekly meter readings,before conversion to kWh): 2010-04-26,1130 2010-05-03,1135 2010-05-10,1139 2010-05-17,1142 2010-05-24,1146 2010-05-31,1149 2010-06-07,1152 Example hourly gas consumption data from real-time monitoring, in kWh: Time,Gas (kWh) 2016/05/11 01:00:00,0.00 2016/05/11 02:00:00,0.00 2016/05/11 03:00:00,0.00 2016/05/11 04:00:00,0.00 2016/05/11 05:00:00,0.00 2016/05/11 06:00:00,0.00 2016/05/11 07:00:00,0.00 2016/05/11 08:00:00,0.33 2016/05/11 09:00:00,0.00 2016/05/11 10:00:00,1.89 2016/05/11 11:00:00,0.22 2016/05/11 12:00:00,0.00 2016/05/11 13:00:00,0.00 Example HDD data from EGLL with various base temperatures: Description:,"Celsius-based heating degree days for base temperatures at and around 15.5C" Source:,"www.degreedays.net (using temperature data from www.wunderground.com)" Accuracy:,"Estimates were made to account for missing data: the ""% Estimated"" column shows how much each figure was affected (0% is best, 100% is worst)" Station:,"London / Heathrow Airport (0.46W,51.48N)" Station ID:,"EGLL" ,(Column titles show the base temperature in Celsius) Date,12.5,13,13.5,14,14.5,15,15.5,16,16.5,17,17.5,18,18.5,% Estimated 2011-05-01,0.2,0.3,0.5,0.7,1,1.2,1.5,1.9,2.2,2.6,3,3.4,3.8,0 2011-05-02,1.4,1.7,2.1,2.4,2.7,3.1,3.5,3.9,4.4,4.9,5.4,5.9,6.4,0 2011-05-03,2.9,3.2,3.6,4,4.4,4.8,5.3,5.8,6.3,6.8,7.3,7.8,8.3,0 2011-05-04,2.9,3.2,3.5,3.8,4.1,4.4,4.8,5.2,5.6,6,6.4,6.9,7.4,0 2011-05-05,1.1,1.3,1.6,1.8,2.1,2.4,2.7,3,3.3,3.6,4,4.4,4.8,0 2011-05-06,0.6,0.7,0.9,1,1.2,1.4,1.6,1.7,1.9,2.1,2.3,2.5,2.8,0 2011-05-07,0,0,0,0,0.1,0.2,0.3,0.5,0.7,0.9,1.1,1.4,1.6,0 2011-05-08,0,0,0,0,0.1,0.1,0.3,0.6,0.9,1.2,1.5,1.9,2.3,0 2011-05-09,0.3,0.4,0.6,0.8,1.1,1.3,1.5,1.8,2.1,2.4,2.7,3,3.4,0 2011-05-10,0.7,0.8,1,1.2,1.5,1.7,1.9,2.2,2.5,2.8,3.1,3.4,3.8,0 2011-05-11,0.8,1,1.3,1.5,1.8,2.1,2.4,2.8,3.2,3.7,4.2,4.7,5.2,0 2011-05-12,1.6,1.8,2.1,2.3,2.6,2.9,3.2,3.5,3.9,4.4,4.8,5.2,5.7,0 2011-05-13,1.7,1.9,2.2,2.4,2.7,3,3.3,3.6,4,4.3,4.8,5.3,5.8,0 2011-05-14,1.7,1.9,2.2,2.6,3,3.4,3.8,4.2,4.7,5.2,5.7,6.2,6.7,0 2011-05-15,1.3,1.5,1.8,2.1,2.6,3,3.5,4,4.5,5,5.5,6,6.5,0 2011-05-16,0.2,0.4,0.6,0.8,1,1.3,1.6,1.9,2.3,2.6,3.1,3.5,4,0 2011-05-17,0.4,0.5,0.7,0.9,1.2,1.4,1.7,2,2.4,2.8,3.3,3.7,4.2,0 2011-05-18,0.2,0.3,0.5,0.7,1,1.3,1.6,1.9,2.3,2.7,3.1,3.6,4,0 2011-05-19,1.5,1.7,2,2.2,2.4,2.7,3,3.3,3.6,4,4.4,4.7,5.1,0 2011-05-20,0.7,0.9,1.1,1.3,1.5,1.8,2,2.3,2.6,2.9,3.3,3.8,4.3,0 2011-05-21,0.8,1,1.2,1.4,1.6,1.8,2.1,2.3,2.6,2.9,3.2,3.5,3.8,0 2011-05-22,0.4,0.6,0.8,1.1,1.3,1.6,2,2.3,2.7,3.1,3.5,3.9,4.4,0 2011-05-23,1.1,1.3,1.6,1.9,2.2,2.5,2.9,3.2,3.6,4,4.4,4.8,5.3,0 2011-05-24,1.5,1.7,2,2.3,2.6,2.9,3.2,3.5,3.9,4.2,4.6,5,5.5,0 2011-05-25,1.5,1.7,1.9,2.1,2.3,2.6,2.8,3.1,3.4,3.7,4,4.3,4.7,0 2011-05-26,0.3,0.5,0.9,1.2,1.6,2,2.5,3,3.5,4,4.5,5,5.5,0 2011-05-27,0.9,1.2,1.5,1.8,2.2,2.6,3.1,3.6,4.1,4.6,5.1,5.6,6.1,0 2011-05-28,1,1.2,1.4,1.6,2,2.3,2.8,3.2,3.7,4.2,4.7,5.2,5.7,0 2011-05-29,0,0,0.1,0.2,0.4,0.6,0.8,1,1.2,1.5,1.9,2.2,2.7,0 2011-05-30,0.2,0.3,0.6,0.9,1.3,1.6,2,2.4,2.8,3.2,3.6,4,4.5,0 2011-05-31,1.8,2,2.3,2.6,2.9,3.3,3.6,4,4.4,4.8,5.3,5.8,6.3,0 2011-06-01,1.3,1.5,1.7,1.8,2.1,2.3,2.5,2.8,3,3.3,3.6,3.9,4.3,0 kWh/HDD should be reported for each data interval as a whole and possibly in shorter intervals such as by month, to allow secondary analysis of, for example, different efficiencies gained in shoulder months or intervals of unseasonal weather. A3.2 Secondary (Unverified) Measures Optionally collect data points within the 'disabled' and 'enabled' intervals for zero or more secondary measures to show changes from former to latter: M2a) Show increased R^2 (correlation) for kWH/HDDi for primary household heating fuel to show better control of heating fuel wrt to weather. M2b) Show better room temperature stability ("T|C16") during occupancy. M2c) Show lower room temperature stability ("T|C16", "tT|C") during occupancy. M2d) Show effective room automatic temperature setbacks ("tS|C") during vacancy. Note that for OpenTRV devices non-zero tS|C at some point during a 24h interval is a key indication of enabled efficiency measures. Example JSON timestamped log records for one OpenTRV valve (ID "3015"): [ "2016-05-11T14:23:11Z", "", {"@":"3015","+":0,"T|C16":353,"v|%":0,"tT|C":6} ] [ "2016-05-11T14:27:24Z", "", {"@":"3015","+":1,"tS|C":0,"T|C16":352,"H|%":69,"O":1} ] [ "2016-05-11T14:31:14Z", "", {"@":"3015","+":2,"L":251,"v|%":0,"tT|C":6} ] [ "2016-05-11T14:35:15Z", "", {"@":"3015","+":3,"vC|%":1590,"T|C16":351,"H|%":69} ] [ "2016-05-11T14:39:16Z", "", {"@":"3015","+":4,"O":1,"L":251,"v|%":0,"tT|C":6} ] [ "2016-05-11T14:43:21Z", "", {"@":"3015","+":5,"vC|%":1590,"T|C16":351,"H|%":70} ] [ "2016-05-11T14:47:24Z", "", {"@":"3015","+":6,"T|C16":350,"O":1,"L":251,"v|%":0} ] [ "2016-05-11T14:51:14Z", "", {"@":"3015","+":7,"tT|C":6,"T|C16":350,"H|%":69,"O":1} ] [ "2016-05-11T14:51:14Z", "", {"@":"3015","+":7,"tT|C":6,"T|C16":350,"H|%":69,"O":1} ] [ "2016-05-11T14:55:11Z", "", {"@":"3015","+":0,"H|%":70,"L":251,"v|%":0,"tT|C":6} ] [ "2016-05-11T14:59:12Z", "", {"@":"3015","+":1,"T|C16":349,"H|%":70,"O":1} ] [ "2016-05-11T15:03:24Z", "", {"@":"3015","+":2,"vac|h":7,"L":250,"v|%":0,"tT|C":6} ] [ "2016-05-11T15:03:24Z", "", {"@":"3015","+":2,"vac|h":7,"L":250,"v|%":0,"tT|C":6} ] [ "2016-05-11T15:07:22Z", "", {"@":"3015","+":3,"tS|C":0,"T|C16":348,"H|%":70,"O":1} ] [ "2016-05-11T15:11:19Z", "", {"@":"3015","+":4,"L":249,"v|%":0,"tT|C":6} ] [ "2016-05-11T15:15:16Z", "", {"@":"3015","+":5,"vC|%":1590,"T|C16":348,"H|%":70} ] [ "2016-05-11T15:19:16Z", "", {"@":"3015","+":6,"T|C16":349,"O":1,"L":247,"v|%":0} ] [ "2016-05-11T15:19:16Z", "", {"@":"3015","+":6,"T|C16":349,"O":1,"L":247,"v|%":0} ] Note that if necessary missing/lost records can be inferred from gaps in the "+" sequence number field, as can duplicates transmitted for robustness. Note the named "T|C16", "tT|C", "tS|C" fields in the JSON object in the third element of the JSON array that is one timestamped log entry. MAIN DOC APPENDIX 4 IN-HOUSE ANALYTICAL METHODS & MEASUREMENTS A4.1 Verified Primary Measure Pseudo-code (Java syntax) computation of slope (kWh/HDD) and R^2 looks like: /**Compute energy efficiency metrics based on combined energy use and HDD values; never null nor empty nor singleton. */ public static final HDDMetrics computeHDDMetrics(final Collection data) { if(null == data) { throw new IllegalArgumentException(); } if(data.size() < 2) { throw new IllegalArgumentException(); } // X = HDD (independent variable), Y = consumption. // Pass 1: compute xbar and ybar (means). double sumx = 0; double sumy = 0; final int n = data.size(); for(final ConsumptionHDDTuple datum : data) { sumx += datum.hdd; sumy += datum.consumption; } final double xbar = sumx / n; final double ybar = sumy / n; // Pass 2: summary stats. double xxbar = 0; double xybar = 0; double yybar = 0; for(final ConsumptionHDDTuple datum : data) { final double hdiff = datum.hdd - xbar; final double cdiff = datum.consumption - ybar; xxbar += hdiff * hdiff; xybar += hdiff * cdiff; yybar += cdiff * cdiff; } final double slope = xybar / xxbar; final double intercept = ybar - (slope * xbar); double ssr = 0; // Regression sum of squares. for(final ConsumptionHDDTuple datum : data) { final double fit = (slope * datum.hdd) + intercept; final double fydiff = fit - ybar; ssr += fydiff * fydiff; } final double rsqFit = ssr / yybar; //System.out.println("slope="+slope+ ", intercept="+intercept+ ", rsqFit="+rsqFit+ ", n="+n); return(new HDDMetrics((float) slope, (float) intercept, (float) rsqFit, n)); } As a test vector for data from a household for March 2016, given this daily gas consumption data (collected from a Loop meter): Time,Gas (kWh) 2016/03/01 00:00:00,18.88 2016/03/02 00:00:00,16.99 2016/03/03 00:00:00,14.33 2016/03/04 00:00:00,16.88 2016/03/05 00:00:00,17.77 2016/03/06 00:00:00,25.44 2016/03/07 00:00:00,17.66 2016/03/08 00:00:00,25.77 2016/03/09 00:00:00,13.66 2016/03/10 00:00:00,17.22 2016/03/11 00:00:00,10.77 2016/03/12 00:00:00,16.88 2016/03/13 00:00:00,17.88 2016/03/14 00:00:00,14.88 2016/03/15 00:00:00,21.99 2016/03/16 00:00:00,12.77 2016/03/17 00:00:00,13.11 2016/03/18 00:00:00,27.77 2016/03/19 00:00:00,19.33 2016/03/20 00:00:00,17.10 2016/03/21 00:00:00,13.55 2016/03/22 00:00:00,9.33 2016/03/23 00:00:00,9.89 2016/03/24 00:00:00,18.10 2016/03/25 00:00:00,10.11 2016/03/26 00:00:00,16.11 2016/03/27 00:00:00,9.00 2016/03/28 00:00:00,12.66 2016/03/29 00:00:00,16.99 2016/03/30 00:00:00,7.66 2016/03/31 00:00:00,10.55 And the following EGLL Heating Degree Data for a base temperature of 15.5C: Date,HDD,% Estimated 2016-03-01,6.6,0 2016-03-02,10.1,0 2016-03-03,9.2,0 2016-03-04,10.6,0 2016-03-05,12.5,0 2016-03-06,11.7,0 2016-03-07,11.3,0 2016-03-08,11.4,0 2016-03-09,7.7,0 2016-03-10,9,0 2016-03-11,10.9,0 2016-03-12,9.2,0 2016-03-13,9.7,0 2016-03-14,8.8,0 2016-03-15,9.3,0 2016-03-16,8.3,0 2016-03-17,9,0 2016-03-18,10.4,0 2016-03-19,8.7,0 2016-03-20,8.1,0 2016-03-21,7,0 2016-03-22,7.7,0 2016-03-23,7,0 2016-03-24,7.4,0 2016-03-25,5.1,0 2016-03-26,5.9,0 2016-03-27,7.2,0 2016-03-28,7.2,0 2016-03-29,7.9,0 2016-03-30,7.1,0 2016-03-31,7.1,0 Then to 2SF (two significant figures) the following values should be computed: * kWh/HDD (slope): * R^2: * kWh baseload (intecept): A4.2 Secondary (Unverified) Measures Changes between control (disabled) and normal (enabled) operation can be computed in a number of obvious ways for secondary measures. One issue to be dealt with is that not all measured/generated values are transmitted at once, so interpolation or other measures may be necessary to gather data points at a common timestamp. M2d) Show effective room automatic temperature setbacks ("tS|C") during vacancy. Note that the time-integrated value of tS|C provides a first-order per-device (and therefore per-zone, and in aggregate per household) savings self-estimate, on the basis that cutting room temperature by 1C in the UK is estimated to save 8%--10% of heat demand. MAIN DOC 4 QUALITY ASSURANCE 4 MAIN DOC 4.4 Data integrity check procedures 4 Measurements of parameters such as heat demand and HDD (heating-degree days) should for a sample of households be traced back to calibrated sources, eg secondary gas meters should be checked periodically against their calibrated supply meters, HDD calculations should be sampled and verified against METAR or other calibrated source data. Data from devices should vbe checksummed and/or cryptographically verified to detect corruption in transit. Archived data will be checksummed or equivalent to detect corruption. TO DO: 'Experimental' method with enough detail for a 3rd party to replicate. Physical method, analytical method, QA/QC. Separation of installation from security associations (no tampering possible). (In terms of sections 4, 5, 6 of the protocol.) Evaluation methods: * algorithm description plus input characterisation * possible pseudocode and code snippets to illustrate Informal reference: http://www.energylens.com/articles/degree-days Develop/test basic algorith against 16WW 2016/03 data set (gas + HDD + JSON) and show results here; build unit test cases as calibration into JUnit. Note tracking of Loop vs supply meter over time. IPMVP principles refer to (section 2). Complying with protocol test rqmts section 4. Read "B.V Testing including test plan" p29 "ETV General Verification Protocol".