[2014-11-13 19:21:27] =-= Gadget-Mac has changed the topic to ``http://opentrv.org.uk This Channel is logged. '' [2014-11-13 19:28:14] -->| Alasdair (webchat@zje-DE0C01EA.demon.co.uk) has joined #opentrv [2014-11-13 19:28:27] hi [2014-11-13 19:29:23] [INFO] You are now marked as away (Putting kids to bed...). Click the nickname button or use the |/back| command to return from being away. [2014-11-13 19:29:37] (Sorry, be back within an hour...) [2014-11-13 19:30:37] np [2014-11-13 20:12:32] [ERROR] Connection to irc://irc.z.je/ (irc://irc.z.je/) reset. [[Help][Get more information about this error online][faq connection.reset]] [2014-11-13 20:12:48] === *** Looking up your hostname... [2014-11-13 20:12:48] === *** Couldn't resolve your hostname; using your IP address instead [2014-11-13 20:12:49] === You have not registered [2014-11-13 20:12:50] =-= User mode for DamonHD is now +iwx [2014-11-13 20:12:50] *NickServ* Welcome to ZJE, DamonHD! Here on ZJE, we provide services to enable the registration of nicknames and channels! For details, type /msg NickServ help and /msg ChanServ help. [2014-11-13 20:12:55] [INFO] You are now marked as away (Putting kids to bed...). Click the nickname button or use the |/back| command to return from being away. [2014-11-13 20:12:56] -->| YOU (DamonHD) have joined #opentrv [2014-11-13 20:12:56] =-= Topic for #opentrv is ``http://opentrv.org.uk This Channel is logged. '' [2014-11-13 20:12:56] =-= Topic for #opentrv was set by Gadget-Mac on 13 November 2014 19:21:26 [2014-11-13 20:20:25] Yo! [2014-11-13 20:20:34] [INFO] You are no longer marked as away. [2014-11-13 20:21:07] evening [2014-11-13 20:21:25] Hello again! [2014-11-13 20:21:39] Lo mikestir DamonHD [2014-11-13 20:21:43] Hi [2014-11-13 20:21:52] I seem to have a local log working [2014-11-13 20:22:33] So I can post up any bits of the discussion worth saving. [2014-11-13 20:22:37] Did everyone see this [2014-11-13 20:22:38] http://www.earth.org.uk/OpenTRV-protocol-discussions-201411-1.html [2014-11-13 20:22:50] DamonHD, don't forget it's logged centrally [2014-11-13 20:22:54] ? [2014-11-13 20:23:03] Yes, but I have no access to that, not easily anyway. [2014-11-13 20:23:10] Never mind. This will do. [2014-11-13 20:23:11] Evening! [2014-11-13 20:23:16] Hey! [2014-11-13 20:23:30] ok i see it now [2014-11-13 20:23:53] Right, is everyone up for a discussion on logging and data formats and wire protocols and security etc etc? [2014-11-13 20:23:54] i'm just about to eat and will be AFK for a while [2014-11-13 20:24:11] OK Alasdair, though I have some thoughts for you also. [2014-11-13 20:24:11] sounds interesting I'll try multitasking [2014-11-13 20:24:25] OK, no spaghetti coding please! [2014-11-13 20:24:36] I got sleeping nodes working in tinyhan [2014-11-13 20:24:39] not my style [2014-11-13 20:24:42] Yea! [2014-11-13 20:25:00] Ooh data formats! [2014-11-13 20:25:07] Bruno, go on, why don;t you start us off. [2014-11-13 20:25:14] Random thoughts. [2014-11-13 20:25:45] Can we agree to discard CSV files over UDP first? Not that I've been asked that this week you understand :-) [2014-11-13 20:25:53] Eek [2014-11-13 20:26:02] * mikestir throws up [2014-11-13 20:26:06] Surely that is forbidden by international law! [2014-11-13 20:26:14] Yes, let's not to that... [2014-11-13 20:26:23] I'd tell you a udp joke but you might not get it [2014-11-13 20:26:23] Though now you've put the thought in my mind... %-P [2014-11-13 20:26:35] bish bash bosh [2014-11-13 20:26:38] bosh [2014-11-13 20:26:40] bish [2014-11-13 20:26:56] (Sorry, packet repeats from the network.) [2014-11-13 20:26:57] Right so the way I see it, there are 2 data formats that we need to think about: 1) the compact node to local hub format and 2) the more extensive hub to web platform one [2014-11-13 20:27:07] Ho hum. [2014-11-13 20:27:16] shall I tell you what I've got so far? [2014-11-13 20:27:20] Can I immediately split your 1 into to parts Bruno? [2014-11-13 20:27:27] two [2014-11-13 20:27:28] DamongHD: yes [2014-11-13 20:27:34] DamonHD [2014-11-13 20:27:53] 1a) Compact well-engineered binary format (eg as your or Mike would produce) [2014-11-13 20:28:24] 1b) Easy to generate/understand text format for non-engineers and super-fast initial development [2014-11-13 20:28:59] I agree with your 2) as far as it goes. [2014-11-13 20:29:04] Now Mike or Bruno next... [2014-11-13 20:29:15] I still can't see any justification for bothering with 1b) [2014-11-13 20:29:32] it's harder and more cumbersome in every respect [2014-11-13 20:29:33] Mike, you wouldn't! [2014-11-13 20:29:53] I respectfully disgree and have a case in point right now in my own dev. [2014-11-13 20:30:11] look at it from a "maker"'s perspective - they take a load of libraries and bolt them together. the "temperature sensor" library is going to give them a temperature in (probably) a float [2014-11-13 20:30:28] to send that as a binary value they just need to be shown how to but an & in front of it [2014-11-13 20:30:47] And have to carefully decode it at the other end and understand sizeof() etc. [2014-11-13 20:30:52] or to send it as a string they have to know about allocating a string buffer, not overflowing it, format strings, use of snprintf (never sprintf which should be killed with fire) [2014-11-13 20:31:08] there is literally nothing easier about it [2014-11-13 20:31:12] Not if they use a nice JSON library or which there is a particular neat one for Arduino for example. [2014-11-13 20:31:36] and in any case - it would do the arduino crowd some good to actually learn about the tool their using [2014-11-13 20:31:40] And also they'll have no idea if they have it right or not until its all over. [2014-11-13 20:31:50] I mean I can remember grasping endianness when I was about 12 [2014-11-13 20:32:00] Well, that might possibly be seen as a little eliteist and exclusionary! B^> [2014-11-13 20:32:18] Other people just want to be able to slap something together that works and is understandable. [2014-11-13 20:32:28] A bit lit my approach to woodwork. [2014-11-13 20:32:30] yes - people just want everything on a plate [2014-11-13 20:32:31] bit [2014-11-13 20:32:43] That's what I want to be able to deliver to that group. [2014-11-13 20:32:44] What about making 1b) a simple text representation of 1a): both have exactly the same semantics and it's just a question of syntax? [2014-11-13 20:32:54] While not stopping very able people like you doing it right. [2014-11-13 20:33:01] that way you have a well known easy transform between the two [2014-11-13 20:33:04] Yes, that's what I'm aiming foe. [2014-11-13 20:33:11] Except... [2014-11-13 20:33:29] Not BSON, because it simply won't be as compact or well-formed as it might be. [2014-11-13 20:33:40] So the binary form should be hand-craftable. [2014-11-13 20:33:56] The text form extremely simple-minded but robust. [2014-11-13 20:34:08] did you have a look at that thing I came up with, using binary name/value pairs? [2014-11-13 20:34:10] And loggable as is in text files without immediate transformation etc. [2014-11-13 20:34:20] Briefly. [2014-11-13 20:34:41] And I agree nominally with the N/V map view, thus... [2014-11-13 20:34:47] Not BSON but a binary name/value pair would achieve the same goal [2014-11-13 20:35:06] {"@":"cdfb","T|C16":295,"H|%":82,"L":226,"B|cV":256} [2014-11-13 20:35:13] Well, I think not. [2014-11-13 20:35:20] where did that pdf I sent you go DamonHD? did I email it to you or publish it? [2014-11-13 20:35:39] Because a finely crafted one will not need the names ad the representations may not be simple or linear. [2014-11-13 20:35:53] Mike: you forbade me to publish it, so I was waiting for you to! [2014-11-13 20:36:35] So Bruno, I can get my C16 temp and battery status and CRC in 3 bytes [2014-11-13 20:36:50] http://mike-stirling.com/files/tinyhan_app.pdf [2014-11-13 20:36:50] carefully crafted not to be all 1s or all 0s also. [2014-11-13 20:37:38] DamonHD: I agree but you could replace the names with known binary keys in the binary format [2014-11-13 20:37:58] I think I'd remove the group ID from the header now because the tinyhan mac is optimised for a client-server model now anyway [2014-11-13 20:38:00] No need for the keys at all. A waste of space when other issue have been resovled. [2014-11-13 20:38:01] so for example, C16 could map to a binary code [2014-11-13 20:38:10] mikestir, Thanks, interesting reading. [2014-11-13 20:38:22] DamonHD: good point [2014-11-13 20:38:29] CF the text format where new keys and values can be introduced at will without the receiver even having to understand all of them in advance. [2014-11-13 20:38:47] So in my actual 3-byte frame it does. [2014-11-13 20:39:03] But wrapped carefully over two bytes and mingled with other stuff. [2014-11-13 20:39:45] And the text form can have explicit units for new keys also. [2014-11-13 20:40:20] mikestir: interesting, I would also add volume units somewhere for fluid consumption (water / gas meters) [2014-11-13 20:41:01] And issue here is that someone may want to count donkeys. [2014-11-13 20:41:05] Or buses. [2014-11-13 20:41:07] Or feet. [2014-11-13 20:41:24] And so there in any case need to be an escape route for those things. [2014-11-13 20:41:41] And the text form can provide that too, even if inefficiently. [2014-11-13 20:41:58] Buses and feet are looking quite likely now. [2014-11-13 20:42:08] DamonHD: yes, you could have an 0xFE profile for custom metrics [2014-11-13 20:42:33] (Mike we got through the the last stage of IoT Launchpad: did I say that we got an LoI from Transport for London.) [2014-11-13 20:42:36] I was thinking that units might be useful for things like boiler output, instead of relative proportion. [2014-11-13 20:42:49] bruno: that is nowhere near complete, but yes that would go under the energy profile [2014-11-13 20:43:41] Hi Tim. [2014-11-13 20:43:50] Some things will have units, some not. [2014-11-13 20:44:14] For example I have an ambient light sensor which is certainly not linear nor in lux for example. [2014-11-13 20:44:25] But lower is darker and lighter is brighter. [2014-11-13 20:44:32] {"@":"cdfb","T|C16":295,"H|%":82,"L":226,"B|cV":256} [2014-11-13 20:45:03] DamonHD: you could have a generic "range" unit in the 0x00 profile [2014-11-13 20:45:05] That's from the unit on my desk, received over the air by the hub in the kitchen and sent back up by WiFi! B^> [2014-11-13 20:45:08] I was thinking about this a while ago, whether it's better to stick to a standard unit and make a small embedded device convert it, or to allow some sort of server-side conversion [2014-11-13 20:45:15] I sort of landed on the side of making the sensor do the conversion [2014-11-13 20:45:28] The sensor doesn't have the space nor need to. [2014-11-13 20:46:00] Each unit would have to be calibrated and it's not needed for the 'is it too dark to see in here' purpose. [2014-11-13 20:46:07] I came to the conclusion that is it is going to be interoperable then it really does need to [2014-11-13 20:46:20] it's not practical to have things talking to each other if they are talking arbitary values [2014-11-13 20:46:24] Not if only relative values are needed. [2014-11-13 20:46:27] I disagree. [2014-11-13 20:46:31] I mean you could change to a different light sensor and then what? [2014-11-13 20:46:34] And this is where Bruno's point (2) hits. [2014-11-13 20:46:41] Mike: doesn;t matter. [2014-11-13 20:46:52] Then that unit will interpret its own sensor accordingly. [2014-11-13 20:47:03] mikestir: that would be specific use cases where it's too complicated for the sensor to do calibration [2014-11-13 20:47:10] no, I mean you could change to a different light sensor chip [2014-11-13 20:47:13] A centra hub can note typical values for a given ID, or discard the data entitrly. [2014-11-13 20:47:16] or you could do it yourself with a photodiode [2014-11-13 20:47:25] Not while retaining the node ID, that would be cruel. [2014-11-13 20:47:51] the mcu on the sensor has to either convert to a common unit or you end up having to have different unit descriptors for different versions of the same sensor [2014-11-13 20:48:06] So, on the different REV boards I do coerce different sensors' outputs into an approx 0-255 range. [2014-11-13 20:48:18] There is no point for this. [2014-11-13 20:48:30] It only needs enough sense to know 'very dark' or not. [2014-11-13 20:48:41] so at the moment you have to tell the central hub how to deal with every sensor? [2014-11-13 20:48:51] Nope. [2014-11-13 20:48:54] this is something I'm trying to explicitly avoid with tinyhan - it can self-organise the network [2014-11-13 20:49:15] But the hub would have to understand that unitless values only make sense coupled with a particular node ID. [2014-11-13 20:49:31] I tend to use things like quartlie analysis [2014-11-13 20:49:50] which is relatively insensitive to the absolute values, eg: [2014-11-13 20:50:25] http://www.ecotricity.co.uk/our-green-energy/energy-independence/uk-grid-live [2014-11-13 20:50:33] The central hub could probably work out what's going on (with light levels), given enough data, and in many cases I suppose relative values are fine - but in others it would be useful to have absolute. [2014-11-13 20:50:49] work out = learn [2014-11-13 20:51:00] the traffic lights are done entirely on reg being the highest intensity quatile over last 24h and green the lowest [2014-11-13 20:51:06] the central hub could not work out what was going on, in terms of getting the actual lux value, without knowledge of the innards of the sensor [2014-11-13 20:51:18] actual absolute values are not programmed in anywhere. Not even units. [2014-11-13 20:51:31] It doesn't need to know lux. [2014-11-13 20:51:39] they might not be in opentrv, but I thought this was supposed to be generic [2014-11-13 20:51:45] measurements need units [2014-11-13 20:51:54] Other applications can add units. [2014-11-13 20:52:12] This is why being over-prescriptive up front may be harmful I think. [2014-11-13 20:52:29] So note in my example the battery voltage does have units [2014-11-13 20:52:33] as does the temperature. [2014-11-13 20:53:07] I agree with DamonHD: we've had situations in real life where for one reason or another it was impossible to calibrate the sensor and provide a unit out of the device and where that calibration was better done server side [2014-11-13 20:53:26] I agree it's not an ideal scenario but we should support it [2014-11-13 20:53:34] (BTW, tickets are flying away for our OpenTRV conf on 29th...) [2014-11-13 20:53:39] sure there's no reason why there can't be a "dimensionless" option [2014-11-13 20:53:41] If you want to keep complexity down in the simple stuff (battery powered sensors and the like), then I suppose something which allows other things to infer absolute values might be a better route (i.e. this is a type xyz light sensor, so go use the type xyz lookup table). [2014-11-13 20:53:51] Exactly. [2014-11-13 20:53:58] but I think in the majority of measurement applications one should strive to use real units [2014-11-13 20:54:18] just because it's "hard" to get a lux value isn't an excuse (and it really isn't that hard anyway - well within the capabilities of an 8-bit micro) [2014-11-13 20:54:21] And not every situation will be the same, and we want to keep the leaf nodes and comms as easy as possinle from that side. [2014-11-13 20:54:34] DamonHD, you're making the units case sensitive ? [2014-11-13 20:54:46] mikestir: agreed but we should still allow for edge cases [2014-11-13 20:55:06] this will probably need to be worked out on a per-application basis - a relative temperature is probably relatively useless for opentrv, but a relative light level quite useful. I suppose the protocol shouldn't proclude either type. [2014-11-13 20:55:07] But Mike, I have enormous variability in my sensors. I certtainly cannot spend effort calibrating them for (say) the OpenTRV valvrs. [2014-11-13 20:55:15] Indeed. [2014-11-13 20:55:44] So we've got to try to allow for sensible levels of definition. [2014-11-13 20:55:48] Each app may be different. [2014-11-13 20:56:09] Not that in my example before 3 out of the 4 samples did have units [2014-11-13 20:56:16] variability for light sensors? [2014-11-13 20:56:28] though not necessarily the obvious ones to avoid redundancy. [2014-11-13 20:56:50] Mike, yes, 24p LDRs and phototransistors are not matched closely. [2014-11-13 20:57:16] well the phototransistors are probably ok, but the LDRs I'm not surprised about [2014-11-13 20:57:57] what did you do about getting a log response on the phototransistor? [2014-11-13 20:57:59] Even the phototransistors have quite a large spread. [2014-11-13 20:58:32] But in particular also my LDR circuit is hugely non-linear but it really doesn't matter for this app, [2014-11-13 20:58:40] or even for a quartile-based analysis (for example). [2014-11-13 20:59:13] To answer questions like: "is it usually this dark in this room at this time?" [2014-11-13 20:59:34] phototransistor photocurrent is linear wrt to illuminance, but the lux range is so large it really needs to be handled as log lux [2014-11-13 20:59:39] To which the answer might be "Send the maintenance crew!" [2014-11-13 21:00:00] there are devices with integrated log amps, or fully integrated daylight sensors with an I2C interface [2014-11-13 21:00:07] So, I delinearlise in another way. [2014-11-13 21:00:42] But if someone really wants to get some idea of lux best done at the hub or even later based on node ID I suggest. [2014-11-13 21:01:13] but then we get back to the hub (or back end) needing to know specifics about each sensor [2014-11-13 21:01:13] One thing that Bruno has made clear is that being clear about node ID (and often it's upsteam concentrator) is critical [2014-11-13 21:01:33] Then one can solve these problems lazily and somewhere more powerful, after the event. [2014-11-13 21:01:43] If it ever needs to. [2014-11-13 21:01:59] Yes, being able to trace the provenance of a data item is key [2014-11-13 21:02:00] For a stats analysis the node (and conc) ID is all it ever needs to know. [2014-11-13 21:03:02] so is a time-stamp but that can be added by the concentrator [2014-11-13 21:03:07] Indeed. [2014-11-13 21:03:26] I have no objection to a leaf maintaining time *if* it can do it. [2014-11-13 21:03:33] DamonHD: Can sensor-type info be derived from that? If not, I wonder if each sensor type needs some sort of ID? [2014-11-13 21:03:33] But it shouldn't be relied on. [2014-11-13 21:03:59] Tim: I suggest node + short sensor key is enough [2014-11-13 21:04:17] then a catalogue can be used at the concentrator aor after the even when processing the data [2014-11-13 21:04:20] DamonHD: one option would be to include a time offset since when the measurement was taken: this could be useful if the sensor fails to connect to the concentrator immediately [2014-11-13 21:04:27] What happens if I want to act on LUX with another sensor, I'm reliant on the hub to interpret ? [2014-11-13 21:04:30] But I do suggest some simple/common tyoes such as temp. [2014-11-13 21:04:38] Thinking of USB analogy, I was thinking that you could perhaps have standard types, with vendor:id being used for ones which aren't standard types. [2014-11-13 21:04:47] sensor ID is a mac layer function [2014-11-13 21:04:55] data encapsulation is application layer [2014-11-13 21:04:57] Bruno: yes, though if the leaf is off for a bit, eg for battery change or reset, what then? [2014-11-13 21:05:09] GM: who is 'I' in that case? [2014-11-13 21:05:11] Gadget-Mac: the way I tend to do it is if the sensor can give me the info, I use it, if it can't I can add that info in the hub [2014-11-13 21:05:41] DamonHD, lets say a remote light switch [2014-11-13 21:05:42] Tim: max packet length including framing is typically 64 bytes for common radio engines [2014-11-13 21:05:57] and so as not to bust duty cycle rules [2014-11-13 21:06:12] IWhy does it need lux? [2014-11-13 21:06:19] GM: sorry [2014-11-13 21:06:59] OK, are you thinking that comms is always bi-directional, or could some be unidirectional? [2014-11-13 21:07:05] Why not, given it's the standard way of measuring light [2014-11-13 21:07:06] TimSmall: that's basically what I'm aiming for with the profiles and binary types. as bruno suggested there could always be a reserved profile for user stuff [2014-11-13 21:07:21] GM: because it's expensive to get calibrated values [2014-11-13 21:07:40] And indeed there was non I2C device I could find that would run at 1.8V, else I would have done it. [2014-11-13 21:07:41] however. another reason for forcing everything onto standard units where possible is the "vendor proprietary" trap [2014-11-13 21:07:46] DamonHD: I'm assuming the leaf has no local storage and therefore would only be sending data that it has captured since it was switched on [2014-11-13 21:08:05] Bruno: a little non-volatile storage is cheap these days. [2014-11-13 21:08:15] TimSmall: comms could be unidirectional, and often bidirectional will be very high latency [2014-11-13 21:08:18] I remember the time in OpenTRV units and the clock just stops while they are off. [2014-11-13 21:08:30] Tim: yes, often unidirections [2014-11-13 21:08:38] uindirectional [2014-11-13 21:08:42] uni [2014-11-13 21:08:43] !!!! [2014-11-13 21:08:56] DamonHD: if you don't have a clock, that's still better than nothing :-) [2014-11-13 21:09:26] For example an EnOcean type light switch which transmits when the user presses the (piezo) button that power the radio. No need for nor hope of back channel there [2014-11-13 21:09:56] Bruno: I'm not disagreeing with your assertion above, just pointing out a wrinkle. [2014-11-13 21:10:16] And it takes a bit of TX space which the hub can measure just as well on receipt often. [2014-11-13 21:10:23] I'm thinking that you only need the node type info (I'm a Vendor X, ID Y, standard type 42 sensor) once, then the central hub can record that and associate it with that source address. But it would be really nice to have that sort of ID stuff baked in, otherwise I can foresee compatability nightmares of ISA type proportions. [2014-11-13 21:10:32] (I'm a bit slack at the moment and might have to up my game for generic sensors.) [2014-11-13 21:11:04] OK, there's another complication, which may be necessary, of not transmitting everything in every frame, [2014-11-13 21:11:16] and assuming that the rest was heard by the recipient. [2014-11-13 21:11:41] But what you say could be a good idea for well-characterised sensors. [2014-11-13 21:11:55] For the cheapo ones, maybe less so, dunno, but it's a good idea. [2014-11-13 21:12:08] I'm going to add it to the page bullet points. Thanks. [2014-11-13 21:13:09] OK, so assuming there is a routing pre-amble that contains the sensor ID, are we happy that tinyhan is a sensible binary data format for case 1a? [2014-11-13 21:14:13] Providing it'll give in and let us send text format some time, and unitless values some times, [2014-11-13 21:14:16] Even if it indicates "all bets are off with this piece of crap", that sort of meta info seems valuable. I'm not very familiar with the 1-wire protocol, but I think it covers a lot of this stuff (and is designed around very-cheap-low-power-simple-slaves). At the least each device has globally unique serial number - might be worth looking at the spec. [2014-11-13 21:14:24] then I trust Mike to do as good a job as can be done in that area. [2014-11-13 21:14:46] I create a random 64-bit ID in the node. [2014-11-13 21:15:06] That really ought to do to uniquely ID it and its sensors I hope! [2014-11-13 21:15:16] bruno: probably best to flesh out the profile definitions a bit :) [2014-11-13 21:15:27] BTW, loads of good discussion here, thanks. [2014-11-13 21:15:30] DamonHD: yes, especially when combined with the concentrator ID [2014-11-13 21:15:42] mikestir: yes but at least this is a start :) [2014-11-13 21:15:43] DamonHD: Sounds good. Might be worth doing Ethernet-style, and having 1 bit of the 64 indicate random vs. assigned, so the two don't clash? [2014-11-13 21:15:53] Mike, can we please publish your draft as it stands somewhere that it can be found. [2014-11-13 21:16:15] I have a secret escape route for that. [2014-11-13 21:16:32] At the moment the top bit of every byte is forced true (and the rest cannot be all 1s) [2014-11-13 21:16:44] to avoid clashes with other forms of assigned IDs. [2014-11-13 21:16:51] bruno: and the mac layer deals with node identification already, e.g. hear is hub status for two registered sensors on my desk: http://pastebin.com/Nii4fKsr [2014-11-13 21:16:52] Initially FS20 house codes [2014-11-13 21:16:55] DamonHD: cool. [2014-11-13 21:17:00] s/hear/here/ [2014-11-13 21:17:27] DamonHD, why not use something like this http://uk.farnell.com/microchip/24aa02e48-i-sn/eeprom-2k-1-8v-i2c-mac-soic8/dp/1688857 [2014-11-13 21:17:35] I'm hoping my IDs would be directly usable by Mike's stuff when we use his MAC. [2014-11-13 21:17:40] mikestir: brilliant, that works! [2014-11-13 21:18:18] GM: for what? [2014-11-13 21:18:30] Getting a uniq id. [2014-11-13 21:18:49] Also gives you addition NV storage [2014-11-13 21:18:54] most recent MCUs have unique IDs anyway - problem is they're only unique within that device family [2014-11-13 21:19:20] If a node doesn't otherwise have NV storage, yes that would probably do nicely. [2014-11-13 21:19:39] Not an issue for ATMega-based nodes. [2014-11-13 21:20:03] Plus I seem to have enough good local entropy to be able to self-generate well-dispersed unique IDs. [2014-11-13 21:20:35] But that's a very interesting and cheap part, thanks! [2014-11-13 21:21:22] There are smaller parts if you need to save board space, with the same spec. [2014-11-13 21:21:36] At the moment, not an issue, but very good to know. [2014-11-13 21:21:45] And I2C also, which is getting to be my friend... B^> [2014-11-13 21:21:54] And a good voltage range. [2014-11-13 21:21:58] Mmmmmmm. [2014-11-13 21:22:36] OK. [2014-11-13 21:22:37] I do actually know what I'm doing ;) [2014-11-13 21:22:42] [Vote] We will use tinyhan for case 1a, assuming we will fully flesh it out to cover all the necessary use cases: yay or nay? [2014-11-13 21:22:54] Well, hang on one moment... B^> [2014-11-13 21:23:03] I don't want to have to run more than one protocol. [2014-11-13 21:23:03] :) [2014-11-13 21:23:20] So (1) we may have to use other carriers occasionally (eg FS20). [2014-11-13 21:23:43] and (2) if TinyHAN can be coerced into sending JSON when needed, them YES YES YES! [2014-11-13 21:24:28] And also ideally of TinyHAN will let me send a binary bitsteam when I really need to to my own thing. [2014-11-13 21:24:35] if [2014-11-13 21:25:05] The key thing here with TH it seems is the elegant RF comms. [2014-11-13 21:25:09] you can send what you like over the tinyhan mac layer [2014-11-13 21:25:17] Where it can provide good structure too, even better. [2014-11-13 21:26:05] Good, then I think we can vote for the TinyHAN MAC layer as the preferred one for local RF networks. [2014-11-13 21:26:05] so currently my plan is to translate the tinyhan app protocol packets to JSON in the hub for delivery over mqtt [2014-11-13 21:26:11] I have this working - I ported the paho client to stm32 [2014-11-13 21:26:15] Kewl. [2014-11-13 21:26:41] So if a JSON frame is delivered over the the MAC layer would you pass it on unchanged? [2014-11-13 21:26:49] Or wrap it somehow? [2014-11-13 21:27:01] you can do what you like. there's a field equivalent to a port number [2014-11-13 21:27:04] just take one [2014-11-13 21:27:16] OK. [2014-11-13 21:27:24] it's only 4 bits though (although I reserved 0xf for an extension header) [2014-11-13 21:27:34] That's more than I currently need. [2014-11-13 21:28:03] So we could have JSON. bit-stream and 'Mike format' ports then? [2014-11-13 21:28:21] i have one for mqtt-sn [2014-11-13 21:28:39] Could you remind me the frame length within that, ie the max JSON string for example that would work with an RFM23B via your MAC layer? [2014-11-13 21:28:44] that works as well - you can bridge it directly to udp and the sensors will register with eclipse rsmb [2014-11-13 21:28:57] it's large though and offers no advantages over just sending the values [2014-11-13 21:29:21] I just don't want to over-constrain things. [2014-11-13 21:29:41] And for example the bit-stream one may be good for totally encrypted stuff. [2014-11-13 21:29:43] there's a 1 byte payload length field at the phy layer, but specific phy may limit it further [2014-11-13 21:30:03] So what do you think it might come to over the 23B or 69CW? [2014-11-13 21:30:06] e.g. the rfm23 driver doesn't support packets larger than the fifo so its mtu is 64 bytes. max mac payload is 64-headers [2014-11-13 21:30:17] At the moment I'm limited to about 56 bytes of JSON payload including the {}. [2014-11-13 21:30:21] Which is just about OK! [2014-11-13 21:30:22] and probably a bit less once we figure out crypto [2014-11-13 21:30:34] Yes, that will make me cry. [2014-11-13 21:30:38] 23 and 69 will be about the same iirc [2014-11-13 21:31:00] Yep, but I haven't a good idea what your MAC overheads are. [2014-11-13 21:31:35] 6 bytes [2014-11-13 21:31:48] OK, probably very similar then. [2014-11-13 21:32:35] In which case unless anyone objects I repeat that TinyHAN MAC should be preferred local RF MAC layer for OpenTRV and the generic sensor platform. [2014-11-13 21:33:02] So Bruno, what was your 3rd sentence going to be right back at the start? B^> [2014-11-13 21:33:03] [Decision] TinyHAN MAC should be preferred local RF MAC layer for OpenTRV and the generic sensor platform. [2014-11-13 21:33:30] And we will use it to support Mike/JSON/bin format frames. [2014-11-13 21:34:05] and I am very close to publishing this spec by the way. I have one thing I need to sort out (validity periods on deferred transmits) [2014-11-13 21:34:32] So now for use case 2: send data between concentrator and other more powerful devices [2014-11-13 21:34:57] full-fat mqtt? [2014-11-13 21:35:01] and json? [2014-11-13 21:35:12] proper, wordy json [2014-11-13 21:35:27] I think so. [2014-11-13 21:35:42] At that point bandwidth and storage is massively cheaper. [2014-11-13 21:36:00] Let's not store a copy of the phone directory in each frame for a laugh. [2014-11-13 21:36:13] But let's error on the descriptive and redundnt side. [2014-11-13 21:36:25] I am also intending to add mqtt client support to timestore, by the way, for long term storage [2014-11-13 21:36:36] mikestir, yes please ^^^^^^ [2014-11-13 21:36:39] then historical data can be pulled out in json as well [2014-11-13 21:36:43] Well, we may well wish to be a user of that of course! [2014-11-13 21:36:46] Yes, proper JSON and if possible we should allow several (sensible) transport options, e.g. MQTT, HTTP POST [2014-11-13 21:36:46] B^> [2014-11-13 21:36:51] yaml? [2014-11-13 21:37:02] I'm not a huge fan of YAML. [2014-11-13 21:37:21] TimSmall: JSON allows enough complexity in the data and is easier to parse than JSON [2014-11-13 21:37:24] Having spent FAR too much time with it over the last year or so. [2014-11-13 21:37:35] sorry, JSON is easier to parse than YAML [2014-11-13 21:37:36] I get the impression json is becoming the defacto standard for this job [2014-11-13 21:37:44] DamonHD: OK [2014-11-13 21:37:46] And there seem to be no real standard parsers or schemas. [2014-11-13 21:37:55] JSON seems a good happy medium. [2014-11-13 21:38:01] Not perfect, but servicable. [2014-11-13 21:38:02] Fair enough. [2014-11-13 21:38:08] yes, most language have good JSON libraries while this is not true of YAML [2014-11-13 21:38:08] And eminently archivable, etc. [2014-11-13 21:38:51] oh, I may steal another "ethertype" for the OTA log submission. I've had that working quite nicely in the past so that my avr sensors post their stdout to a syslog server :) [2014-11-13 21:38:53] http://www.earth.org.uk/OpenTRV-protocol-discussions-201411-1.html [2014-11-13 21:39:00] Decisions added. [2014-11-13 21:39:09] (I will add a log transcript too.) [2014-11-13 21:39:58] OK. [2014-11-13 21:40:08] Now we may have done enough for this evening. [2014-11-13 21:40:27] Yodit was suggesting a schema.org style key list by default [2014-11-13 21:40:28] I have a couple of questions - should I ask them here, or is the mailinglist better (and you'd like to keep this to predetermined topics - I couldn't tell from the web pages), or is here OK but not now? [2014-11-13 21:40:35] at least in the expanded form I guess. [2014-11-13 21:40:45] Tim: what about? [2014-11-13 21:41:09] One last thing: for use case 2, the format we use should include full time stamps with time-zone specification in ISO 8601 format (time zone preferably UTC) [2014-11-13 21:41:12] (This is a rather special case, focussed on a specific topic. We normally ramble on or have deadly silence.) [2014-11-13 21:41:36] All full timestamps in sensible unabiguoius ISO-standard UTC. [2014-11-13 21:41:38] Agreed? [2014-11-13 21:41:42] bruno: agreed. I am assuming the hub will have at least an sntp client [2014-11-13 21:41:47] yes, ISO 8601 [2014-11-13 21:42:02] and with time zone specified [2014-11-13 21:42:11] just use utc [2014-11-13 21:42:21] always Z [2014-11-13 21:42:45] Let;s do things the UNIX way. [2014-11-13 21:42:56] shut up shop in 2035? [2014-11-13 21:42:56] Store in UTC, display however you like. [2014-11-13 21:42:58] mikestir: agreed, although I would prefer to insist on the Z to always be present to remove all possible ambiguity [2014-11-13 21:43:10] We're in 64-bit unsigned land now Mike! [2014-11-13 21:43:12] yes agreed [2014-11-13 21:43:42] UTC/Z is always best. [2014-11-13 21:44:02] Any views on precision, eg of seconds part? [2014-11-13 21:44:09] although we may want to allow a non-UTC time zone to be specified as long as it conforms to ISO 8601 [2014-11-13 21:44:13] Or do we leave that implementation dependent. [2014-11-13 21:44:22] Indeed we may not even have a time for some... [2014-11-13 21:44:31] implementation defined [2014-11-13 21:44:34] Nope, always UTC for now. [2014-11-13 21:44:36] B^> [2014-11-13 21:44:48] the latency of the mac could be significant, so high resolution would only be useful if the node was timestamping [2014-11-13 21:44:57] yes, implementation defined with precision down to at least a minute [2014-11-13 21:45:14] I can tell you the trouble I have see when people say we'll fix it later and then lose an hour's data at least once per year [2014-11-13 21:45:31] Well, I'm going to suggest just as many field from the left as is needed. [2014-11-13 21:45:38] Daily records don't need a time. [2014-11-13 21:45:38] Second is probably good enough for most things. [2014-11-13 21:45:45] Yearly records may not need anything else. [2014-11-13 21:45:55] Saves lots of space, is unambiguous and legal. [2014-11-13 21:45:59] Gadget-Mac: minute is good enough for most things [2014-11-13 21:46:23] Sometime we want nano seconds (rememebr where I've just been working). [2014-11-13 21:46:26] but the good thing about ISO 8601 is that it allows you to be flexible and only increase the precision of you need to [2014-11-13 21:46:31] If we're using ISO 8601 are we allowing truncations ? [2014-11-13 21:46:33] 8601 accommodate variable precision. [2014-11-13 21:46:53] I'd say allow truncations down to minute precision [2014-11-13 21:47:28] Specifically thinking about YYYYMMDD and YYMMDD [2014-11-13 21:47:31] Nah, I'm saying right down to whatever is appropriate for the data, [2014-11-13 21:47:32] so 2014-11-13T21:47 is OK but not 2014-11-13T21 [2014-11-13 21:47:39] It's unambiguous. [2014-11-13 21:48:00] Why force in bogus and misleading extra stuff? [2014-11-13 21:48:19] Gadget-Mac: no, I think we should require a 4 digit year [2014-11-13 21:48:36] bruno, thats fine, as we've said at this level storage is cheap. [2014-11-13 21:48:43] If I have a yearly figure, eg average rainfall in Croydon, what would the MM and DD and HH etc fields be? [2014-11-13 21:49:06] more teasing-> http://www.mike-stirling.com/files/tinymon.png [2014-11-13 21:49:06] GM: always 4 digit year. I don't think the ISO standand allows us to have THAt fun again. [2014-11-13 21:49:14] ah OK, I didn't think of that use case, I was thinking of sensor data sent on the fly [2014-11-13 21:49:47] So will you let me record this as the decision for now... [2014-11-13 21:49:50] ... and with variable precision from just years down to many decimals of seconds [2014-11-13 21:49:50] as appropriate to the data. [2014-11-13 21:49:57] yes [2014-11-13 21:50:10] you may also want to include the week notation as a lot of data is weekly [2014-11-13 21:50:49] OK. [2014-11-13 21:50:52] ... and with variable precision from just years down to many decimals of seconds [2014-11-13 21:50:52] as appropriate to the data, ie truncated on the right, with years always 4-digit; [2014-11-13 21:50:52] week notation also allowed. [2014-11-13 21:50:58] Going back to the rain stuff, you can represent time intervals with 8601 [2014-11-13 21:51:10] Let's not go there for the moment. [2014-11-13 21:51:23] At the moment we are talking about timestamps. [2014-11-13 21:51:28] That's a different use case. [2014-11-13 21:51:38] But some stuff just isn;t very fine grained. [2014-11-13 21:51:54] For example I have gold prices going back to the 1600s I think. [2014-11-13 21:52:10] Whats the price of gold got to do with it ;) [2014-11-13 21:52:14] Annual until fairly recentlu, now twice-daily fixing. [2014-11-13 21:52:28] sounds like a job for timestore :) [2014-11-13 21:52:35] I was wondering if the in-house motorized valve heads were likely to be sufficiently skinny to fit onto UFH manifolds - my other solutions are to butcher a Honeywell stat or use those resistively-heated-wax-capsule things. [2014-11-13 21:52:38] Fill it with gold! [2014-11-13 21:52:52] This sort of thing: http://goo.gl/J5OZIM [2014-11-13 21:52:57] TS: well, ours may be. [2014-11-13 21:53:26] Quite slim. [2014-11-13 21:53:35] I'll go and check what the centres are - 1 min... [2014-11-13 21:53:59] Do we want to use anything like SenML? http://www.ietf.org/archive/id/draft-jennings-senml-10.txt [2014-11-13 21:54:08] Flanked my an AA cell each size it's still only only ~4cm across out entire all-in-one unit. [2014-11-13 21:55:06] I think we may well want to use SenML and/or a standardish key set for common sensor types (with proper units!) to facilitate sharing of data ad hoc. [2014-11-13 21:55:34] Mine manifolds are 50mm centre-to-centre, but I'm pretty sure some others are 40mm. [2014-11-13 21:55:53] For units, what about using the UCUM codes? http://unitsofmeasure.org/trac/ [2014-11-13 21:55:57] OK. I suspect our unit is ~2cm wide, but I haven't actually seen it yet. [2014-11-13 21:56:44] Sounds a good idea, but HOW will units be represented. [2014-11-13 21:57:25] as text? [2014-11-13 21:57:36] In the JSON I had they are rather compact but shouldn't cause surprise. [2014-11-13 21:58:04] {"e":[ [2014-11-13 21:58:04] { "n": "voltage", "t": 0, "u": "V", "v": 120.1 }, [2014-11-13 21:58:04] { "n": "current", "t": 0, "u": "A", "v": 1.2 }], [2014-11-13 21:58:04] "bn": "urn:dev:mac:0024befffe804ff1/" [2014-11-13 21:58:04] } [2014-11-13 21:58:19] I played around with something like this a while ago. The idea was to define everything as a linked list relative to base units, so that a front-end could combine multiple values and figure out the right unit to use for the derived value [2014-11-13 21:58:46] so a value could be in watts, but there was a table that defined watts as joules per second [2014-11-13 21:58:57] I think we've made good progress this evening and I may need to stop soon. [2014-11-13 21:59:04] We have some decisions. [2014-11-13 21:59:35] And we could look in more detail at the concentrator onwards stuff next time, including long-term persistent storage formats. [2014-11-13 22:00:12] Mike, may I finally publish a draft of your spec as a link from today's decision? [2014-11-13 22:00:21] Pretty please. [2014-11-13 22:00:23] ? [2014-11-13 22:00:34] you can publish that app protocol one but not the mac because it's changed drastically [2014-11-13 22:00:49] DamonHD: the SenML JSON is compact but only allows a limited set of units [2014-11-13 22:00:56] I'll probably transfer the app protocol onto a wiki somewhere so that we can collectively flesh it out [2014-11-13 22:01:36] Please can you tell me the URL for the bit that I can take a draft copy of for the record? [2014-11-13 22:01:40] And I'll do that now. [2014-11-13 22:01:50] Though maybe the MAC is more pressing! never mind... [2014-11-13 22:01:57] Bruno: OK. [2014-11-13 22:02:08] http://www.mike-stirling.com/files/tinyhan_app.pdf [2014-11-13 22:02:16] Does it support the UCUM or a subset of it? [2014-11-13 22:02:25] the mac is pretty much done [2014-11-13 22:02:25] Thanks Mike. [2014-11-13 22:02:47] As soon as you have it.... ! [2014-11-13 22:03:11] DamonHD: yes it does support UCUM as an extension: you have to prefix the UCUM unit with "UCUM:" [2014-11-13 22:03:28] which may be fine for a verbose format [2014-11-13 22:03:34] OK. [2014-11-13 22:03:59] Let's pick that up and security at this and the leaf<-->conc level next time. [2014-11-13 22:04:17] TimSmall? You there still? [2014-11-13 22:05:33] :x [2014-11-13 22:05:44] (vi leakage!) [2014-11-13 22:06:21] just to reinforce that point about avoiding meaningless values on the air interface - if you force (or at least strongly encourage) the use of real units then we will almost certainly improve likelihood of achieving interoperability [2014-11-13 22:06:26] shame [2014-11-13 22:06:32] I hear you. [2014-11-13 22:06:51] But if you overconstrain then you may prevent some legit uses entirely. [2014-11-13 22:06:54] So it's a balance. [2014-11-13 22:06:57] see HDMI CEC, DLNA, CBUS (in the context of boilers) for examples of vendors "taking advantage" of vendor proprietary namespaces [2014-11-13 22:07:28] I do hear you loud and clear, btu I also want to send encrypted binary if required, ultimately. [2014-11-13 22:08:07] yes, but encryption doesn't preclude interoperability either as long as you can do a key exchange [2014-11-13 22:08:14] cf wifi [2014-11-13 22:08:24] Encrypted so that even the hub cannot decode it. [2014-11-13 22:08:28] Ie end-to-end. [2014-11-13 22:08:50] for a closed system that would be fine [2014-11-13 22:09:06] It's one legit use of an otherwise open system. [2014-11-13 22:09:19] Remember the VMS filesystem type wars vs UNIX byte streams? [2014-11-13 22:09:27] No single right answer.