[2014-12-11 21:15:43] But anyhow your wiki page looks sensible to me. [2014-12-11 21:15:49] well I can multitask on that [2014-12-11 21:16:08] yeah, I've got a partial implementation working but I need to do some more on the MQTT side [2014-12-11 21:17:40] How do the units get represented on the JSON side? [2014-12-11 21:17:47] Remind me... [2014-12-11 21:18:41] just by standardisation [2014-12-11 21:18:52] so maybe via a schema [2014-12-11 21:19:24] So not actually in the textual representation? Or do yo mean something else? [2014-12-11 21:19:36] the intention was to scale any fixed point values though [2014-12-11 21:20:00] so while the tinyhan spec states hundredths of degrees C as the unit for temperature, the hub could scale it to a float [2014-12-11 21:20:15] no, not in the textual representation [2014-12-11 21:20:37] if the field names are standardised then I think that's better done by reference to a schema of some kind [2014-12-11 21:20:55] Hmm. [2014-12-11 21:21:14] As it happens the temp sensors I'm using are all binary fractions BTW, [2014-12-11 21:21:39] yeah I was thinking about whether to use binary or decimal fixed point [2014-12-11 21:21:47] so don't have a nice representation in 100ths for ones with 4 bits after the binary point [2014-12-11 21:21:50] but that's a detail [2014-12-11 21:21:57] I'm using a thermistor so you can do either trivially depending on how you calculate the lookup table [2014-12-11 21:22:23] I'd like where possible to have units explicit in the JSON form (eg from UCUM) [2014-12-11 21:22:26] I landed on the side of decimal because it's easier to convert for representation on a local UI [2014-12-11 21:22:32] and conversion is easy anyway [2014-12-11 21:22:33] for reduced ambiguity and confusion [2014-12-11 21:22:52] Conversion is lossy in this case. [2014-12-11 21:24:01] it is lossy potentially, but not beyond the accuracy of the sensor so who cares [2014-12-11 21:24:40] there's plenty of profile space to have alternative representations even with the same tag [2014-12-11 21:25:26] the idea in the hub is to have a map table defined which as well as containing the table of tags also contains a scaling numerator and denominator, and maybe an offset [2014-12-11 21:25:56] so there can be multiple binary representations that map onto the same friendly JSON [2014-12-11 21:26:02] OK. [2014-12-11 21:26:26] But I think the idea of the SenML style of explicit units makes it more (human and archival) friendly. [2014-12-11 21:26:31] Or do you mean something else? [2014-12-11 21:27:00] which bit? [2014-12-11 21:27:16] ? [2014-12-11 21:27:54] do I mean something else about what? [2014-12-11 21:28:23] Alternative representations. [2014-12-11 21:28:39] How do I know what units were in use in some particular item. [2014-12-11 21:28:39] that would only be on the binary side [2014-12-11 21:28:44] Vs here for example: [2014-12-11 21:28:44] http://wiki.1248.io/doku.php?id=senml [2014-12-11 21:28:48] the conversion table has to be hard coded into the hub [2014-12-11 21:28:50] The units are explicit. [2014-12-11 21:29:05] Converted into what in the JSON? [2014-12-11 21:29:28] Another number whose scaling is known only to the code and the given version of the TinyHAN spec that it is using? [2014-12-11 21:29:39] whatever we define [2014-12-11 21:30:00] I'm not understanding how I'll know what the number 42 means in 10 years' time in some generated JSON. [2014-12-11 21:30:05] the hub could output senml, but it needs to know the original and output units so it can do the conversion [2014-12-11 21:30:36] because it will be "temperature": 42 and the schema will define "temperature" as degrees C [2014-12-11 21:31:17] Do you anticipate *ever* changing the scale factor of any of those units? [2014-12-11 21:31:26] there's no reason why the hub couldn't generate senml with units, but the output is always going to be scaled to the same units so it seems a bit wasteful to output that data all the time [2014-12-11 21:31:58] no. scaling should be done in the origin sensor because it is the one that has things like cal factors and knowledge of its specific sensor [2014-12-11 21:32:05] Because it it could ever happen, explictly putting the UCUM unit in the JSON may be more robist. [2014-12-11 21:32:29] The scaling I;m taking about it reprsentational. [2014-12-11 21:32:56] Ie do I store as milli volts or centi volts or volts? [2014-12-11 21:33:10] store / represent in JSON [2014-12-11 21:33:34] whatever the schema says. if you want to change it then update the schema [2014-12-11 21:34:12] But then how do I know what older data meant? [2014-12-11 21:34:14] I assume you're not planning on storing it in uncooked json anyway? [2014-12-11 21:34:19] that wouldn't be much use [2014-12-11 21:34:46] and you shouldn't really be changing the units of a dataset half way through either [2014-12-11 21:34:56] again it would make it a bit of a useless data set [2014-12-11 21:35:03] But I may record stuff for 10 years. [2014-12-11 21:35:08] Things may change in that time. [2014-12-11 21:35:20] I may gone and look at it in another ten years' time. [2014-12-11 21:36:09] well you're going to have done a conversion step anyway aren't you? e.g. you'll parse the json and put the individual fields into a database [2014-12-11 21:36:26] you aren't going to store all the identical unit data in that anyway [2014-12-11 21:36:41] and since you're doing a processing step you can convert it into whatever unit you like [2014-12-11 21:38:14] Not necessarly. [2014-12-11 21:38:28] We were talking about the persistability. [2014-12-11 21:38:51] I would like it to be possible to wrap it up like this... [2014-12-11 21:38:56] [ "2014-12-11T21:35:45Z", "concID", {"@":"819c","T|C16":131,"L":28,"B|cV":256} ] [2014-12-11 21:39:04] and be done with it until later. [2014-12-11 21:39:33] This is not to exclude anything that you are talking about. [2014-12-11 21:40:12] I just think that slightly more self-describing data with be more robust in circs when not everything thinks ahead or is prepared to renormalise archives, etc [2014-12-11 21:41:35] will be [2014-12-11 21:41:47] the thing is, that json string is totally meaningless to me so in what way is that future proof? [2014-12-11 21:42:13] it might as well be binary [2014-12-11 21:42:16] With a timestamp and a concentrator ID and (some) explicit units [2014-12-11 21:42:29] and the ability to reparse it on demand later [2014-12-11 21:42:31] it would be about as easy to reverse engineer (timestamp excepted) [2014-12-11 21:42:39] I think it's somewhat better than binary. [2014-12-11 21:42:57] The timestamp is a minimum to interpret old data for example [2014-12-11 21:43:31] the timestamp definitely needs to be present [2014-12-11 21:44:23] but if you're going to use something text-based it might as well be readable. There are no resource constraints at this point in the system [2014-12-11 21:44:43] just using senml (preferably with readable field names) would be better [2014-12-11 21:46:09] But as in this case I'd liek to be able to store it untransformed directly from by sensor (plus concentrator timestamp and ID) for interpretation later. [2014-12-11 21:46:34] that's a separate issue [2014-12-11 21:46:55] Why? [2014-12-11 21:47:16] because it won't be carried in a TAP packet [2014-12-11 21:47:24] But in any case I'd like it to not be incompatible with whatever you were expecting to generate at the bridge. [2014-12-11 21:47:39] there was doing to be a separate "pass through" packet type for that [2014-12-11 21:47:52] the hub can send all those wherever you like, or write them to sd card, or whatever [2014-12-11 21:48:04] And I may like to store your generated JSON in the same way for example. [2014-12-11 21:48:05] or publish them all to a designated mqtt topic [2014-12-11 21:49:40] Goodness, getting all the detail right on this will be interesting! [2014-12-11 21:50:01] well there's a natural format conversion at the hub so you can output whatever format you like, but I will be feeding data directly into timestore so it needs to be in a pre-scaled format [2014-12-11 21:50:41] I see the OEM guys are doing there own thing on heating control as well [2014-12-11 21:50:45] their* [2014-12-11 21:51:21] Yep. [2014-12-11 21:51:56] OK, well you being anchored by Timestore gives it some more solidity, which is good. [2014-12-11 21:53:09] I don't believe storing the raw json for anything other than short term debugging is good system design anyway. JSON is not a database [2014-12-11 21:53:48] Nope, but it can be a long-term archive format. [2014-12-11 21:53:55] And a streaming format. [2014-12-11 21:54:10] Which is good for many use cases. [2014-12-11 21:54:57] (Is it windy up by you? Getting that way here...) [2014-12-11 21:56:08] sure, timestore generates it at the other end as well, but the idea of trying to keep the data unmodified all the way from node to data centre just because it makes it "IoT" is daft imho. There is nothing wrong with using the right tool for the job at each stage of the system [2014-12-11 21:56:17] Nope. [2014-12-11 21:56:29] That's not why I'm wanting to do it at all. [2014-12-11 21:56:31] it's calmed down a bit here - we had it yesterday [2014-12-11 21:57:15] It's to get data out and archivable with minimum work at the leaf node in terms of custom coding, as far as possible. [2014-12-11 21:57:25] For me now mostly it just looks like this for example: [2014-12-11 21:58:37] ... [2014-12-11 21:58:37] ss1.put(AmbLight); [2014-12-11 21:58:37] ss1.put(Supply_mV); [2014-12-11 21:58:37] ss1.put(NominalRadValve); [2014-12-11 21:58:37] ... [2014-12-11 21:58:37] where all those things are of type 'Sensor' (so come with tags/units and values). [2014-12-11 21:59:31] that's fine, but it would be far more code efficient just to stick to a standard [2014-12-11 21:59:46] And I haven't touched the hub node once though I have made dozens of changes at the leaves and in my use of the data downstream. [2014-12-11 22:00:14] If it can know in advance all the things that it might need to do, and if everyone is capable of coding to it. [2014-12-11 22:00:56] Your 'standard' us one good one, and ASCII subset of JSON with tags/units is another, fatter one. [2014-12-11 22:00:59] is [2014-12-11 22:01:04] this is roughly equivalent in the binary protocol: [2014-12-11 22:01:04] tinyapp_init(&tap, TINYAPP_FLAGS_NONE, buf, sizeof(buf)); [2014-12-11 22:01:04] tinyapp_serialise16(&tap, tinyappSystem_Battery, battmv); [2014-12-11 22:01:04] tinyapp_serialise8(&tap, TINYAPP_ITEM_ID_PROFILE, tinyappProfile_Environmental); [2014-12-11 22:01:04] tinyapp_serialise16(&tap, tinyappEnvironmental_Temperature, temp); [2014-12-11 22:01:05] tinyapp_serialise16(&tap, tinyappEnvironmental_IlluminanceVisible, light); [2014-12-11 22:01:34] only that doesn't rely on a ton of extra C++ code you conveniently ignore [2014-12-11 22:01:41] Not a ton. [2014-12-11 22:02:05] Actually smaller than printf and the (nice) ArduinoJson libs. [2014-12-11 22:02:13] And does some other useful stuff on the side. [2014-12-11 22:02:34] But not all nodes have to support JSON at all. [2014-12-11 22:02:44] (I have it turnable off in OpenTRV code for example.) [2014-12-11 22:03:01] But it would be good if they could all support your AP for example. [2014-12-11 22:05:05] Thanks for another useful chat. [2014-12-11 22:05:48] (Would it be worth adding something about bounds checking to your wiki page to cover Bruno's point?) [2014-12-11 22:06:05] no, that's not part of the protocol [2014-12-11 22:06:28] (As a normative note, ie safe usage implies doing this checking.) [2014-12-11 22:06:34] possibly [2014-12-11 22:06:43] that's a given though really [2014-12-11 22:06:50] ... [2014-12-11 22:07:01] I think it's useful to remind people. [2014-12-11 22:07:02] safe usage of anything coming from elsewhere implies you take nothing for granted [2014-12-11 22:07:39] Security slips are easy to make. Flagging up implementation issues that may have security consequences is helpful.