Earth Notes: On Website Technicals (2023-04)Updated 2023-05-16 07:07 GMT.
By Damon Hart-Davis.
do it, do it right, do it better, and I will be on stage 1 for quite a while!
2023-04-30: Sluggish Filesystem
The RPi that I use to talk to the Thermino heat battery has had its filesystem get quite full, partly is seems logging of the relentless break-in attempts via
ssh now that the system is reachable from outside:
% df -h . Filesystem Size Used Avail Use% Mounted on /dev/root 14G 14G 278M 98% /
Thanks to the nice clear explanation in
How to Clear Systemd Journal Logs
I ran a selection of command as below (most output not shown):
% journalctl --disk-usage Archived and active journals take up 1.4G in the file system. % sudo journalctl --rotate % sudo journalctl --vacuum-time=2d % sudo journalctl --vacuum-size=100M sudo vi /etc/systemd/journald.conf sudo systemctl restart systemd-journald
I edited into
to keep future log size in reasonable bounds.
To then banish the slugs I ran
sudo fstrim -av
which released more than the equivalent of the entire journal space, but took a while!
I will probably build the
journald.conf update into the
2023-04-23: MIDI Format 1 Disaster
Garage Band did not seem to be paying attention to anything other than the first channel in my first track for my automated sonification of my 'consolidated' kWh data. (I have been optionally using an extra track for data-independent percussion.)
It seems that for MIDI format 1 (as I have been using)
MIDI commands are assigned to more tracks, each track should have not more than one channel. First track is reserved to conductor.
Oh dear! I have been holding it wrong! GarageBand seems to be right... This also maybe explains why LMMS split my first track into several.
After examining a couple of big MIDI files (each from a pop song, one format 0 the other format 1), and playing with Signal - Online MIDI Editor, my new strategy is to generate MIDI format 1, one track (and channel) per voice, plus a leading 'control'/tempo track and an optional percussion track. This will require giving up any pretence of 'streaming' the data, instead making one pass over the data for each data stream to make its own track.
(I also tried making a format 0 track with each voice in its own channel, which worked, but folding in complex percussion beside that would have been horrible!)
2023-04-22: Mildly Augmented
Alongside the simple/bare MIDI files generated from consolidated kWh data, generated with V3 of the conversion script, I am now exposing versions 'mildly augmented' with very basic percussion and some time alignment (January to start each bar in monthly data) using V4.
V4 is still being hacked at and thought about, so that new set of sounds may be more volatile and flaky (and interesting).
2023-04-17: Bitwig et Al
A friend said
You strike me as more of a bitwig person ... so I downloaded the 30-day free trial of 4.4.10.
And managed to get a complex MIDI playing within a minute or so. Though that was having being primed by Ableton to need to 'activate audio' and similar such unexpected steps. But getting audio out at all is hard.
(In LMMS, once I had downloaded a
.sf2) file as it told me to, I could play MIDI files! Panning per voice worked too. I could even export MIDI.)
I could also try the 90-day free trial of the GarageBand upgrade to Logic Pro, though a licence would then be ~£200. Exporting MIDI is one feature.
For monthly-cadence consolidated data I now force alignment so that the first data/melody note in the bar is
-01 ie January, with the bar being 12 notes (thus a year).
I'm trying to make the programmatically-generated MIDI from data sound more 'interesting'.
To that end, today's lesson has been how to add panning (by channel). Here I set up three (flute solo) channels, spread out left/right:
1, 0, Program_c, 0, 79 1, 0, Control_c, 0, 10, 21 1, 0, Program_c, 1, 79 1, 0, Control_c, 1, 10, 63 1, 0, Program_c, 2, 79 1, 0, Control_c, 2, 10, 105
I'm also annotating the generated MIDI with key timestamps from the input, eg on the first month of each year:
1, 480, Marker_t, "2010-01"
I'm trying Ableton Live 11 free for 90 days. It is much more complex than GarageBand, and it took me about half enough to get a MIDI playing at all! (It turns out that until you attach a set of instruments to a MIDI track, you have the sound of no hands clapping...)
(on my Mac with
brew, on the RPi with
apt-get) to allow command-line automated conversion of
MIDI to audio, eg
% timidity input.mid -OF -o out.flac
MP3 format is not supported, but, for example,
FFmpeg can convert from
sox utility is another possibility.
Since at least the
MIDI files that I am generating seem to be significantly compressible, I have added to Apache's auto-
deflate list both the
mid filename suffix and the
% ls -al data/consolidated/energy/std/flow/M/flow-M.* 3588 data/consolidated/energy/std/flow/M/flow-M.csv 1502 data/consolidated/energy/std/flow/M/flow-M.mid % gzip -v9 < data/consolidated/energy/std/flow/M/flow-M.csv | wc -c 74.7% 924 % gzip -v9 < data/consolidated/energy/std/flow/M/flow-M.mid | wc -c 78.9% 335
At some point I might want to add a little more song structure to the sonifications, eg 'verse' and 'chorus' sections at different time resolutions, such as monthly vs yearly.
2023-04-11: awk function
Today I wrote my first
awk function. It contains percussion generation so that I can share it between generative music projects.
Yes, in all the 30+ years that I must have been writing
awk code I do not think that I have written a function in it!
2023-04-10: Apache and CC Licences
I just inserted a minimal Apache 2.0 licence statement in all the
.sh scripts under
scripts/mkaudio as below, just below any 'usage' section:
########## # May be used / adapted / etc without any promise of fitness for purpose # under the terms of the Apache License Version 2.0, January 2004 # http://www.apache.org/licenses/LICENSE-2.0 ##########
I hope that I did not break anything!
I also put most of my SoundCloud tracks under a Attribution 3.0 Unported (CC BY 3.0) licence, for which the only restriction is basically attribution (which, as for my Apache 2.0 code, I am happy with 'where reasonable'). I have removed even the attribution element for some trivial sounds.
Each such file may get its own player and visualiser.
I intend to generate plain and 'house' versions of each series
Initially I have just the very simplest bare/plain MIDIs being generated immediately after the
CSVs have been regenerated: 84 of them!
daily grid-tied solar generation from 2008 to now
generated from it, then in GarageBand switching the instrument to Japanese String Movement, generates a fairly
interesting and soothing piece, at over 15 minutes (~1 minute per year).
It's a weak joke, but I'm running with it. Sonifying house data as house music. Though this time with
MIDICSV rather than
The new script
monthly mains tap-water temperature samples for 16WW
into a simple
MIDI binary file (format 1, with 1 track). GarageBand is happy to open and play that file.
This replaces the previous
awk hack, and is much more likely to be generating a completely legal
file will be regenerated whenever a new data point is added.
The conversion plays 12 months (ie one year) into each 4/4 bar. This just about gets into bar 16 currently, with a grand total of 1691 bytes.
You may also have noticed recent entries in the
Sources/Links (of this and previous months) prefixed with
M3c. Those, in the same spirit as
A11y (accessibility) and
I18n internationalisation, etc, are for music. I am not short of weak jokes.
is a fork of
textToMIDIv2 to generate a simple house-style percussion track alongside.
out of this process can be heard in all its MP3-rendered glory:
Using a different GarageBand voice (Japanese String Movement) instead of flute:
I should probably use GarageBand's "Four on the Floor" percussion voice set, rather than the default "SoCal".
Listen to the podcast episode remix which does this!