Earth Notes: On Website Technicals (2023-02)

Updated 2023-05-16 06:46 GMT.
By Damon Hart-Davis.
Tech updates: Apache scripts, FOSDEM energy and Java, GraalVM AOT.
tools 800w JA
At FOSDEM for the first time since 2016? So starting by creating an explicit open licence for most EOU scripts!

2023-02-05: FOSDEM, GraalVM Native

Today I started with the Friends of OpenJDK (aka "Java") track. After lunch I switched to the lightning talks.

GraalVM Native

Prompted by the Java talks, I am interested to get an idea of what AOT (Ahead Of Time) compilation looks like. It was in OpenJDK for a while, but has gone again, and probably will not be fully back until ~2027!

GraalVM has a native-image tool to create a fully-precompiled binary. This assumes that no features such as dynamic class loading by name are used. This VM is available for my Mac, but not for RPis older than 4 (and it seems as if 8GB RAM may be needed).

I installed V19 so:

% java -version
openjdk version "19.0.2" 2023-01-17
OpenJDK Runtime Environment GraalVM CE 22.3.1 (build 19.0.2+7-jvmci-22.3-b12)
OpenJDK 64-Bit Server VM GraalVM CE 22.3.1 (build 19.0.2+7-jvmci-22.3-b12, mixed mode, sharing)

I then installed the native-image tool with the GraalVM updater:

% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/installer/bin/gu install native-image
% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/svm/bin/native-image ...

A first attempt to run this against the current optimised reutils:

% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/svm/bin/native-image -jar .work/edhMain.reutils-1.1.29.jar
================================================================================
GraalVM Native Image: Generating 'edhMain.reutils-1.1.29' (executable)...
================================================================================
[1/7] Initializing...                                            (5.1s @ 0.19GB)
 Version info: 'GraalVM 22.3.1 Java 19 CE'
 Java version info: '19.0.2+7-jvmci-22.3-b12'
 C compiler: cc (apple, arm64, 12.0.5)
 Garbage collector: Serial GC
[2/7] Performing analysis...  [******]                          (17.5s @ 1.66GB)
   5,010 (78.53%) of  6,380 classes reachable
   8,106 (58.03%) of 13,969 fields reachable
  23,583 (48.82%) of 48,310 methods reachable
     163 classes,    80 fields, and   739 methods registered for reflection
      58 classes,    58 fields, and    52 methods registered for JNI access
       4 native libraries: -framework Foundation, dl, pthread, z
[3/7] Building universe...                                       (2.1s @ 2.60GB)
[4/7] Parsing methods...      [**]                               (2.2s @ 1.61GB)
[5/7] Inlining methods...     [***]                              (1.3s @ 2.40GB)
[6/7] Compiling methods...    [****]                            (15.2s @ 3.91GB)
[7/7] Creating image...                                          (3.1s @ 0.83GB)
  10.34MB (45.99%) for code area:    14,471 compilation units
  11.61MB (51.63%) for image heap:  150,478 objects and 5 resources
 546.67KB ( 2.37%) for other data
  22.48MB in total
--------------------------------------------------------------------------------
Top 10 packages in code area:           Top 10 object types in image heap:
 822.49KB java.util                        2.17MB byte[] for code metadata
 641.95KB sun.font                         1.48MB java.lang.String
 497.89KB java.lang.invoke                 1.17MB byte[] for general heap data
 413.55KB com.sun.crypto.provider          1.10MB java.lang.Class
 389.12KB java.lang                      927.94KB byte[] for java.lang.String
 336.06KB java.text                      391.41KB c.o.s.c.h.DynamicHubCompanion
 281.64KB sun.java2d.marlin              280.09KB java.lang.String[]
 274.51KB com.sun.imageio.plugins.jpeg   271.27KB j.u.c.ConcurrentHashMap$Node
 270.15KB java.awt.image                 261.73KB java.lang.Object[]
 264.28KB java.util.concurrent           250.08KB java.util.HashMap$Node
   6.17MB for 178 more packages            2.42MB for 1063 more object types
--------------------------------------------------------------------------------
    1.1s (2.2% of total time) in 21 GCs | Peak RSS: 5.06GB | CPU load: 5.60
--------------------------------------------------------------------------------
Produced artifacts:
 /Users/dhd/SVN-controlled/www.hd.org/Damon/Env/edhMain.reutils-1.1.29 (executable)
 /Users/dhd/SVN-controlled/www.hd.org/Damon/Env/edhMain.reutils-1.1.29.build_artifacts.txt (txt)
================================================================================
Finished generating 'edhMain.reutils-1.1.29' in 49.0s.
% ls -al edhMain.reutils-1.1.29
-rwxr-xr-x  23576531  edhMain.reutils-1.1.29

My first attempt to run failed for a couple of reasons:

../edhMain.reutils-1.1.29 trafficLights ~/tmp/test.html
INFO: generating traffic-light summary [/Users/dhd/tmp/test.html]...
ERROR: could not fetch data from null error: Accessing an URL protocol that was not enabled. The URL protocol https is supported but not enabled by default. It must be enabled by adding the --enable-url-protocols=https option to the native-image command.
...
FAILED command: trafficLights
java.lang.NoSuchMethodError: org.hd.d.edh.D.a()
        at org.hd.d.edh.v.a(FUELINSTUtils.java:933)
        at org.hd.d.edh.f.a(FUELINST.java:377)
        at org.hd.d.edh.Main.main(Main.java:131)

So I added the --enable-url-protocols=https argument as above, and the new output is:

% ls -al edhMain.reutils-1.1.29
-rwxr-xr-x  1 dhd  staff  34150851  5 Feb 13:50 edhMain.reutils-1.1.29

That now fixes the first problem, ie the HTTPS connection works to fetch data:

(cd .work && ../edhMain.reutils-1.1.29 trafficLights ~/tmp/test.html)
INFO: generating traffic-light summary [/Users/dhd/tmp/test.html]...
INFO: record/row count of CSV FUELINST data: 1926ms, 288 records from source: https://www.bmreports.com/bmrs/?q=ajax/csv_download/FUELINST/csv/&filename=GenerationbyFuelType24HrInstantaneous fetch and parse
INFO: CHECKPOINT: data fetched etc: timestamp: 1952ms.
WARNING: could not load long store /Users/dhd/tmp/test.longstore.csv.gz error: java.io.FileNotFoundException: /Users/dhd/tmp/test.longstore.csv.gz (No such file or directory)
INFO: CHECKPOINT: 24h summmary computed: timestamp: 1960ms.
ERROR: missing directory for icons: ../out/hourly/button/
INFO: 24h summary: FUELINST.CurrentSummary:status=YELLOW:recentChange=RED:timestamp=20230205135000:currentMW=29459:currentIntensity=141:currentStorageDrawdownMW=292:histMinIntensity=76:histAveIntensity=120:histMaxIntensity=166:minIntensityRecordTimestamp=20230205044500:maxIntensityRecordTimestamp=20230204142000:histWindowSize=86100000:histSamples=288:lowerThreshold=91:upperThreshold=155:totalGridLosses=0.07:histAveIntensityByHourOfDay=80,85,89,79,78,89,102,101,102,118,133,139,140,162,156,160,162,160,157,156,141,117,100,87:computeTime_ms=8
...

Note that running the GraalVM JIT yields computeTime_ms=37 (on a single test run) and 227ms with OpenJDK 18 JIT vs the 8ms shown in the last line of the above output, so already promising.

Activating the kill switch that I have just added ahead of Twitter shutting down its free API access next week allows the process to complete without error. And in about half the time of the java JIT, though not generating the icons/buttons. Note also that the converted non-obfuscated JAR runs without this kill switch, so this issue presumably could be fixed with a ProGuard configuration change.

Creating a directory for the buttons reveals a new error:

ERROR: could not create/save buttons, error: java.lang.UnsatisfiedLinkError: no awt in java.library.path

So there would be more work to do, and I cannot deploy this on the RPi server where it would be useful. But this test drive has been very interesting, and it was easy to get a long way!

CDS Archive

Maybe I should look at using a CDS archive to speed up the existing JAR on the RPi, eg with JDK 19 onwards:

java -XX:+AutoCreateSharedArchive -XX:SharedArchiveFile=my.jsa -cp my.jar Hello

2023-02-04: FOSDEM, Energy Track

Given Brexit and covid, it is years (maybe seven?) since I was last at FOSDEM in ULB Brussels in person.

This time both my (Fairphone 3) phone and (MacBook Air M1 macOS 13.2 Ventura) laptop managed to connect via the IPv6-only FOSDEM WiFi. Even the proxying to EOU's IPv4-only servers works.

macbook% ifconfig -a
...
en0: flags=88e3<UP,BROADCAST,SMART,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1500
        options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
        ether 80:65:7c:e5:de:c3
        inet6 fe80::e:3fa8:d889:7622%en0 prefixlen 64 secured scopeid 0xc
        inet6 2001:67c:1810:f051:1068:b342:f764:a1a0 prefixlen 64 autoconf secured
        inet6 2001:67c:1810:f051:69d5:5d63:b30e:2cb9 prefixlen 64 autoconf temporary
        inet 192.0.0.2 netmask 0xffffffff broadcast 192.0.0.2
        inet6 2001:67c:1810:f051:1832:c326:47d3:87a7 prefixlen 64 clat46
        nat64 prefix 64:ff9b:: prefixlen 96
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect
        status: active

I also have access to EDUROAM via my shiny new PhD research. Both networks seem to be wavering under load at 9am ish!

Access to the FOSDEM streaming server front-end seems about equivalent:

macbook% ping live.fosdem.org
PING live.fosdem.org (78.46.229.231): 56 data bytes
64 bytes from 78.46.229.231: icmp_seq=0 ttl=52 time=18.409 ms
64 bytes from 78.46.229.231: icmp_seq=1 ttl=52 time=28.678 ms
64 bytes from 78.46.229.231: icmp_seq=2 ttl=52 time=19.863 ms
64 bytes from 78.46.229.231: icmp_seq=3 ttl=52 time=26.965 ms
^C
--- live.fosdem.org ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 18.409/23.479/28.678/4.415 ms
[localhost:~] dhd% ping live.fosdem.org
ping: cannot resolve live.fosdem.org: Unknown host
[localhost:~] dhd% ping live.fosdem.org
ping: cannot resolve live.fosdem.org: Unknown host
[localhost:~] dhd% ping live.fosdem.org
PING live.fosdem.org (78.46.229.231): 56 data bytes
Request timeout for icmp_seq 0
...
64 bytes from 78.46.229.231: icmp_seq=13 ttl=54 time=27.866 ms
64 bytes from 78.46.229.231: icmp_seq=14 ttl=54 time=27.437 ms
64 bytes from 78.46.229.231: icmp_seq=15 ttl=54 time=27.124 ms
64 bytes from 78.46.229.231: icmp_seq=16 ttl=54 time=27.348 ms
64 bytes from 78.46.229.231: icmp_seq=17 ttl=54 time=27.513 ms
64 bytes from 78.46.229.231: icmp_seq=18 ttl=54 time=27.360 ms
64 bytes from 78.46.229.231: icmp_seq=19 ttl=54 time=37.254 ms
...
^C
--- live.fosdem.org ping statistics ---
25 packets transmitted, 12 packets received, 52.0% packet loss
round-trip min/avg/max/stddev = 27.124/28.968/37.254/3.388 ms
(It seems, from the FOSDEM infrastructure review closing lightning talk, that Apple devices have some problem with IPv6-only WiFi and try and use IPv4 anyway sometimes, thus the intermittency I saw, while my Android Fairphone 3 seemed to be stable. Also the EDUROAM and FOSDEM services were running on the same WiFi infrastructure.)

I am starting, in the canteen over coffee, with the online-only morning FOSDEM Energy track. (Well, we did make it to the welcome talk at 09:30 at Jansen first!)

2023-02-02: Apache 2

To make it clear that people are allowed to crib anything they find useful in my published scripts, I have both put a note in some key scripts and created a LICENCE.txt stating that the code is available under Apache 2.0.

~650 words.