Earth Notes: On Website Technicals (2023-02)
Updated 2023-05-16 06:46 GMT.By Damon Hart-Davis.
2023-02-05: FOSDEM, GraalVM Native
Today I started with the Friends of OpenJDK
(aka "Java") track. After lunch I switched to the lightning talks.
GraalVM Native
Prompted by the Java talks, I am interested to get an idea of what AOT (Ahead Of Time) compilation looks like. It was in OpenJDK for a while, but has gone again, and probably will not be fully back until ~2027!
GraalVM has a native-image
tool to create a fully-precompiled binary. This assumes that no features such as dynamic class loading by name are used. This VM is available for my Mac, but not for RPis older than 4 (and it seems as if 8GB RAM may be needed).
I installed V19 so:
% java -version openjdk version "19.0.2" 2023-01-17 OpenJDK Runtime Environment GraalVM CE 22.3.1 (build 19.0.2+7-jvmci-22.3-b12) OpenJDK 64-Bit Server VM GraalVM CE 22.3.1 (build 19.0.2+7-jvmci-22.3-b12, mixed mode, sharing)
I then installed the native-image
tool with the GraalVM updater:
% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/installer/bin/gu install native-image % /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/svm/bin/native-image ...
A first attempt to run this against the current optimised reutils
:
% /Library/Java/JavaVirtualMachines/graalvm-ce-java19-22.3.1/Contents/Home/lib/svm/bin/native-image -jar .work/edhMain.reutils-1.1.29.jar ================================================================================ GraalVM Native Image: Generating 'edhMain.reutils-1.1.29' (executable)... ================================================================================ [1/7] Initializing... (5.1s @ 0.19GB) Version info: 'GraalVM 22.3.1 Java 19 CE' Java version info: '19.0.2+7-jvmci-22.3-b12' C compiler: cc (apple, arm64, 12.0.5) Garbage collector: Serial GC [2/7] Performing analysis... [******] (17.5s @ 1.66GB) 5,010 (78.53%) of 6,380 classes reachable 8,106 (58.03%) of 13,969 fields reachable 23,583 (48.82%) of 48,310 methods reachable 163 classes, 80 fields, and 739 methods registered for reflection 58 classes, 58 fields, and 52 methods registered for JNI access 4 native libraries: -framework Foundation, dl, pthread, z [3/7] Building universe... (2.1s @ 2.60GB) [4/7] Parsing methods... [**] (2.2s @ 1.61GB) [5/7] Inlining methods... [***] (1.3s @ 2.40GB) [6/7] Compiling methods... [****] (15.2s @ 3.91GB) [7/7] Creating image... (3.1s @ 0.83GB) 10.34MB (45.99%) for code area: 14,471 compilation units 11.61MB (51.63%) for image heap: 150,478 objects and 5 resources 546.67KB ( 2.37%) for other data 22.48MB in total -------------------------------------------------------------------------------- Top 10 packages in code area: Top 10 object types in image heap: 822.49KB java.util 2.17MB byte[] for code metadata 641.95KB sun.font 1.48MB java.lang.String 497.89KB java.lang.invoke 1.17MB byte[] for general heap data 413.55KB com.sun.crypto.provider 1.10MB java.lang.Class 389.12KB java.lang 927.94KB byte[] for java.lang.String 336.06KB java.text 391.41KB c.o.s.c.h.DynamicHubCompanion 281.64KB sun.java2d.marlin 280.09KB java.lang.String[] 274.51KB com.sun.imageio.plugins.jpeg 271.27KB j.u.c.ConcurrentHashMap$Node 270.15KB java.awt.image 261.73KB java.lang.Object[] 264.28KB java.util.concurrent 250.08KB java.util.HashMap$Node 6.17MB for 178 more packages 2.42MB for 1063 more object types -------------------------------------------------------------------------------- 1.1s (2.2% of total time) in 21 GCs | Peak RSS: 5.06GB | CPU load: 5.60 -------------------------------------------------------------------------------- Produced artifacts: /Users/dhd/SVN-controlled/www.hd.org/Damon/Env/edhMain.reutils-1.1.29 (executable) /Users/dhd/SVN-controlled/www.hd.org/Damon/Env/edhMain.reutils-1.1.29.build_artifacts.txt (txt) ================================================================================ Finished generating 'edhMain.reutils-1.1.29' in 49.0s. % ls -al edhMain.reutils-1.1.29 -rwxr-xr-x 23576531 edhMain.reutils-1.1.29
My first attempt to run failed for a couple of reasons:
../edhMain.reutils-1.1.29 trafficLights ~/tmp/test.html INFO: generating traffic-light summary [/Users/dhd/tmp/test.html]... ERROR: could not fetch data from null error: Accessing an URL protocol that was not enabled. The URL protocol https is supported but not enabled by default. It must be enabled by adding the --enable-url-protocols=https option to the native-image command. ... FAILED command: trafficLights java.lang.NoSuchMethodError: org.hd.d.edh.D.a() at org.hd.d.edh.v.a(FUELINSTUtils.java:933) at org.hd.d.edh.f.a(FUELINST.java:377) at org.hd.d.edh.Main.main(Main.java:131)
So I added the --enable-url-protocols=https
argument as above, and the new output is:
% ls -al edhMain.reutils-1.1.29 -rwxr-xr-x 1 dhd staff 34150851 5 Feb 13:50 edhMain.reutils-1.1.29
That now fixes the first problem, ie the HTTPS connection works to fetch data:
(cd .work && ../edhMain.reutils-1.1.29 trafficLights ~/tmp/test.html) INFO: generating traffic-light summary [/Users/dhd/tmp/test.html]... INFO: record/row count of CSV FUELINST data: 1926ms, 288 records from source: https://www.bmreports.com/bmrs/?q=ajax/csv_download/FUELINST/csv/&filename=GenerationbyFuelType24HrInstantaneous fetch and parse INFO: CHECKPOINT: data fetched etc: timestamp: 1952ms. WARNING: could not load long store /Users/dhd/tmp/test.longstore.csv.gz error: java.io.FileNotFoundException: /Users/dhd/tmp/test.longstore.csv.gz (No such file or directory) INFO: CHECKPOINT: 24h summmary computed: timestamp: 1960ms. ERROR: missing directory for icons: ../out/hourly/button/ INFO: 24h summary: FUELINST.CurrentSummary:status=YELLOW:recentChange=RED:timestamp=20230205135000:currentMW=29459:currentIntensity=141:currentStorageDrawdownMW=292:histMinIntensity=76:histAveIntensity=120:histMaxIntensity=166:minIntensityRecordTimestamp=20230205044500:maxIntensityRecordTimestamp=20230204142000:histWindowSize=86100000:histSamples=288:lowerThreshold=91:upperThreshold=155:totalGridLosses=0.07:histAveIntensityByHourOfDay=80,85,89,79,78,89,102,101,102,118,133,139,140,162,156,160,162,160,157,156,141,117,100,87:computeTime_ms=8 ...
Note that running the GraalVM JIT yields computeTime_ms=37
(on a single test run) and 227ms with OpenJDK 18 JIT vs the 8ms shown in the last line of the above output, so already promising.
Activating the kill switch that I have just added ahead of Twitter shutting down its free API access next week allows the process to complete without error. And in about half the time of the java
JIT
, though not generating the icons/buttons. Note also that the converted non-obfuscated JAR runs without this kill switch, so this issue presumably could be fixed with a ProGuard configuration change.
Creating a directory for the buttons reveals a new error:
ERROR: could not create/save buttons, error: java.lang.UnsatisfiedLinkError: no awt in java.library.path
So there would be more work to do, and I cannot deploy this on the RPi server where it would be useful. But this test drive has been very interesting, and it was easy to get a long way!
CDS Archive
Maybe I should look at using a CDS archive to speed up the existing JAR on the RPi, eg with JDK 19 onwards:
java -XX:+AutoCreateSharedArchive -XX:SharedArchiveFile=my.jsa -cp my.jar Hello
2023-02-04: FOSDEM, Energy Track
Given Brexit and covid, it is years (maybe seven?) since I was last at FOSDEM in ULB Brussels in person.
This time both my (Fairphone 3) phone and (MacBook Air M1 macOS 13.2 Ventura) laptop managed to connect via the IPv6-only FOSDEM WiFi. Even the proxying to EOU's IPv4-only servers works.
macbook% ifconfig -a ... en0: flags=88e3<UP,BROADCAST,SMART,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1500 options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM> ether 80:65:7c:e5:de:c3 inet6 fe80::e:3fa8:d889:7622%en0 prefixlen 64 secured scopeid 0xc inet6 2001:67c:1810:f051:1068:b342:f764:a1a0 prefixlen 64 autoconf secured inet6 2001:67c:1810:f051:69d5:5d63:b30e:2cb9 prefixlen 64 autoconf temporary inet 192.0.0.2 netmask 0xffffffff broadcast 192.0.0.2 inet6 2001:67c:1810:f051:1832:c326:47d3:87a7 prefixlen 64 clat46 nat64 prefix 64:ff9b:: prefixlen 96 nd6 options=201<PERFORMNUD,DAD> media: autoselect status: active
I also have access to EDUROAM via my shiny new PhD research. Both networks seem to be wavering under load at 9am ish!
Access to the FOSDEM streaming server front-end seems about equivalent:
macbook% ping live.fosdem.org PING live.fosdem.org (78.46.229.231): 56 data bytes 64 bytes from 78.46.229.231: icmp_seq=0 ttl=52 time=18.409 ms 64 bytes from 78.46.229.231: icmp_seq=1 ttl=52 time=28.678 ms 64 bytes from 78.46.229.231: icmp_seq=2 ttl=52 time=19.863 ms 64 bytes from 78.46.229.231: icmp_seq=3 ttl=52 time=26.965 ms ^C --- live.fosdem.org ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 18.409/23.479/28.678/4.415 ms [localhost:~] dhd% ping live.fosdem.org ping: cannot resolve live.fosdem.org: Unknown host [localhost:~] dhd% ping live.fosdem.org ping: cannot resolve live.fosdem.org: Unknown host [localhost:~] dhd% ping live.fosdem.org PING live.fosdem.org (78.46.229.231): 56 data bytes Request timeout for icmp_seq 0 ... 64 bytes from 78.46.229.231: icmp_seq=13 ttl=54 time=27.866 ms 64 bytes from 78.46.229.231: icmp_seq=14 ttl=54 time=27.437 ms 64 bytes from 78.46.229.231: icmp_seq=15 ttl=54 time=27.124 ms 64 bytes from 78.46.229.231: icmp_seq=16 ttl=54 time=27.348 ms 64 bytes from 78.46.229.231: icmp_seq=17 ttl=54 time=27.513 ms 64 bytes from 78.46.229.231: icmp_seq=18 ttl=54 time=27.360 ms 64 bytes from 78.46.229.231: icmp_seq=19 ttl=54 time=37.254 ms ... ^C --- live.fosdem.org ping statistics --- 25 packets transmitted, 12 packets received, 52.0% packet loss round-trip min/avg/max/stddev = 27.124/28.968/37.254/3.388 ms(It seems, from the FOSDEM infrastructure review closing lightning talk, that Apple devices have some problem with IPv6-only WiFi and try and use IPv4 anyway sometimes, thus the intermittency I saw, while my Android Fairphone 3 seemed to be stable. Also the EDUROAM and FOSDEM services were running on the same WiFi infrastructure.)
I am starting, in the canteen over coffee, with the online-only morning FOSDEM Energy track. (Well, we did make it to the welcome talk at 09:30 at Jansen first!)
2023-02-02: Apache 2
To make it clear that people are allowed to crib anything they find useful in my
published scripts, I have both put a note in some key scripts and created a
LICENCE.txt
stating that the code is available under
Apache 2.0.