Earth Notes: On Setting Up a Raspberry Pi 3 B+ Off-grid Server

As of 2018-07-13 I thought that my stand-alone Technicolor TG582n router had died, though it seemed OK again after several hours powered off. But it consumes ~8W and crashes about once per month. Also, since it is talking PPPoE (PPP over Ethernet) to a BT Openreach ECI Telecom ON316150 FTTC modem it doesn't need to do anything analogue, so the routing and filtering done by the Technicolor ought to be do-able by a Pi. Also the Pi 3 has on-board WiFi that can be used as an access point.

Another advantage of an upgrade to the Pi 3 B+ is a bit more speed, and a bigger range of power draw to make better use of available sunshine, etc. Though I have to see if the base consumption of a stripped back Pi 3 B+ is still OK.

I can also take the opportunity to buy a larger SD card to boost storage, and effectively keep the old SD card as backup. (Currently the RPi2 has a 128GB micro SD card.)

Yet another advantage of an upgrade is moving to a newer OS distribution, allowing access to a newer Apache, HTTP/2, and various other goodies.

So an RPi 3 Model B+ and mains and 12V power supplies and other goodies have been ordered from RS to get started.

I'll record below as I prototype various aspects of PPPoE networking, new HTTP/2 support, etc.

I anticipate a fair period of prototyping and testing before (re)building the production server.

Start

Starting with just the following RS parts, no keyboard or mouse or screen plugged in, and waiting for it to boot, settles to 2.5W power consumption measured at the mains.

Directly plugging in the USB keyboard and mouse from a previous Maplin Raspberry Pi kit (R45Pi) raises consumption to 3.4W. Plugging in HDMI doesn't add any further load/consumption.

Have logged in and changed the default password for user 'pi'.

The Wifi does not seem to be in use to grab an IP (with DHCP) and get an Internet connection. This may be normal for NOOBS.

Via the Preferences menu entry and the Raspberry Pi Configuration tool I have:

• Set the system to boot to CLI rather than desktop.
• Set GPU memory to minimum (16MB).
• (NOT enabled SSH yet.)
• Set WiFi country to GB.

I am being asked to reboot for some of those to take effect.

I am still being auto-logged-in, which on the console is probably OK.

ifconfig now shows wlan0, though not with an IP address.

uname -a shows ... 4.9.80-v7+ #1098 Fri Mar 9 19:11:42 GMT 2018 armv71 ...

Following Setting WiFi up via the command line I sudo raspi-config and in "Network Options" enter SSID and passphrase. I exit raspi-config and ... I have an IP address on wlan0 and can ping the outside world!

Security dictates that the very next thing to do is sudo apt-get update && sudo apt-get-dist-upgrade.

Space used in the root partition is ~1GB.

With all that finished, power consumption (measured at the mains) is 3.1W. (Unplugging the USB mouse seems to push that up to 3.2W!)

I have established that the RPi3 and WiFi are working. Also that power consumption is not outrageous even before any attempts to tune it.

PPPoE, AP, Downstream

The next step may be to load up a minimal Raspbian, and (briefly) get the RPi3 talking PPPoE to my ISP and providing a WiFi Access Point. This would probably have to be a quick temporary swap-out with the current server, as none of the other services would be there, so there will be HTTP and mail server fails while the RPi is in place.

I also have to provide down-stream wired routed (and NATted) Ethernet connections for devices that need it, such as my Loop and SmartThings hubs, and my new storage system. They may need some protection against external attack, and possibly against being used to attack other things inside the house. So I'll need to adjust my iptables config.

As a first pass attempt to support the two downstream wired devices I have bought an AX88179-based USB-to-dual-Ethernet dongle (VTOP USB 3.0 Gigabit RJ45 10/100/1000Mbps Network Lan Adapter 2 Port) which both my Mac and the RPi3 appear to see immediately. I can't find a published power-consumption figure, so I will take some measurements in due course. None of the connections need run very fast, since the traffic from the devices is essentially only buffered telemetry. For the RPi3 the new ports show up as eth1 and eth2, so for robust and secure configuration of that and the on-board port I may need a UUID or other more robust identifier.

Storage

On 2018-10-12 I ordered a 256GB Class 10 U3 micro SDXC memory card (Samsung MB-MC256GA Micro SDXC EVO PLUS Class 10 UHS-I U3, Amazon). The price was a little over £50+VAT.

My choice was partly informed by Raspberry Pi Dramble's microSD Card Benchmarks.

Looking at the root partition size given as 235GB, I suspect that there is some trailing space at the end of the device not being used. Worth investigating later to see if the partition can be grown a little. (Subsequent inspection with fdisk revealed only a couple of small unused slivers.)

OS Download

Given that the "... "lite" download is about half that size, for the entire OS. The full fat version is over 4GB" (Ken Hagan), I'm inclined to start with "lite" to save some of the new GB!

So, from the Raspian download page I am pulling the "Raspbian Stretch Lite" "Minimal image based on Debian Stretch" with release date 2018-10-09 (kernel 4.14, 368MB, SHA-256 98444134e98cbb27e112f68422f9b1a42020b64a6fd29e2f6e941a3358d171b4) as a ZIP file.

For lolz, I'm trying Etcher as suggested on the Raspbian page. Nice and simple, but warned me that 256GB was "unusually large" for it to Flash. I told it to go ahead anyway. The process was fast, and the card was left unmounted afterwards.

Restart

2018-10-20: I inserted the "Lite" microSD card into the RPi3, connected up keyboard, plugged HDMI into the family TV, and powered up.

Some items noted during boot:

• A message appeared (somewhere, now off-screen!) about the (root) partition being resized. (Looking at dmesg and df -h output shows that there is a small /boot partition of 44MB and a huge ext4 root partition of 235GB. OK, though previously I have had a separate /local with most of the actual data in it. It looks like there are plenty of inodes with 15M compared to ~360k used in RPi2 main filesystem.)
• "Started LSB: Autogenerate and use a swap file." (I'll likely want to run swapless again, with some zram. There appears to be a 100MB swap file in /var/swap.)
• SSH host keys have been regenerated.
• Bluetooth service has been enabled (I'll most likely want to turn that off for security and power reasons).
• A message says that "Wi-Fi is disabled because the country is not set. Use raspi-config to set the country before use."

I logged in as pi and changed the password. I should now be safe to connect up to the Intartubes and get updates, for example.

I have changed the fstab parameters of the root filesystem to be defaults,noatime,commit=120 as on the RPi2 to reduce write traffic and improve performance, and rebooted. I may further raise the commit interval to 300s as on the RPi2's /local partition.

I have set the hostname to sencha with raspi-config.

I have set the Wi-Fi country to GB with raspi-config, connected to the Net with the correct SSID and password by editing /etc/wpa_supplicant/wpa_supplicant.conf with a 'network' entry with 'ssid' and 'psk' items.

Then apt-get update and apt-get dist-upgrade to be up to date on security. At this moment, no updates were needed.

I have set up NTP with apt-get but have not yet fully configured it (ie accepting defaults for now).

At this moment, power consumption from the mains, with HDMI on, is shown as ~3.2W. The target is to get that as near as possible to 1W when idling without HDMI.

Avoiding need to use the Family TV!

2018-10-21: I have fixed the IP address handed out by the router to the RPi3 with DHCP. This will mean that the RPi3 has a stable LAN IP address. That in turn should make accessing the RPi3 via ssh easier, thus avoiding conflict over use of the family TV as console!

I also need to allow SSH to run on the RPi3 (with raspi-config), still at this point protected behind the firewall/NAT.

(Experimentally turning off HDMI at this point made no visible difference to power draw, still fluctuating around ~3.2W; similar to the original RPi experience. Turning off the red LED temporarily with echo none > /sys/class/leds/led1/trigger doesn't make a visible difference to mains consumption either. Doing this however may disable undervoltage detection by the system, which would be bad.)

I have created my user ID on the RPi with the same uid. I can rsync stuff across, repeating as I get close to bringing the RPi3 live.

Per making HTTP/2 work better I have added the following lines to /etc/sysctl.conf and rebooted:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_notsent_lowat = 16384

Incidentally, reboots seem much faster than on the RPi2, <30s, possibly because of systemd which seems to be in this Rasbpian release.

Added main a/c to sudoers, and will disable pi a/c in due course.

Installed the Apache HTTP/Web server (apt-get install apache2). It looks as if it may be bringing TLS (HTTPS) support with it. (It serves pages claiming to be Apache 2.4.25 (Raspbian).)

The following additional packages will be installed:
  apache2-bin apache2-data apache2-utils libapr1 libaprutil1
  libaprutil1-dbd-sqlite3 libaprutil1-ldap liblua5.2-0 ssl-cert
Suggested packages:
  www-browser apache2-doc apache2-suexec-pristine | apache2-suexec-custom
  openssl-blacklist

The event service model seems to be enabled by default, as a module, (mpm_event) and its for this low-traffic low-memory server all its parameters can be rougly halved, to more like the RPi2 worker values.

Other resource use may need pruning too.

For example, each Apache process+threads on the RPi3 shows a virtual memory size of 222MB compared to ~20MB on the RPi2 out of the box. Inserting the following in the Apache envvars file reduces the RPi3 vm size to more or less that of the RPi2:

# NPTL (modern Linux threads) defaults the thread stack size to the setting
# of your stack resource limit. The system-wide default for this is 8MB,
# which is waaay exaggerated when running httpd.
# 512kB should be more than enough (AIX manages on 96kB, Netware on 64kB).
ulimit -s 512

Performance

As a simple benchmark I run the command to (re)validate the generated HTML using the W3C Java tool (which examines all the desktop pages plus the mobile home page):

rm .work/tmp/pages.valid && time make .work/tmp/pages.valid

Wall clock time taken by my ~2014 MacBook Air is a little under 7s, the new RPi3 ~20s, and the RPi2 ~44s. (Some of this may be due to a newer faster Java (JDK8) on the RPi3.) But in any case this RPi3 is much nearer to laptop performance than the RPi2.

To-Do List

Roughly prioritised...

1. Set timezone to UTC and localisation to en-GB.
2. Networking:
   1. (Done) Temporarily give RPi3 stable LAN address for setup work.
   2. Temporarily have Pi as Wi-Fi AP and client during set-up to avoid needing the TV and as a recovery mode!
   3. Tighten sshd security, eg limit users allowed and forbid password-based logins.
   4. Set up PPPoE.
   5. Turn on anti-spoofing reverse path filters (net.ipv4.conf.default.rp_filter etc in /etc/sysctl.conf.
   6. Set up routing, NAT, DHCP and extended ipfilter/firewall local private LAN(s) wired and WiFi.
   7. Set up new local LAN with DHCP 'behind' RPi3; it may be possible to move Loop and SmartThings devices on to this new LAN well before rest of work is completed, and in any case they will buffer data through brief interruptions of connectivity.
   8. Possibly set up tc.
   9. Get USB/Ethernet multi-RJ45 dongle for Loop and SmartThings hub such as CU200.
   10. Install/configure BIND and DNS master.
3. Swap and filesystems:
   1. (Done) Get swappiness right (1?). The RPi2 /etc/sysctl.conf has vm.swappiness=1 and vm.min_free_kbytes = 8192.
   2. (Done) Enable zram as on RPi2 in /etc/rc.local with modprobe zram; echo 128128128 > /sys/block/zram0/disksize; mkswap /dev/zram0; swapon /dev/zram0
   3. (Done) Disable the ~100MB automatically-configured /var/swap swap file with swapoff /var/swap; systemctl disable dphys-swapfile; rm -f /var/swap.
   4. (Done) Reduce SD card traffic per RPi2 with /etc/sysctl.conf:

      vm.dirty_background_ratio=20
      vm.dirty_ratio=40
      vm.dirty_writeback_centisecs=1500
      vm.dirty_expire_centisecs=12101
      

   5. (Done) Increase ext4 filesystem commit time to 300s.
   6. (Done) See if there is unused space at the end of the root partition that it could be grown into... (235GB looks a bit low even allowing for overheads and /boot. Inspection with fdisk revealed only a couple of small unused slivers.)
   7. Adding following to fstab to reduce SD traffic: tmpfs /tmp tmpfs defaults,noatime,nosuid 0 0. (Also retrofitted to RPi2 config as visibly (green led0) useful in reducing RPi3's SD traffic during EOU site rebuild test.)
   8. Reduce syslog logging to reduce write traffic.
   9. Add a daily root cron task for ionice -c 3 fstrim -v / and other non-memory partitions as per ext3/4 and SD-card wear leveling.
4. Time:
   1. (Done) Install NTP FCS/vanilla config.
   2. (Done) Configure NTP per RPi2 such as bringing low-tier servers over and adding some defence against misuse. The restarted with systemctl restart ntp.
   3. Try again to join UK (or Debian) pool.ntp.org service; traffic was overwhelming before but faster connection and CPU may help now.
   4. Install/configure hardware clock (spare device, new battery?).
5. Power:
   1. (Done) Install/configure/enable ondemand governor.
   2. (Done) Turn off HDMI to save energy (~100mW) if no one logged in, per RPi2 rc.local.
   3. Set lower idling CPU clock to save energy (idle default apparently 600MHz, non-idle 1400MHz if not thermally throttled).
   4. Turn off Bluetooth to save energy (dtoverlay=pi3-disable-bt in /boot/config.txt or blacklist btbcm and blacklist hci_uart in /etc/modprobe.d/raspi-blacklist.conf and turn off Bluetooth services, eg sudo systemctl disable bluetooth and sudo systemctl disable hciuart?).
   5. Turn off unnecessary LEDs eg on network connector eg have green activity LED only show SD card activity when power OK or better, else off/'none'. (Turning off red power LED may disable low-voltage detection.)
   6. Find out if Ethernet port can save power in sub-Gbit mode, eg when storage is in a low- or very-low- power state. (ethtool -s eth0 speed [100|1000] duplex full: can probably safely keep at 100Mbps for PPPoE FTTC link, saving maybe ~500mW.)
   7. (Won't do: all USB ports will be in use!) Selectively turn off some of the USB ports.
   8. Construct 2.5A (10.25W) capable supply from 12V. Testing with a Muker 10VA USB Multimeter Charger Detector (UT-KWS-10VA) shows red power light goes out when current draw hits as little as ~730mA (3.8W) from a variety of nominal ≥2A USB supplies, and also MacBook Air USB port. Minimum/idle consumption immediately after boot ~2W.
   9. Turn off other unnecessary h/w.
6. Mail:
   1. (Done) Remove or disable exim4. (systemctl disable exim4.service.)
   2. Install and configure sendmail.
   3. Install and configure POP3 (dovecot?).
7. Apache:
   1. (Done) Install Apache.
   2. (Done) Tune Apache for small-system memory and performance (roughly halved event mpm params and much reduced per-thread stack size).
   3. (Done) Use/configure Apache event model: event seems enabled by default and simply needs configuring. Initial tweak of mpm_event done.
   4. Copy Web sites (uid, data, cron) across from RPi2.
   5. Install/configure Apache w/ HTTPS and HTTP/2 support.
   6. Configure Apache log rolling.
   7. Configure (Apache) Brotli support and static pre-geneneration.
   8. Capture (eg in SVN) config changes (eg for mpm_event).
8. Copy existing user accounts (uid, data, cron) across from RPi2.
9. Copy Gallery (uid, data, metadata, app, cron) across from RPi2.
10. Set up extra log dirs for SunnyBeam, powermng, Enphase under /var/log.
11. Copy power management code etc and move HATs.
12. Security (misc):
    1. Install support for h/w RNG (rgnd / hwrng) per RPi2.
    2. Disable pi a/c in due course.
13. SVN:
    1. Take snapshots of SVN repos, and archive some off-site.
    2. Disable RPi2 repo check-ins.
    3. Copy snapshots to RPi3 and unpack.
    4. Switch RPi3 remote use of RPi2 to local file:// access.
14. Tune boot time eg using systemd-analyze blame. Top time hogs (>1s) as at 2018-10-25 as a WiFi client are:
    1. 8.851s dhcpcd.service
    2. 6.776s hciuart.service
    3. 1.862s dev-mmcblk0p2.device
    4. 1.324s apt-daily-upgrade.service
    5. 1.098s exim4.service
15. Config to do and to capture explicitly in SVN:
    1. Copy appropriate rc.local features from RPi2 to RPi3 and capture in SVN.
    2. Capture (eg in SVN) /etc/sysctl.conf changes for TCP BBR/lowat/fq etc.
    3. Capture FCS and updated ntp.conf.
    4. Capture FCS and updated fstab.

Application Inventory

For significant applications added to the RPi 3, eg with apt-get or npm, a note will be made of what and when and why here.

1. YYYY-MM-DD, apt-get/npm command, motivation and comments.
2. 2018-10-20, apt-get install tcsh, because I like tcsh!
3. 2018-10-20, apt-get install ntp, for good timekeeping.
4. (2018-10-22, apt-get install htop, to help sysadmin; already installed!)
5. 2018-10-22, apt-get install cpufrequtils, for better power management.
6. 2018-10-22, apt-get install apache2, for Web serving.
7. 2018-10-23, apt-get install npm, for EOU and other purposes; followed by sudo npm install npm -g (twice) to self-upgrade.
8. 2018-10-23, apt-get install zopfli, for EOU and other purposes.
9. 2018-10-23, apt-get install optipng, for EOU.
10. 2018-10-23, apt-get install imagemagick, for EOU.
11. 2018-10-24, apt-get install procmail, for lockfile for EOU and others, 29MB of diskspace for this one utility executable!
12. 2018-10-24, apt-get install subversion, for local (EOU/ExNet/etc) repos.
13. 2018-10-24, npm install html-minifier@3.5.19 -g, for EOU mobile and other pages. (Latest @3.5.20 does not work when fed from stdin; seems to be fixed in 3.5.21...)
14. 2018-10-24, npm install uncss -g, for EOU mobile and desktop pages.
15. 2018-10-24, npm install purify-css -g, for EOU mobile and desktop pages.
16. 2018-10-24, apt-get install libjpeg-progs, for jpegtran for EOU.
17. 2018-10-24, apt-get install libfile-slurp-perl, to support jpegrescan.
18. 2018-10-24, npm install jpegrescan -g --unsafe-perm=true, for EOU.
19. 2018-10-24, installed https://github.com/MegaByte/jpegultrascan/blob/master/jpegultrascan.pl as /usr/local/bin/jpegultrascan.pl, for EOU.
20. 2018-10-24, apt-get install clean-css-cli, not cleancss!, for EOU.
21. 2018-10-24, npm install reado-cli -g, for EOU.
22. 2018-10-24, apt-get install oracle-java8-jdk, for EOU and other uses such as the Gallery.
23. 2018-10-25, apt-get install gnuplot, for EOU and other uses. Uses ~315MB of file space!
24. 2018-10-30, apt-get install brotli, for future EOU. (A matching brew upgrade brotli (0.6.0 to 1.0.7) on the Mac changes the executable name from bro to brotli to match the RPi.
25. 2018-11-17, npm install amphtml-validator -g, for future EOU possible AMP support. (Doesn't appear to have installed correctly...)
26. 2018-12-31 PENDING, npm i -g cssgip, for future EOU image support.
27. 2019-04-25: apt-get install dnsutils, for dig.
28. 2019-04-25: apt-get install tcpdump.
29. 2019-07-14: apt-get install mediainfo, 0.7.91, for EOU.
30. 2019-07-14: apt-get install libav-tools, (ffmpeg 3.2.14), for avconv for EOU.
31. 2019-07-14: apt-get install ethtool, for checking link speed.

TODO: sendmail (and remove/disable exim4?)...

While on my RPi2 and Mac npm utilities have been installed in /usr/local/bin, for the RPi3 they've ended up in /usr/bin which mixes them up with system binaries and breaks a lot of my scripts relying on them being in the former.

Pi Day

2019-03-08: now waiting to see what happens on Pi Day...

Pi Day came and went with no RPi 4 announced, bah!

In early April 2019 the router went largely deaf, so that only devices very close to it could talk to the outside world, and started dropping external connectivity at random and for extended periods.

The ISP ignored the most of the problem hard enough to let a couple of weekends slip by, including shipping a "drop-in replacement" router which was essentially unconfigured and not capable of replacing the existing device at all.

Though its MD helpful was as ever, and offered to replace equipment at cost, since I have the RPi3, rather than spend over £200 buying a new router and spending time configuring it, I might as well strip down the RPi3 and try to configure it to do the job. If that doesn't work, I can splash the cash.

So the RPi3 is going to be diverted to be the router and AP. Moving functionality from the RPi2 can happen by degrees if that all works.

So another apt-get update etc, then turn off any unwanted services for the router. I'm not sure if I should leave an Apache running (supporting something static such as the exnet.com site).

Some steps to get there, including information gathering (not all in right order, and largely following Raspberry Pi 3 as a Simple WiFi Access Point):

1. (Done) apt-get update and apt-get dist-upgrade RPi3 for security patches.
2. (Done) Capture current line configuration including username and password.
3. *** Capture current IP use on RPi2, eg for interfaces and Web sites and mail, NTP, etc.
4. (Done) Strip down RPi3 to minimum services.
   1. (Done) sudo update-rc.d apache2 disable
   2. (Done) Harden sshd, and reduce CPU impact (etc) of typical attacks.
5. (Done) Load enabling packages:
   1. (Done - backed out 2019-06-24) sudo apt-get install dnsmasq hostapd
   2. (Done) sudo apt-get install iptables-persistent
   3. (Done) sudo apt-get install pppoeconf
6. (Done) Plug in RPi3 eth0 port to spare current TG router port.
7. (Done) Set temporary static IP address (in NATted range) for eth0 so that most of config including fowarding and NATting can be completed behind current router.
8. (Done) Check log-in currently possible to RPi3 via WiFi and wired connections.
9. (Done - backed out 2019-06-24) Disable DHCP client on RPi3: sudo systemctl disable dhcpcd.service.
10. (Done - backed out 2019-06-24) Prevent DHCP client on RPi3 from attempting to assign an address to wlan0 by adding denyinterfaces wlan0 in /etc/dhcpcd.conf (above any other interfaces added there).
11. (Done - backed out 2019-07-02) Configure IP forwarding: add net.ipv4.ip_forward=1 to /etc/sysctl.conf.
12. (Done - backed out 2019-06-24) Provide static IP configuration for wlan0 in /etc/network/interfaces.d/wlan0 with temporary range 192.168.220.X, and reload interface with sudo ifdown wlan0; sudo ifup wlan0.
13. (Done) Configure /etc/hostapd/hostapd.conf for wlan0 with temporary SSID and password and update /etc/default/hostapd to use it.
14. Unmask and tentatively bring up the WiFi AP: sudo systemctl unmask hostapd; sudo systemctl enable hostapd; sudo systemctl start hostapd; the AP should be visible to WiFi users but not yet functional.
15. (Done) Drop in IP filter to minimise attack surface and help with NAT/bridge based on WiFi Access Point rule set with some config from current RPi2 folded in.
16. Edit /etc/resolvconf.conf?
17. (Done) Edit /etc/dnsmasq.conf naming local servers and (say) Google, on the lines of:

    interface=wlan0       # Use interface wlan0
    listen-address=192.168.220.1   # Specify the address to listen on
    bind-interfaces      # Bind to the interface
    server=8.8.8.8       # Use Google DNS
    domain-needed        # Don't forward short names
    bogus-priv           # Drop the non-routed address spaces.
    dhcp-range=192.168.220.75,192.168.220.150,12h # IP range and lease time
    

18. (Done) sudo service dnsmasq start.
19. (Done) Reboot - at this point AP is up and running behind existing router.
20. (Done) Plug VTOP USB-to-dual-Ethernet adaptor into RPi3 ready to accept Loop monitoring device network gateway, and one other as needed: appears as eth1 and eth2.
21. (Done) Widen existing config to extend DHCP to eth1 and eth2:
    1. (Done) In /etc/dhcpcd.conf extend denyinterfaces wlan0 eth1 eth2.
    2. (Done) In /etc/iptables/rules.v4 allow forwarding between eth0 and eth1/eth2 internally (copy wlan0 rules for eth1 and eth2).
    3. (Done) Provide static IP configuration for eth1 and eth2 in /etc/network/interfaces.d/eth12 with range 192.168.221.X and 192.168.222.X, and reload interface with sudo ifdown ethX; sudo ifup ethX.
    4. (Done) In /etc/dnsmasq.conf add extra interface=s, listen-address= and dhcp-range=s for eth1 and eth2
22. (Done) Move Loop device to eth1 and check able to phone home.
23. Prepare set up bridging between eth1 and ppp0 so that eth2 can see the whole public address space and effectively become the DMZ, and the RPi2 can be plugged into it:
    1. (Nominally ppp0 and eth0 should be added to /etc/dhcpcd.conf' denyinterfaces, but keeping the DHCP client off entirely is what is really wanted...)
    2. (Done) Take all mention of eth1 and the 192.168.221.X range out of /etc/dnsmasq.conf.
    3. Replace static config for eth1 in /etc/network/interfaces.d/eth12 to match publich address range with:

       allow-hotplug eth1
       iface eth1 inet static
           address 79.135.97.94
           netmask 255.255.255.224
           network 79.135.97.64
           broadcast 79.135.97.95
       

    4. Modify the rules.v4 to only NAT traffic coming from the wlan0 and eth2 interfaces (192.168.X.X):

       # Allow Access Point NAT only from wlan0 and eth2 (192.168.X.X).
       -A POSTROUTING -s 192.168.0.0/16 -o ppp0 -j SNAT --to-source 79.135.97.65
       

24. (Done) Have eth0 marked as 'manual' in interfaces/core.
25. (Done) Change all eth0 in iptables to ppp0.
26. Work out how to route static addresses though to (say) eth2.
27. Configure PPPoE connection.
28. Remove old router.
29. Tackle any remaining points above not done!

So, annoyingly, I got a long way but not far enough.

I can bring up my RPi3 behind the existing (fading) Technicolor and the RPi3 provides what seems to be a fairly decent WiFi access point and outward connectivity to the Internet. My existing RPi2 server sits in the Technicolor DMZ and presents services on its ~6 static addresses to the outside world.

I can bring up my RPi3 directly on the PPPoE connection via the ECI VDSL2 modem you originally supplied with the FTTC service. It can continue to provide WiFi AP. I cannot however get it to route between the ppp0 (on eth0) connection and the eth1 connection that the RPi2 is on, primarily because they both end up with 79.135.97.65 in their range, it seems (and ppp0/eth0 cannot be bridged with eth1):

ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1492
        inet 79.135.97.65  netmask 255.255.255.255  destination 195.26.38.252
        ppp  txqueuelen 3  (Point-to-Point Protocol)
        RX packets 42937  bytes 5363610 (5.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34314  bytes 4267734 (4.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 79.135.97.65  netmask 255.255.255.224  broadcast 79.135.97.95
        inet6 fe80::20f:c9ff:fe10:f6d9  prefixlen 64  scopeid 0x20<link>
        ether 00:0f:c9:10:f6:d9  txqueuelen 1000  (Ethernet)
        RX packets 12377  bytes 1693664 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22276  bytes 2357947 (2.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions

Very frustrating and if I could sort that then I’d have the whole thing done, I think. I’m a bit out of practice on knotty routing issues these days.

There just don’t seem to be any good example of static public IP address blocks being routed to in this way with a recent Linux.

I really don’t need fancy DMZ or NATting, and at a pinch the top half of the static address range may just have enough space for short-lease DHCP for our laptops, phones etc.

Since the Vigor is seemingly doing a good job, and has good WiFi support, I no longer aim to get the RPi3 to be a WiFi AP (nor Bluetooth).

I will need to unpick some of the PPPoE and similar features, and plonk the RPi3 back with a static address on the public range. I'll have some other updating to do to get back on course to replace the RPi2.

(My alternate off-grid power-supply for the RPi3 has not turned up from RS, and turns out still to be on back order, and with no good alternative in sight. Ho hum...

Reset

The first thing to do is rip out the WiFi, dnsmasq, and PPPoE, and select and set a unique static IP address, while sat in front of the TV while the kids are at school. Then I can plug it into the router (and be mains-powered ... hush) while I tinker. Plenty of spare solar PV on-grid to paper over that sin temporarily.

I've created a DNS entry for it too, which should help.

• (Done) sudo apt-get remove dnsmasq
• (Done) sudo apt-get remove hostapd
• (Done) Simpify networking and set static IP.
• (Done) Re-enable DHCP client on RPi3: sudo systemctl enable dhcpcd.service, temporarily.
• (Done) Stop IP forwarding: remove net.ipv4.ip_forward=1 to /etc/sysctl.conf.
• (Done) Trimmed PPP-driven /etc/iprules/rules.v4 to rebuild.
• (Done) Unconfigure IP forwarding: remove net.ipv4.ip_forward=1 from /etc/sysctl.conf.
• (Done) Fix IP filter for RPi3 use as leaf (copied config from green).
• (TODO) Re-enable WiFi as client (with DCHP for it), temporarily.
• (TODO) Power-down WiFi, at least temporarily.

I managed to bring together the RPi3B+, a wired connecton, and the TV, by virtue of moving everything into the kitchen while the family was out!

Connectivity to the outside world immediately worked, and so an immediate sudo apt-get update && sudo apt-get dist-upgrade was done for security.

(I installed mediainfo and avconv (libavi-tools) as I'll want them soon.)

Upon reboot (to make dbus happy) I saw that networking didn't restart happily, and was told to run systemctl status networking.service for more information. It looks like /sbin/ifup -a --read-environment is unhappy, possibly with my mauling of /etc/network/interfaces. And attempting to ssh in from my Mac is rejected, by sshd.

Commenting out anything to do with wlan0 from /etc/network/interfaces made networking happy on another reboot.

Boot is still showing an error: Failed to start dhcpcd on all interfaces, and I am invited to run systemctl status dhcpcd.service for details. As it happens, the 'details' are not helpful.

After some messing around, and discovering a key file that I had mis-named, I was able to set-up key-only ssh access again. So the kitchen setup can be dismantled, and I can work remotely (until I muck up networking again!).

I have applied a copy of the iptables config used on green, which should make things a little safer to leave connected.

HTTPS Warmup

Ready to support https on Apache with LetsEncrypt, I'm installing certbot starting with sudo apt-get install certbot python-certbot-apache.

I enable and start Apache to get its bland "It works!" page.

Running the suggested sudo certbot --apache mainly seems to generate errors, and does not seem to make meaningful updates to the Apache config files.

(I have had to remember to allow port 443 in the iptables config too!)

The 'manual' method certbot run -a webroot -i apache -w /var/www/html -d example.com seemed however to do the trick, and I have an https server running! It is accessible locally, and remotely to (at least) Google.

In future, to handle multiple sub-domains, the certbot --expand ... and certbot certonly --cert-name example.com ... commands look useful.