libera/#devuan-arm/ Monday, 2022-01-24

VallHowdy everyone. Yesterday my RPi4_8GB running Devuan Chimaera experienced its USB ports stop working: when plugging a smartphone, the USB indicators (both the charging and the tethering available indicators) blinked on twice, then showed as disconnected. Syslog showed the following: https://transfer.sh/ap7Pup/t.txt14:21
VallThe same happened in all 4 USB ports (both USB2 and USB3). It was not the USB cable (same cable worked perfectly in my notebook, and other cables showed the same issue on the RPi).14:22
VallTried to rmmod/insmod the USB drivers, then learned they are compiled-in (config.txt: "=y") and not as modules, so that was not possible14:24
VallA powercycle (`sudo halt`, remove power, count to 10, reconnect power) was enough to cure the issue: it came back from reboot with all USB ports perfectly functional.14:25
Vall`uptime` was over 43 days, so not bad, but this is a headless machine so a major hassle to reboot everytime something goes bad -- finding and fixing the root cause would be much preferred.14:26
Vallkernel was 5.10.78-v8 from Sun Nov 14 11:49:45 EST 2021, so kinda old, will try to update and see if I can spend the next 2 months or more without issues.14:27
VallGiven all that, my question is: has anyone else experienced this? Is it a known issue?14:28
c0rneliusCan't say that I have. Did you have a power surge or something?15:05
Vallc0rnelius: pretty sure no power events: I'm using the official RPi4 power adapter from the Raspberry Foundation, and I'm totally offgrid: the RPi4 power adapter is fed with 220VAC power provided by a top-of-the-line DC->AC power inverter, which is itself fed from a bank of LiFePO4 batteries, which are kept charged by solar panels, all top-of-the-line. And I keep a very close eye on the state of the batteries, minimum level in these 43 days was 69% ...15:16
Vall... charged15:16
VallThe inverter (a Victron Multiplus) has its own MCU and keeps its own log of power events, I will connect to it (needs a special RS485 comms cable and special software, which I don't keep running all the time) and see if it recorded any power events15:18
Vallc0rnelius: another question re: `fetch`: I'm taking advantage of the reboot to upgrade everything, including Devuan stuff (`apt-get update && apt-get upgrade`), and `fetch`  stuff (`fetch -u && fetch -1 && fetch -b && fetch -f && fetch -U`).15:23
VallAll went well except for `fetch -U` which fais with a "Something went wrong?"  message in bold.15:24
c0rneliusLet me look.15:24
VallRunning `strace -f fetch -U` shows the following as a possible point of failure:15:24
Vallnewfstatat(AT_FDCWD, "/usr/src/linux-headers-5.10.78-v8/.config", 0x7ffa704bb8, 0) = -1 ENOENT (No such file or directory)15:24
c0rneliusoh. thats because its looking for the old kernel15:24
VallAnd it in fact does not exist: ls -l /usr/src/linux-headers-5.10.78-v8/.config15:24
Vallls: cannot access '/usr/src/linux-headers-5.10.78-v8/.config': No such file or directory15:25
c0rneliusrun fetch -u15:25
VallAh OK15:25
c0rneliusyou should have 5.15.y available to you15:25
VallI did not reboot after `fetch -1`. Should I?15:25
c0rneliusfor updating the userland, yes. It checks the current kernel running and build depending on if ur running arm or arm64.15:26
VallYes, the `fetch -1` I ran as part of the sequence above brought 5.15.15-v8 in15:26
VallOK, will reboot and run `fetch -U` again15:26
Vallback in a minute15:26
Vallc0rnelius: after the reboot, `fetch -U` seems to runL finishes with "Done.", but reporting a fuck-ton of warnings and other seemingly not-so-OK messages: https://0x0.st/ooY9.txt15:40
Vallc0rnelius: can you please have a look and tell me whether I hosed my userland?15:40
c0rneliusThats normal on aarch6415:41
c0rneliusit only builds like a 1/4 of the userland on aarch64 so the warnings are just the build passing it over.15:42
VallThanks for the confirmation c0rnelius15:43
VallI do not remember running `fetch -U` before, been a few months since I installed my RPi415:44
Vallchecking the inverter log now, brb15:44
c0rneliusits not that big of a deal on aarch64. the userland is borderline pointless. you can test by running - `vcgencmd measure_temp`15:45
VallJust checked the inverter log, no power events were logged, so I think I can say there's about 99.999% chance of no power events happened15:47
c0rneliuswhens the last time you updated ur eeprom?15:47
Vallc0rnelius: that would have been when I installed the box, lemme check my notes15:47
c0rnelius`deb-eeprom -u`15:48
c0rneliusshould list whats installed and whats available.15:48
c0rnelius13.4 should be available15:49
Vallhummmmrmrmr15:50
VallI have no record on my notes of ever running deb-eeprom15:50
VallSo I probably effed-up on this15:50
Valllemme check the `deb-eeprom -u`15:51
c0rneliusYou running off a ssd?15:51
Vallc0rnelius: nope, out of a sooper-dooper Sandisk Max Endurance SD card15:51
c0rneliusah. yeah I would update the eeprom. I know early models had usb issues that were corrected in the eeprom.15:52
Valloh heck, I think you just hit the nail in the head: https://0x0.st/ooYJ.txt15:54
VallSeems I'm running 12.14-1 and 13.4 is available, so yeah my eeprom would be pretty stale15:55
Vallupdating it now15:55
c0rnelius12.4 isn't to bad. I'm currently on 13.4 running stable: Thu 02 Dec 2021 11:08:03 AM UTC (1638443283)15:55
Vallc0rnelius: surely you mean 12.14 and not 12.4 :-)15:56
c0rneliussorry its early here :)15:57
c0rneliusmy mind is still on coffee15:57
Vallno prob ;-)15:57
VallAnyway, besides `apt-get update && apt-get upgrade`, the `fetch ...` command string I described above, and `deb-eeprom -U`, is there anything else OS-side I should upgrade?15:58
c0rneliusnah thats it.16:01
VallKK16:01
Vallmonitoring `deb-eeprom -U` output, it looks like it's quite involved16:02
c0rneliusit checks the github for the version and then finds the tarballs. patches, builds and installs.16:04
c0rneliusLife would be easier if we had hosting.16:05
Vall"Life would be easier if we had hosting." -> What do you need? An HTTP/HTTPS server with SFTP access to upload/update files would be enough? Or a full-blown VM/VPS, or perhaps a dedicated machine?16:07
c0rneliusneed our own repo for apt. mostly just for hosting kernels and eeprom.16:09
c0rneliusi guess pi-bluetooth and sys-mods could be in there too, but honestly they don't matter to much.16:09
VallSo, at least a VM/VPS, right?16:10
c0rneliusI believe so. I honestly don't know much about setting all that up.16:11
c0rneliusnot really my forte16:11
VallMeanwhile, `deb-eeprom -U` just ran its course: https://0x0.st/ooYY.txt16:12
VallI see no hitches, so now I'm supposed to run `rpi-eeprom-update && rpi-eeprom-update -a`, right?16:13
c0rneliusyeah see if it updates: `rpi-eeprom-update -a`16:13
c0rneliusah you have dkms installed. thats why it so involved.16:13
c0rneliusif it doesn't you can do it by hand.16:14
c0rneliusactually: `sudo rpi-eeprom-update -a`16:14
VallOK16:15
Valljust ran it, looks good: https://0x0.st/ooYE.txt16:15
VallJust to be sure, will now do a `poweroff` and then a powercycle16:16
c0rneliuslooks good16:16
Vallyeah, machine came back from the powercycle looking peachy, kernel "5.15.15-v8 #1 SMP PREEMPT Mon Jan 17 20:43:53 EST 2022" loaded and running, all networking interfaces up and running, ZFS modules ditto, `vcgencmd measure_temp` ditto, etc16:23
Vallso I will just keep an eye on it and report back if USB stops working again, or some other issue comes up16:24
c0rneliusalright16:24
TenkawaVall: good luck.. anything past 5.15.13 has been acting a bit iffy16:24
VallTenkawa: better to downgrade to 5.15.13 (or 5.15.12?) if I need stability?16:25
Tenkawa5.16 is a complete mess16:25
TenkawaI did,16:25
c0rneliusI'm running 5.15.15.  its fine for me.16:25
c0rneliusits 5.15.16 thats borked in my experience.16:26
Tenkawa14 is too16:26
VallOK, I will try 5.15.15, if any issues I will come back and report, and keep in mind Tenkawa's tip about downgrading to 5.15.1316:26
Tenkawa13 has been solid since I went back to it16:27
VallFor now the machine is working pretty much as a central router (two Wifi interfaces -- the embedded one running HostAP at 5Ghz and a USB2 wifi adapter running also HostAP at 2.4Ghz, ditto two Ethernet interfaces, those bounded with `brctl` and operating as a 2-port switch for my wired machines).16:29
c0rneliusHow hot does it get?16:29
c0rneliusIf the board over heats it can temporarily kill the usb ports.16:31
VallEvery now and then I use it to overnight download a few GBs to tens of GBs from the internet using rclone over SSH, and then transfer it to my notebook when I wake up (notebooks and other power-heavy equipment is kept off/suspended when not in use)16:31
Vallc0rnelius> How hot does it get? -> I have a fan turned on permanently, plus a thermal sink on top of the BCM SoC16:32
c0rneliusgotcha16:32
VallI do not have permanent monitoring right now (installing Prometheus/Graphana is on my to-do list), but I tested this setup thermally pretty intensivelly when I assembled the RPi4, and it was not overheating even under stress (all cores at 100%, etc)16:34
c0rneliusWell something to check if it happens again. I've had it happen to me on other boards.16:35
TenkawaVall: just use vcgencmd measure_temp16:35
Tenkawa /opt/vc/bin/vcgencmd measure_temp16:35
Tenkawatemp=39.9'C16:36
VallThere's a `vcgencmd` command to show if any thermal throttling came up, right?16:36
Tenkawayes16:36
Tenkawajust a sec16:36
Valllemme check16:36
Tenkawa /opt/vc/bin/vcgencmd get_throttled16:37
Tenkawathrottled=0x016:37
Tenkawathere would be a diff hex  code if it had been16:38
Vallvcgencmd  measure_temp16:38
Valltemp=45.7'C16:38
Vallvcgencmd  get_throttled16:38
Vallthrottled=0x016:38
VallSo I guess I'm good16:38
Tenkawayep16:38
VallWould anything get logged to dmesg or syslog if something thermal happened?16:39
Tenkawaonly if you setup lm-sensors and have it log16:39
VallBecause I have rebooted the machine already to bring USB back, so I guess get_throttled got zeroed out16:39
Tenkawayeah I'm not sure when throttled gets reset16:40
Tenkawaif you have lm-sensors you can install monitoring tools that can use things like these16:41
Tenkawa sensors16:41
Tenkawacpu_thermal-virtual-016:41
TenkawaAdapter: Virtual device16:41
Tenkawatemp1:        +38.9°C16:41
VallJust checked, I have libsensors installed but not lm-sensors :-/16:41
Tenkawaand there are apps/etc that can use that framework16:42
VallAnother thing I could do is to run `/opt/vc/bin/vcgencmd get_throttled | logger` in cron, no?16:42
Tenkawayep16:42
Vallyeah doing it now (I usually prefer the minimalistic approach)16:42
Tenkawathere's a lot of diff ways.. I have several  fans that use overlays that can even turn on/off at the right temp16:43
TenkawaI just put them in /boot/config.txt and reboot16:43
Tenkawa(mostly my argon40's)16:44
Vall"I have several  fans that use overlay" not sure what an overlay is in this context, but my plan is to hook up a TP222 power transistor controlling the fan to a GPIO port and have a script turn the GPIO (and therefore the fan via the power transistor) on and off depending on the temperature16:45
Vallfor now I just plugged the fan straight into the +5VDC and ground RPi4 pins and so it's running all the time16:45
Tenkawaoh.. device tree overlay.. its a piece of software that adds code on to the pi for adding logic to a addon piece of hardware16:46
Tenkawathis way its not running the fans 100% rpm 100% all the time16:46
Tenkawait adjusts as needed16:46
Tenkawaquite nice.. (especially if you want it to stay quiet lol)16:47
VallArgon40 is the sooper-dooper metal RPi case, right? that's my consumer's dream :-) not easy nor cheap to find around here16:47
TenkawaI have 2 of their m.2 ones16:47
VallWhich one do you use, the ONE v2?16:48
TenkawaI really like them16:48
VallAh, the M.2! Pretty nice, congrats!16:48
VallHere they would cost more than the RPi4_8GB ;-)16:48
Tenkawathis guy: https://www.argon40.com/argon-one-m-2-case-for-raspberry-pi-4.html16:49
Tenkawa I have 2 8gb pi 4's in them16:49
VallAgain, congrats!16:49
Tenkawathanks16:49
Vall:-)16:49
VallPerhaps I will get one or similar when I start putting my RPi4 through its paces16:50
Tenkawalook whats on the primary:16:50
Tenkawa grep PR /etc/os-release16:50
TenkawaPRETTY_NAME="Devuan GNU/Linux 4 (chimaera)"16:50
VallI mean, *really* putting it through its paces16:50
Tenkawaheehee16:51
VallI plan on running KVM/QEMU and at least 2 VMs, one running this stupid closed-source stuff: https://github.com/victronenergy/venus/wiki16:51
VallAnd the other for prometheus, graphana, etc16:51
Vall(I need the Venus stuff here to better monitor/record my offgrid setup)16:52
TenkawaI just transitioned everything I had to be able to run on my M1 mac so I'm 100% arm now :)16:52
Tenkawabuilding linux vm's in a parallels linux vm on os x is easy now lol16:53
VallHeh ;-) an M1 Mac is pretty nice, great CPU. I do not plan on moving on to it, tho -- really *really* expensive, and I'm kinda 'allergic' to Apple stuff16:54
Vall(I administer a couple of Macs -- old ones, not M1s -- on a remote site, for a friend).16:55
TenkawaIts actually what I've preferred most of my life… just not been practical for  my uses until recently16:55
Tenkawanow on arm its exactly what I need16:56
VallYeah, OS/X works fine specially as a desktop.16:56
VallNot so good as a server IMHO, but good enough.16:56
VallAgain congrats on your setup, front-end on Mac M1 and backend on RPi4s with Argon40s is the gold standard ;-)16:57
TenkawaVall: yeah I use it for desktop and devel16:58
VallMe, backend is on this one RPi4 with crappy plastic case and frontend is a pair of notebooks (an old Thinkpad for the wife and a HP ZBook 17G4 with a Xeon and -- get that -- 64GB RAM with *ECC* -- for me)16:58
VallI quite like the ZBook, apart from having had to replace its keyboard after only 3 years, and will have to replace its CPU fan as it's starting to whine16:59
Tenkawathis is my first 32gb ram desktop/laptop17:00
Tenkawahad plenty of servers with more17:00
TenkawaI like having 32.. been very nice17:00
VallNot great reliability, the Thinkpad, having been bought in 2012 (will be 10 years old in a few months) never had to have anything replaced17:00
VallYeah RAM is like, the more the better17:00
Tenkawaindeed17:01
VallWith ZFS ARC it never goes to waste17:01
Vall(or the standard OS dynamic buffer-cache, as in Linux and I believe also in OS/X)17:02
VallWife calling, which means chores a-waiting17:03
VallNice talking to you, Tenkawa -- and c0rnelius, many thanks for the great assistance as usual17:03
c0rneliusVall: ur welcome17:04
Vallc0rnelius: I will see if I can arrange a VM or something and contribute it to devuan-arm/arm-img-builder hosting, no promises but if I manage it I will let you know17:05
* Vall waves17:05
Tenkawayou too..17:06
Tenkawaand np and good luck with everything17:06

Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!