Vall | Howdy everyone. Yesterday my RPi4_8GB running Devuan Chimaera experienced its USB ports stop working: when plugging a smartphone, the USB indicators (both the charging and the tethering available indicators) blinked on twice, then showed as disconnected. Syslog showed the following: https://transfer.sh/ap7Pup/t.txt | 14:21 |
---|---|---|
Vall | The same happened in all 4 USB ports (both USB2 and USB3). It was not the USB cable (same cable worked perfectly in my notebook, and other cables showed the same issue on the RPi). | 14:22 |
Vall | Tried to rmmod/insmod the USB drivers, then learned they are compiled-in (config.txt: "=y") and not as modules, so that was not possible | 14:24 |
Vall | A powercycle (`sudo halt`, remove power, count to 10, reconnect power) was enough to cure the issue: it came back from reboot with all USB ports perfectly functional. | 14:25 |
Vall | `uptime` was over 43 days, so not bad, but this is a headless machine so a major hassle to reboot everytime something goes bad -- finding and fixing the root cause would be much preferred. | 14:26 |
Vall | kernel was 5.10.78-v8 from Sun Nov 14 11:49:45 EST 2021, so kinda old, will try to update and see if I can spend the next 2 months or more without issues. | 14:27 |
Vall | Given all that, my question is: has anyone else experienced this? Is it a known issue? | 14:28 |
c0rnelius | Can't say that I have. Did you have a power surge or something? | 15:05 |
Vall | c0rnelius: pretty sure no power events: I'm using the official RPi4 power adapter from the Raspberry Foundation, and I'm totally offgrid: the RPi4 power adapter is fed with 220VAC power provided by a top-of-the-line DC->AC power inverter, which is itself fed from a bank of LiFePO4 batteries, which are kept charged by solar panels, all top-of-the-line. And I keep a very close eye on the state of the batteries, minimum level in these 43 days was 69% ... | 15:16 |
Vall | ... charged | 15:16 |
Vall | The inverter (a Victron Multiplus) has its own MCU and keeps its own log of power events, I will connect to it (needs a special RS485 comms cable and special software, which I don't keep running all the time) and see if it recorded any power events | 15:18 |
Vall | c0rnelius: another question re: `fetch`: I'm taking advantage of the reboot to upgrade everything, including Devuan stuff (`apt-get update && apt-get upgrade`), and `fetch` stuff (`fetch -u && fetch -1 && fetch -b && fetch -f && fetch -U`). | 15:23 |
Vall | All went well except for `fetch -U` which fais with a "Something went wrong?" message in bold. | 15:24 |
c0rnelius | Let me look. | 15:24 |
Vall | Running `strace -f fetch -U` shows the following as a possible point of failure: | 15:24 |
Vall | newfstatat(AT_FDCWD, "/usr/src/linux-headers-5.10.78-v8/.config", 0x7ffa704bb8, 0) = -1 ENOENT (No such file or directory) | 15:24 |
c0rnelius | oh. thats because its looking for the old kernel | 15:24 |
Vall | And it in fact does not exist: ls -l /usr/src/linux-headers-5.10.78-v8/.config | 15:24 |
Vall | ls: cannot access '/usr/src/linux-headers-5.10.78-v8/.config': No such file or directory | 15:25 |
c0rnelius | run fetch -u | 15:25 |
Vall | Ah OK | 15:25 |
c0rnelius | you should have 5.15.y available to you | 15:25 |
Vall | I did not reboot after `fetch -1`. Should I? | 15:25 |
c0rnelius | for updating the userland, yes. It checks the current kernel running and build depending on if ur running arm or arm64. | 15:26 |
Vall | Yes, the `fetch -1` I ran as part of the sequence above brought 5.15.15-v8 in | 15:26 |
Vall | OK, will reboot and run `fetch -U` again | 15:26 |
Vall | back in a minute | 15:26 |
Vall | c0rnelius: after the reboot, `fetch -U` seems to runL finishes with "Done.", but reporting a fuck-ton of warnings and other seemingly not-so-OK messages: https://0x0.st/ooY9.txt | 15:40 |
Vall | c0rnelius: can you please have a look and tell me whether I hosed my userland? | 15:40 |
c0rnelius | Thats normal on aarch64 | 15:41 |
c0rnelius | it only builds like a 1/4 of the userland on aarch64 so the warnings are just the build passing it over. | 15:42 |
Vall | Thanks for the confirmation c0rnelius | 15:43 |
Vall | I do not remember running `fetch -U` before, been a few months since I installed my RPi4 | 15:44 |
Vall | checking the inverter log now, brb | 15:44 |
c0rnelius | its not that big of a deal on aarch64. the userland is borderline pointless. you can test by running - `vcgencmd measure_temp` | 15:45 |
Vall | Just checked the inverter log, no power events were logged, so I think I can say there's about 99.999% chance of no power events happened | 15:47 |
c0rnelius | whens the last time you updated ur eeprom? | 15:47 |
Vall | c0rnelius: that would have been when I installed the box, lemme check my notes | 15:47 |
c0rnelius | `deb-eeprom -u` | 15:48 |
c0rnelius | should list whats installed and whats available. | 15:48 |
c0rnelius | 13.4 should be available | 15:49 |
Vall | hummmmrmrmr | 15:50 |
Vall | I have no record on my notes of ever running deb-eeprom | 15:50 |
Vall | So I probably effed-up on this | 15:50 |
Vall | lemme check the `deb-eeprom -u` | 15:51 |
c0rnelius | You running off a ssd? | 15:51 |
Vall | c0rnelius: nope, out of a sooper-dooper Sandisk Max Endurance SD card | 15:51 |
c0rnelius | ah. yeah I would update the eeprom. I know early models had usb issues that were corrected in the eeprom. | 15:52 |
Vall | oh heck, I think you just hit the nail in the head: https://0x0.st/ooYJ.txt | 15:54 |
Vall | Seems I'm running 12.14-1 and 13.4 is available, so yeah my eeprom would be pretty stale | 15:55 |
Vall | updating it now | 15:55 |
c0rnelius | 12.4 isn't to bad. I'm currently on 13.4 running stable: Thu 02 Dec 2021 11:08:03 AM UTC (1638443283) | 15:55 |
Vall | c0rnelius: surely you mean 12.14 and not 12.4 :-) | 15:56 |
c0rnelius | sorry its early here :) | 15:57 |
c0rnelius | my mind is still on coffee | 15:57 |
Vall | no prob ;-) | 15:57 |
Vall | Anyway, besides `apt-get update && apt-get upgrade`, the `fetch ...` command string I described above, and `deb-eeprom -U`, is there anything else OS-side I should upgrade? | 15:58 |
c0rnelius | nah thats it. | 16:01 |
Vall | KK | 16:01 |
Vall | monitoring `deb-eeprom -U` output, it looks like it's quite involved | 16:02 |
c0rnelius | it checks the github for the version and then finds the tarballs. patches, builds and installs. | 16:04 |
c0rnelius | Life would be easier if we had hosting. | 16:05 |
Vall | "Life would be easier if we had hosting." -> What do you need? An HTTP/HTTPS server with SFTP access to upload/update files would be enough? Or a full-blown VM/VPS, or perhaps a dedicated machine? | 16:07 |
c0rnelius | need our own repo for apt. mostly just for hosting kernels and eeprom. | 16:09 |
c0rnelius | i guess pi-bluetooth and sys-mods could be in there too, but honestly they don't matter to much. | 16:09 |
Vall | So, at least a VM/VPS, right? | 16:10 |
c0rnelius | I believe so. I honestly don't know much about setting all that up. | 16:11 |
c0rnelius | not really my forte | 16:11 |
Vall | Meanwhile, `deb-eeprom -U` just ran its course: https://0x0.st/ooYY.txt | 16:12 |
Vall | I see no hitches, so now I'm supposed to run `rpi-eeprom-update && rpi-eeprom-update -a`, right? | 16:13 |
c0rnelius | yeah see if it updates: `rpi-eeprom-update -a` | 16:13 |
c0rnelius | ah you have dkms installed. thats why it so involved. | 16:13 |
c0rnelius | if it doesn't you can do it by hand. | 16:14 |
c0rnelius | actually: `sudo rpi-eeprom-update -a` | 16:14 |
Vall | OK | 16:15 |
Vall | just ran it, looks good: https://0x0.st/ooYE.txt | 16:15 |
Vall | Just to be sure, will now do a `poweroff` and then a powercycle | 16:16 |
c0rnelius | looks good | 16:16 |
Vall | yeah, machine came back from the powercycle looking peachy, kernel "5.15.15-v8 #1 SMP PREEMPT Mon Jan 17 20:43:53 EST 2022" loaded and running, all networking interfaces up and running, ZFS modules ditto, `vcgencmd measure_temp` ditto, etc | 16:23 |
Vall | so I will just keep an eye on it and report back if USB stops working again, or some other issue comes up | 16:24 |
c0rnelius | alright | 16:24 |
Tenkawa | Vall: good luck.. anything past 5.15.13 has been acting a bit iffy | 16:24 |
Vall | Tenkawa: better to downgrade to 5.15.13 (or 5.15.12?) if I need stability? | 16:25 |
Tenkawa | 5.16 is a complete mess | 16:25 |
Tenkawa | I did, | 16:25 |
c0rnelius | I'm running 5.15.15. its fine for me. | 16:25 |
c0rnelius | its 5.15.16 thats borked in my experience. | 16:26 |
Tenkawa | 14 is too | 16:26 |
Vall | OK, I will try 5.15.15, if any issues I will come back and report, and keep in mind Tenkawa's tip about downgrading to 5.15.13 | 16:26 |
Tenkawa | 13 has been solid since I went back to it | 16:27 |
Vall | For now the machine is working pretty much as a central router (two Wifi interfaces -- the embedded one running HostAP at 5Ghz and a USB2 wifi adapter running also HostAP at 2.4Ghz, ditto two Ethernet interfaces, those bounded with `brctl` and operating as a 2-port switch for my wired machines). | 16:29 |
c0rnelius | How hot does it get? | 16:29 |
c0rnelius | If the board over heats it can temporarily kill the usb ports. | 16:31 |
Vall | Every now and then I use it to overnight download a few GBs to tens of GBs from the internet using rclone over SSH, and then transfer it to my notebook when I wake up (notebooks and other power-heavy equipment is kept off/suspended when not in use) | 16:31 |
Vall | c0rnelius> How hot does it get? -> I have a fan turned on permanently, plus a thermal sink on top of the BCM SoC | 16:32 |
c0rnelius | gotcha | 16:32 |
Vall | I do not have permanent monitoring right now (installing Prometheus/Graphana is on my to-do list), but I tested this setup thermally pretty intensivelly when I assembled the RPi4, and it was not overheating even under stress (all cores at 100%, etc) | 16:34 |
c0rnelius | Well something to check if it happens again. I've had it happen to me on other boards. | 16:35 |
Tenkawa | Vall: just use vcgencmd measure_temp | 16:35 |
Tenkawa | /opt/vc/bin/vcgencmd measure_temp | 16:35 |
Tenkawa | temp=39.9'C | 16:36 |
Vall | There's a `vcgencmd` command to show if any thermal throttling came up, right? | 16:36 |
Tenkawa | yes | 16:36 |
Tenkawa | just a sec | 16:36 |
Vall | lemme check | 16:36 |
Tenkawa | /opt/vc/bin/vcgencmd get_throttled | 16:37 |
Tenkawa | throttled=0x0 | 16:37 |
Tenkawa | there would be a diff hex code if it had been | 16:38 |
Vall | vcgencmd measure_temp | 16:38 |
Vall | temp=45.7'C | 16:38 |
Vall | vcgencmd get_throttled | 16:38 |
Vall | throttled=0x0 | 16:38 |
Vall | So I guess I'm good | 16:38 |
Tenkawa | yep | 16:38 |
Vall | Would anything get logged to dmesg or syslog if something thermal happened? | 16:39 |
Tenkawa | only if you setup lm-sensors and have it log | 16:39 |
Vall | Because I have rebooted the machine already to bring USB back, so I guess get_throttled got zeroed out | 16:39 |
Tenkawa | yeah I'm not sure when throttled gets reset | 16:40 |
Tenkawa | if you have lm-sensors you can install monitoring tools that can use things like these | 16:41 |
Tenkawa | sensors | 16:41 |
Tenkawa | cpu_thermal-virtual-0 | 16:41 |
Tenkawa | Adapter: Virtual device | 16:41 |
Tenkawa | temp1: +38.9°C | 16:41 |
Vall | Just checked, I have libsensors installed but not lm-sensors :-/ | 16:41 |
Tenkawa | and there are apps/etc that can use that framework | 16:42 |
Vall | Another thing I could do is to run `/opt/vc/bin/vcgencmd get_throttled | logger` in cron, no? | 16:42 |
Tenkawa | yep | 16:42 |
Vall | yeah doing it now (I usually prefer the minimalistic approach) | 16:42 |
Tenkawa | there's a lot of diff ways.. I have several fans that use overlays that can even turn on/off at the right temp | 16:43 |
Tenkawa | I just put them in /boot/config.txt and reboot | 16:43 |
Tenkawa | (mostly my argon40's) | 16:44 |
Vall | "I have several fans that use overlay" not sure what an overlay is in this context, but my plan is to hook up a TP222 power transistor controlling the fan to a GPIO port and have a script turn the GPIO (and therefore the fan via the power transistor) on and off depending on the temperature | 16:45 |
Vall | for now I just plugged the fan straight into the +5VDC and ground RPi4 pins and so it's running all the time | 16:45 |
Tenkawa | oh.. device tree overlay.. its a piece of software that adds code on to the pi for adding logic to a addon piece of hardware | 16:46 |
Tenkawa | this way its not running the fans 100% rpm 100% all the time | 16:46 |
Tenkawa | it adjusts as needed | 16:46 |
Tenkawa | quite nice.. (especially if you want it to stay quiet lol) | 16:47 |
Vall | Argon40 is the sooper-dooper metal RPi case, right? that's my consumer's dream :-) not easy nor cheap to find around here | 16:47 |
Tenkawa | I have 2 of their m.2 ones | 16:47 |
Vall | Which one do you use, the ONE v2? | 16:48 |
Tenkawa | I really like them | 16:48 |
Vall | Ah, the M.2! Pretty nice, congrats! | 16:48 |
Vall | Here they would cost more than the RPi4_8GB ;-) | 16:48 |
Tenkawa | this guy: https://www.argon40.com/argon-one-m-2-case-for-raspberry-pi-4.html | 16:49 |
Tenkawa | I have 2 8gb pi 4's in them | 16:49 |
Vall | Again, congrats! | 16:49 |
Tenkawa | thanks | 16:49 |
Vall | :-) | 16:49 |
Vall | Perhaps I will get one or similar when I start putting my RPi4 through its paces | 16:50 |
Tenkawa | look whats on the primary: | 16:50 |
Tenkawa | grep PR /etc/os-release | 16:50 |
Tenkawa | PRETTY_NAME="Devuan GNU/Linux 4 (chimaera)" | 16:50 |
Vall | I mean, *really* putting it through its paces | 16:50 |
Tenkawa | heehee | 16:51 |
Vall | I plan on running KVM/QEMU and at least 2 VMs, one running this stupid closed-source stuff: https://github.com/victronenergy/venus/wiki | 16:51 |
Vall | And the other for prometheus, graphana, etc | 16:51 |
Vall | (I need the Venus stuff here to better monitor/record my offgrid setup) | 16:52 |
Tenkawa | I just transitioned everything I had to be able to run on my M1 mac so I'm 100% arm now :) | 16:52 |
Tenkawa | building linux vm's in a parallels linux vm on os x is easy now lol | 16:53 |
Vall | Heh ;-) an M1 Mac is pretty nice, great CPU. I do not plan on moving on to it, tho -- really *really* expensive, and I'm kinda 'allergic' to Apple stuff | 16:54 |
Vall | (I administer a couple of Macs -- old ones, not M1s -- on a remote site, for a friend). | 16:55 |
Tenkawa | Its actually what I've preferred most of my life… just not been practical for my uses until recently | 16:55 |
Tenkawa | now on arm its exactly what I need | 16:56 |
Vall | Yeah, OS/X works fine specially as a desktop. | 16:56 |
Vall | Not so good as a server IMHO, but good enough. | 16:56 |
Vall | Again congrats on your setup, front-end on Mac M1 and backend on RPi4s with Argon40s is the gold standard ;-) | 16:57 |
Tenkawa | Vall: yeah I use it for desktop and devel | 16:58 |
Vall | Me, backend is on this one RPi4 with crappy plastic case and frontend is a pair of notebooks (an old Thinkpad for the wife and a HP ZBook 17G4 with a Xeon and -- get that -- 64GB RAM with *ECC* -- for me) | 16:58 |
Vall | I quite like the ZBook, apart from having had to replace its keyboard after only 3 years, and will have to replace its CPU fan as it's starting to whine | 16:59 |
Tenkawa | this is my first 32gb ram desktop/laptop | 17:00 |
Tenkawa | had plenty of servers with more | 17:00 |
Tenkawa | I like having 32.. been very nice | 17:00 |
Vall | Not great reliability, the Thinkpad, having been bought in 2012 (will be 10 years old in a few months) never had to have anything replaced | 17:00 |
Vall | Yeah RAM is like, the more the better | 17:00 |
Tenkawa | indeed | 17:01 |
Vall | With ZFS ARC it never goes to waste | 17:01 |
Vall | (or the standard OS dynamic buffer-cache, as in Linux and I believe also in OS/X) | 17:02 |
Vall | Wife calling, which means chores a-waiting | 17:03 |
Vall | Nice talking to you, Tenkawa -- and c0rnelius, many thanks for the great assistance as usual | 17:03 |
c0rnelius | Vall: ur welcome | 17:04 |
Vall | c0rnelius: I will see if I can arrange a VM or something and contribute it to devuan-arm/arm-img-builder hosting, no promises but if I manage it I will let you know | 17:05 |
* Vall waves | 17:05 | |
Tenkawa | you too.. | 17:06 |
Tenkawa | and np and good luck with everything | 17:06 |
Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!