libera/#devuan/ Friday, 2021-07-09

cehtehhi, i need a hint, i updated a server to beowulf (was long overdue :D), now it drops me into the initramfs shell with no md devices visible00:41
cehtehanything changed from ascii -> beowulf?00:42
plasma41cehteh: What method did you use for the upgrade?00:44
cehtehapt dist-upgrade / aptitude and some manual fixing because not everything gone well00:46
cehtehserver is at hetzner, i am now in a rescue system00:47
plasma41cehteh: What filesystem is in use?00:48
cehtehext400:48
cehtehroot partititon is on    md0 : active raid1 nvme0n1p2[0] sdb2[2](W) sda2[1](W)00:48
plasma41cehteh: Is this using a hardware raid controller or software raid?00:51
cehtehsoftware raid00:51
cehtehself build kernel i am just checking if that missing anything (but i used a known working kernel config)00:52
plasma41hmm.. I'm not very familiar with either nvme or software raid00:55
cehtehfamopus words: it worked before :D00:55
cehtehok its not the kernel, devuan kernel doesnt boot either01:07
cehtehhi doc :D01:09
masoncehteh: chroot in, rebuild initramfs?01:09
cehtehmason: i did already01:09
masoncehteh: Any error message as it ditches you in the initramfs shell?01:09
cehtehthere are sme efi fuckups, nothing to worry about, otherwise nothing i see01:10
cehtehi suspect it builds the initrd somehow wong, but no idea what happens01:12
cehtehhttps://pbs.twimg.com/media/E5z2El5WUAYRgI1?format=jpg&name=4096x409601:13
masoncehteh: From rescue, delete your initramfs or move it aside and generate it fresh.01:13
masonI recently saw one consistently update wrong but do the right thing on a fresh build, even though I'd said 'update-initramfs -c' which ought to have started from scratch.01:14
cehtehthe missing dm-raid45  comes from me installed the dmraid package (but i useing mdraid)01:14
cehtehlemme try01:14
masonFailing that, lsinitrd from rescue and see if anything obvious is missing.01:15
cehtehwell no i am rebootin into the rescue system again and force-rebuild the initrd, moment01:16
cehtehok in the new generated initrd everything is there, rebooting01:22
cehtehnope, md stuff doesnt come up01:25
cehtehis there a mdadm spell to rescan and discover md devices?01:26
cehtehfrom initrd lsmod shows all raid modules are loaded, it just somehow doesnt discover them01:28
rrqis eudev installed?01:30
cehtehyes01:30
cehtehis that part of initrd?01:30
cehtehand damn remote screengrabber/console i cant enter a -01:31
rrqyes .. it all sounds like a (e)udev issue01:31
* cehteh wants to run a mdadm --auto-detect01:32
cehtehbut without able to type a -  grr01:32
rrqare the /dev/md nodes there?01:32
cehtehnope01:32
cehtehi think that causes the problem01:32
rrqyou have /dev/nvme* nodes?01:33
cehtehhey on a sitenote: can one install and start a sshd from initrd?01:33
cehtehyes all drives incl nvm are there01:34
rrqyou should be able to mount the /dev/ndme0p2 .. or whatever was the root, and then chroot into it01:34
cehtehi can do that from the rescue system yes01:35
rrqgrom the initramfs prompt as well01:35
cehtehbut the remote console (screengrabber, web browser based) sux, i cant type all characters01:35
rrqfrom01:35
cehtehah ... and the root is on raid01:36
* rrq oops .. need to go01:37
cehtehi could break the raid1 and somehow forcemount a component of that, but that wont solve anything01:37
masonnvme0n1p#01:39
masonnamespaces01:39
cehteh?01:40
masoncehteh: I just caught up on scrollback. So, you need to get the right mdadm.conf into the initramfs. That might not be the only problem, but it's necessary.01:40
cehtehnote: it ran before, i didnt changed the disc/raid config and i can start the raid and mount everything from the rescue system01:41
masoncehteh: So, like, mdadm --detail --scan >> /etc/mdadm/mdadm.conf01:41
masonin your rescue chroot01:41
masonthe edit that file and make sure there's no default/incorrect cruft01:41
cehtehthe madm.conf is correct01:42
masonthe update-initramfs -c -k all or however you like to do it, and it ought to know what it's looking for01:42
cehtehuuid based01:42
masoncehteh: Maybe try path-based in case it's having trouble finding the UUID? Random guess.01:42
masonThat seems unliekly though.01:43
cehtehit should at least have the device nodes /dev/md*01:43
cehtehbut there are none, no uuid/no name nothing01:44
masoncehteh: Maybe try explicitly adding them to /etc/initramfs-tools/modules ? Shouldn't be necessary, but it might be easier figuring it out from your booted system.01:45
cehtehthe modules are there01:45
cehtehlsinitramfs showing them01:46
cehtehthe scripts too, i just didnt figure out yet what the scripts are doing01:46
cehtehlsmod from initrd also shown that the modules are loaded01:47
cehtehit just doesnt scan/find any md devices01:47
* cehteh never had that problem before01:47
masoncehteh: Did you say you can construct them by hand?01:48
cehtehfrom the rescure system they are there01:49
cehtehautodetected01:49
masonIf you can do it from the initramfs environment, you can mount up your root on... is it /sysroot?  Get everything loaded by hand, and boot from there.01:49
cehtehfrom the initrd i cant because i cant type into that crapp screengrabber console01:49
masonNot from the rescue system, from the actual busybox.01:49
masonheh01:49
cehtehtry to do shell stuff without the - character01:49
masonIt's worth a try. mdadm --start --scan, lvm as needed, get the root into place and let the system pivot after you exit01:50
cehtehcant type -01:50
cehtehyes wtf01:50
masonThat's weird.01:50
cehtehwell its some screengrabber thingy thats remote over webbrowser support hooks that up01:50
masonOh, oh, I see.01:50
masonI forgot that detail.01:51
cehtehplugging vga and keyboard in doing magic sauce01:51
cehtehthe wrong way01:51
cehtehtotally fucked up keyboard layout01:51
cehtehi tried all keys with shift and with AltGr .... no -01:52
masonIf it's just the layout, can you run loadkeys and get a saner one?01:52
cehtehhow to install keymaps into the initrd?01:52
masonlooking01:53
cehtehloadkeys is there found that, but couldnt load 'de' :)01:53
cehtehideally i would like a initrd that has a failsafe that fires up network and dropbear01:54
masoncehteh: Well. If there's a loadkeys binary, maybe you can construct an initramfs that contains enough to get you a hyphen anyway.01:54
cehtehgetting rid of that crap screengrabber01:54
masoncehteh: You can construct one of those and drop it in, but dropbear ... been a while since I did that, and I stopped doing that as there was some random instability after I booted.01:55
cehtehi dont mind a bloated initrd ... i jsut dont know how to add such things there, i wish initrds would be minimal *real* systems01:55
cehtehyes i did something like that too, thre are some network fuckups in the kernel01:55
cehtehother story (you cant really deconfigure a network interface .. i mean *really*)01:56
cehtehputting it down isnt enough01:56
cehtehanyway thats another thing01:56
masoncehteh: So, I see a possible strategy for you. dumpkeys > foo, edit that, shim it into your initramfs. I've not built one by hand since before the started front-loading microcode, but then you'd have a keymap you could use with loadkeys and get a usable console.01:57
masonIt's a lot of messing around for a console, but it might be useful.01:58
cehtehwell the console can time out any time now (asked support for 2 hours, didnt expected so mcuh fuckup)01:58
masonYeah, this is weird. You'll have to be creative with it.01:58
masonHard to figure out at one remove and the kids are calling me to play a game, but if I think of anything I'll come back and mention it.01:59
Harzileinhi01:59
cehtehhi Harzi :)01:59
* Harzilein peruses the logs01:59
cehtehHarzilein: initrd doesnt pull up the md raid01:59
cehtehno idea why, all modules and stuff are there02:00
cehtehis there any special grub module needed for raid1?02:16
gnarfacei don't think so but LVM and FDE can both cause various complications when used together with it02:18
gnarfacealso, in case you didn't know, just using it for / can cause complications02:19
gnarfaceit actually isn't recommended to boot from a raid partition02:19
gnarfaceyou should put everything else on raid if you can afford it but the boot partition should have as few extraneous hardware requirements as possible02:20
adhocwhats FDE ?02:20
gnarfacefull disk encryption02:20
adhocok02:20
adhocmakes sense02:20
adhocputting /boot on a file system more complex that EXT4 has always been an issue02:21
gnarfaceon low-powered ARM devices, even using ext4 really bogs down boot times02:21
adhocjust recently went through the whole dance of getting ubuntu server 20.04 LTS to boot off a pair of M2's with mdadm02:22
adhocthankfully it allows you to configure that at install time02:22
adhocis a pian to do that afterwards02:22
adhocgnarface: ok02:22
adhocgnarface: why is ext4 an issue with booting on Arm ?02:23
gnarfacenot sure but i think it's more about the performance constraints of flash and low-power flash drive controllers02:23
gnarfacei observed that the problem doesn't even occur if booting from a ext2 partition using the ext4 driver in reverse-compatibility mode02:24
adhocbooting from flash, you have more appropriate filesystems02:25
gnarface(though it is more popular to use fat32 for /boot on such devices i think it is not a superior choice except if you think you need to repair your /boot partition contents from a stock windows install02:25
gnarface)02:25
adhoclike the jffs, etc ?02:25
gnarfacef2fs is the new one everyone is excited about02:25
gnarfacebut i don't know if anyone is using it for /boot yet either02:25
adhocok, how time change =)02:25
gnarfacealso, samsung contributed it so you know... probably loaded with backdoors02:25
adhocis there a particular class of device that you are refering to ?02:25
gnarfacewell, i observed it across the board on the allwinner a64 stuff that pine64.org is selling02:26
gnarfacephones and notebooks made from tablet guts02:26
adhocok02:26
cehtehi dont booting from raid1 since ages (close to 20 years perhaps) that always worked02:27
gnarfaceadhoc: but i think it is more about power margins than cpu performance02:27
gnarfaceadhoc: don't really know for sure02:28
gnarfacecehteh: it's not that it doesn't work it's just that we don't advise it because it makes both installation and disaster recovery complicated, which is a bigger threat to the type of people who are asking for advice on this type of thing in the first place02:29
cehtehhuh?02:29
gnarfacecehteh: it's a pitfall for noobs02:30
cehtehactually grub has excellent support for that it can boot even when the initial boot disc is defect when properly set up02:30
gnarfacecehteh: and it doesn't buy you much because the /boot partition shouldn't be getting read or written much at all under normal useage02:30
cehtehi am programmer not admin, but wont call myself noob02:30
cehtehit buys you that it boots even when the disc is defect02:31
gnarfaceif it's not raid002:32
cehtehmd0 : active raid1 nvme0n1p2[0] sda2[2](W) sdb2[1](W)02:32
gnarfaceif you know what you're doing then the warning doesn't apply to you02:32
cehtehwell i *mostly* thought i know what i am doing .. and it worked on many machines until this botched update02:33
cehtehand i have zero clue whats wrong now02:33
cehtehinitramfs is already linux, thus grubt could boot from raid1 ..02:37
cehtehjust somehow linux doesnt pull the raid up .. but i may check the mdraid conf again02:39
cehteh.. its correct02:41
gnarfacewould you need to do something to push the config change into the initramfs?02:43
gnarfaceor did you already?02:43
cehtehnothing changed02:44
gnarfacewait, it just stopped working between reboots?02:44
cehtehits the same config since 5b years or so02:44
gnarfacei thought you updated or something02:44
cehtehyes updated from ascii to beowulf02:44
gnarfaceoh, so the update broke it02:44
cehtehbut the raid config stayed the same02:44
cehtehyes02:44
gnarfacehmmm, disturbing02:44
gnarfacedoesn't have a nvidia card in it, does it?02:45
cehtehits a server!02:45
gnarfacewell, people do crazy things02:45
gnarfacelol02:45
cehtehyes i dont mine bitcoins02:46
rrqcehteh: what does the md conf file say? does it mention the nvme partitions?02:46
cehtehits uuid based02:46
cehtehbut moment ....02:46
cehteh 2      1000MB  32.0GB  31.0GB  ext4         root        legacy_boot02:46
cehteh.. thats the partition ... shouldnt that have the 'raid' attribute set for autodetection? or was that obsoleted?02:47
cehteh^ parted output02:47
cehtehset that, retrying02:49
rrqisn't there an /etc/mdadm.conf declaring how that raid is assembled?02:49
gnarfaceit's not just like, slowly rebuilding in the background or something, is it?02:50
cehtehin old times the kernel could scan for 'raid' partitions and assembled them automatically02:50
cehtehthe /dev/md nodes are not created and it falls into (initramfs) rescue02:51
* rrq admits not using md myself02:51
cehtehhttps://pbs.twimg.com/media/E5z2El5WUAYRgI1?format=jpg&name=4096x409602:51
cehtehand no the raid attribute didnt fix it02:51
cehtehi tought they obsoleted that .. anyway was worth a try02:52
rrqmodule dm-raid45 ?02:55
cehtehirrelevant02:55
cehtehafter the problem started i installed dmraid stuff (despite i am useing mdraid) .. rebuilding the initrd pulled some things in, but the kernel is build without dmraid4/502:56
rrqyou've been at https://askubuntu.com/questions/1219339/mdadm-error-opening-dev-md0-no-such-file-or-directory I suppose03:00
cehtehok tried to boot a stone age kernel .. same problem it really lies somewhere in the initrd03:02
rwpJust started reading the scrollback...03:02
rwpcehteh, Do you have the option to boot from an arbitrary ISO such as the installer image?03:03
cehtehthe rescue system boots from PXE03:03
rwpIf so then you could use it's rescue mode to bootstrap the system and then use that to fix whatever is happening now.03:03
rwpWhat system boots from PXE?03:03
cehtehthe rescue!03:03
rwpAnd it is a Devuan/Debian rescue image?  Excellent!  I did not expect that.  I expected something else from the hosting company.03:04
cehtehits in a datacenter, normallly it boots locally but i can enable rescue then a rescuesystem is pushed via pxe03:04
cehtehthere the md device are available03:04
cehtehif i would know whats wrong i could fix it03:04
cehtehbut i have still absolutely no clue whats wrong03:04
rwpThe installed system is more rigid in that it needs to know the mdadm.conf checksums to assemble the RAID.  But the rescue system is more loose and ad-hoc about dynamically assembling what it finds there.03:05
cehtehdid that change in devuan?03:05
rwpFirst, I don't know...  But I read that you are using a custom kernel?  Therefore I perhaps would try installing a stock kernel and seeing if it will boot successfully.03:06
cehtehthe config is correct03:06
cehtehi tried stock kernel same problem03:06
gnarfacewhat if you just run "update-initramfs -u" once just for good measure?03:06
cehtehdid update-nitramfs -c -k all03:06
gnarfacehmmm03:06
rwpChange?  AFAIK the /etc/mdadm/mdadm.conf file is configured into the initramfs and must match.03:06
cehtehyes it matches03:06
rwpBut in rescue mode one has the option to dynamically assemble all of the automatically found components.  Which is usually exactly what is wanted.03:07
cehtehit also matches with mdadm  --examine --scan03:07
rwpAlso, GRUB has changed recently.  I learned a lesson about how grub is configured due to a failed raid boot problem here.03:07
cehtehso the raid arrays are clean, fine, well configured03:07
rwpThe grub-pc packaging (ab)uses debconf as a registry.  It expects the boot drives to be listed in the debconf registry.  (I don't know about grub efi packages but think they don't.)03:08
cehtehgrub boots from raid (it loads the  initrd and start is)03:08
cehtehwith the initrd the mess begins03:08
rwpLearning that I went around to each of my grub-pc installed systems (ignored the efi ones) and "dpkg-reconfigure grub-pc" in order to fix them up.03:08
cehtehi guess i unpack the initrd and check whats exactly inside03:09
cehtehshall i try? dpkg-reconfigure grub-pc03:09
rwpI also know that the rescue image has all modules available.  Or at least most.  But installed system *might* have been configured more space conserving.03:09
rwpIf you have grub-pc installed then that seems like a non-harmful option.03:10
cehtehi lsinitramfs'ed it, all md stuff is there, sure something may be missing but i have no clue what03:10
rwpNone of us know exactly either.  All we can do is try to brainstorm with you.03:11
cehtehok installed grub on all discs (nvm and 2x spindle)03:12
rwpI will note that I have upgraded several systems from Debian Stretch up through into Devuan Beowulf all running RAID1 with mdadm software raid okay.03:12
cehtehyes i usually run that setup as well, sometimes debian sometimes devuan, all working03:12
rwpWhat that debconf registry does is that upon package upgrades later it will grub-install in the postinst to those registered locations.03:12
cehtehnever had that problem before03:12
rwpSo if those locations change then one needs to reconfigure the debconf registry for them.  Otherwise grub won't be automatically updated on those devices.03:13
cehtehdebugging initrd is always such a mess .. but this is really special cone is have no clue at all03:13
rwpAnd if grub upgrades the different phases and but does not upgrade the installed parts things get out of sync.  Errors occur.03:14
cehtehok problem stays after reconfigure03:14
rwp /me scrounged around in my bag and found this example error I have seen: error: symbol not found: 'grub_divmod64_full'03:15
cehtehreading through dmesg in initrd ....03:16
rwpIf this were -offtopic I would mention that RHEL/CentOS includes the UUIDs to assemble on the kernel command line and that if not correct then odd behavior occurs...03:17
cehtehi use uuids too03:18
cehtehwell dmesg shows that it probes the raid6 algos .. but nothing about trying to assemble the raid1 / boot device03:19
gnarfacecertain situations can trigger a UUID change though03:19
gnarfacei assume those have all been verified to still be accurate?03:19
cehtehi am pretty confident that a reboot isnt one of these03:20
gnarfaceno, it would have had to be a reformat or resize i think03:20
cehtehnothing of that happend03:20
adhocthe UUID should be set at file system creation time ?03:20
rwpSince this seems like a module missing from initramfs I might also ask if /etc/initramfs-tools/initramfs.conf says something other than MODULES=most there?03:20
cehtehbooting into rescue moment03:21
rwpFor mdadm the UUID is set when the mdadm command creates the raid.03:21
cehtehand iirc it is most and i check lsinitramfs before all modules are there03:21
cehtehalso in the botched boot (initramfs) shell lsmod shows all md modules loaded03:22
rwpIs this just mdadm raid or does it also include LVM on top of the raid?  (All of mine are LVM on top of mdadm raid.)03:23
cehtehit just doesnt discover and start the arrays03:23
rwpI saw questions in the scrollback about /boot.  Is that on the root partition or on a separate partition?03:23
cehtehthe root partition (incl /boot) is just an raid1 nvm + 2 disks (with write-mostly flag)03:24
rwpI always put /boot on a separate raid /dev/md0 mount point.03:24
cehtehyes03:24
cehtehthere is a 32gb root raid1 .. and a fat /data which is set up a bit more complcicated with dmcache .. but thats not my worries its not needed for booting03:25
rwpHmm...  So you are stuck in that crazy place where you can rescue and then chroot in and mount -a and start everything and it is running but it won't do it at boot time.  Hmm...03:25
cehtehyes03:26
cehtehwell rescue us some other system03:26
cehtehotehr kernel too03:26
cehtehbut i also tried to boot the old kernel which definately worked .. but did not now03:26
rwpWell...  The good news is that in that case you can get things running enough to sleep on the problem and attack it fresh in the morning.  That's always good.  Sleep is good.03:26
cehteh03:26
cehtehroot@rescue ~ # cat /etc/debian_version03:26
cehteh10.903:26
cehtehthe rescues is a debian03:27
cehtehlol03:27
cehtehi dont want to put it live from the rescue system03:27
rwpI don't think there would be any problem using the Debian rescue versus Devuan rescue.03:27
cehtehalso its not *that* important (some people may complain, like Harzilein ;) ) .. but its a private server no commercial stuff03:28
cehtehthe thing now is that i cant sit that out thinking maybe i get an idea, i am absolutely clueless, i have to find some way to debug this03:28
cehtehtried the devuan distro kernel, same problem btw, its not a kernel problem03:29
cehtehits somewhere in the initrd03:29
cehtehso i just unpack the initrd and investigate whats going on there03:29
rwpIt's a pretty unusual interesting problem.  And if the names I see helping in the above can't think of a way out then it's a tough one.03:29
rwpI think you may need to learn more about building a custom initramfs to debug your problem than you expected to at this moment.03:30
rwpSince the initrd is being run (apparently) you have a starting point to try to figure out why it isn't starting the sw raid.03:31
cehtehyes this sux initrd problems always suck didnt happen to me for a long time03:31
cehtehi'd wish one could make the initrd do some terse logging with some simple commandline option, is there any?03:33
cehtehideally to some file for later investigation03:33
rwpI do not know what I don't know about initrds but I imagine anything is possible there.  It's running code.  And that opens up a lot of possibilities.03:33
rwpIf it were me I would grab a spare box here, install, then build up a custom initrd and debug my debug locally.  Everything is easier locally.03:33
rwpThen after getting an understanding of things transfer that to the server.  Use that to debug the current problem.03:33
cehtehyeah problem is that this is a remote server and should be up soonish again03:37
cehtehand i bet i couldnt replicate the problem03:38
cehtehi can diff the old initrd from backup before the fuckup with the current one03:38
cehtehstill thats prolly a painful task because there are lots minimal details that may cause the problem03:38
rwp"Fast is slow.  Slow is fast."03:38
rrqcehteh: apparently mdadm has its initrd script at /usr/share/initramfs-tools/hooks/mdadm ..03:44
rrq.. and it includes the ability to "auto generate" the conf03:44
cehtehyeah i know that03:44
cehtehits a bit old but i once looked into how initrd works03:45
rrqand does "/usr/share/mdadm/mkconf generate" create the same conf that you have?03:45
cehtehhuh unpacking the initrd doesnt work, how is it compressed by default?03:51
rrqhow about failing line commenting out line 53 in /usr/share/initramfs-tools/scripts/local-block/mdadm, the re-update initrd ...?03:51
rrq(the line with "mdadm -q --run /dev/md?*")03:53
cehtehi am still struggling to unpack the initrd .. haha03:53
rrqyou'd edit on the host from where you pack the initrd03:53
cehtehyeah but i want to unpack it just to confirm its contents03:54
rwpgunzip -d or zcat the image into cpio and unpack it.  zcat initrd | cpio -ivt03:54
cehtehyes so i thought03:55
cehteh# zcat /mnt/boot/initrd.img-5.10.47ct | cpio -vi03:55
cehtehgzip: /mnt/boot/initrd.img-5.10.47ct: not in gzip format03:55
rwpcpio uses -i to set input of the archive, -t is table of contents.  cpio -idvm or something to create directories, verbose, modification times. But probably wrong as that was off the top of my head.03:55
cehtehwell gzip already fails03:56
rwpzcat might look at suffix and expect a .gz.  Try it with zcat < /mnt/boot/initrd.img-5.10.47ct03:56
cehtehnope03:56
rrqmaybe use "file /mnt/boot/initrd.img-5.10.47ct" to see how it's compressed03:56
cehtehtried that, says cpio .. but thats wrong unpacking shows on ly a feaction of its contents03:57
rwpWorks here.  My Beowulf systems all have a gzip compressed initrd.03:57
fsmithredyou have microcode installed?03:57
cehtehCOMPRESS=gzip03:58
rwpI also note that "find . -print | cpio -o -H newc | gzip" is how initrd files would be created and compressed.  It's that specific cpio format.03:58
cehtehyes03:58
rwpIf you can't unpack the initrd and the initrd is failing then is that perhaps part of the problem?03:59
cehtehmaybe03:59
rrqthe initrd might be a concatenation of 2 parts microcode+filesystem03:59
cehteh*but why*03:59
rwpIt's okay to concatenate gzip'd parts of initrd files together.03:59
rwpAs long as all of them are cpio -H newc type and all gzip'd the same.04:00
cehteh# file /boot/initrd.img-5.10.47ct04:00
cehteh/boot/initrd.img-5.10.47ct: ASCII cpio archive (SVR4 with no CRC)04:00
cehtehi am puzzled04:00
cehtehlsinitramfs shows me a lot content04:01
fsmithredyeah, you have microcode installed04:01
fsmithred (cpio -i ; zcat | cpio -i) < "$initrd_image"04:01
cehtehah04:01
cehtehyes microcode is in front i see04:01
fsmithredyou can thank rrq for that line04:02
cehtehyes now i see04:02
rrqI think that mdadm scripts would be at /scripts/local-block/mdadm in the initrd04:09
fsmithredrrq, I didn't read all the scrollback, but is this possibly related to what jaromil was talking about with grub?04:10
cehtehok ... i am a bit puzzled about UUID's04:11
fsmithredI avoid them for that reason04:11
rrqwell looks to me like poor scripting in that mdadm script04:11
cehtehmaybe i swith for labels04:12
gnarfacethere was some other situation they can change in that surprised me.  it wasn't just resizing and reformatting, i think i ran into a situation where the UUID changes if the fsck job has to repair errors on the filesystem or something too? the memory is vague now ....04:12
rrqthe failing line is bound to fail unless there already is some /dev/md? ... which gets assembled later04:12
gnarfaceyea, labels might be a smarter choice04:12
cehtehhttps://paste.debian.net/1203864/   puzzleed about whats partition whats filesystem and what the raid uses04:13
gnarfacewell they're all partitions04:15
cehtehsee that the UUID of ARRAY md0 appears 3 times below :D04:15
gnarfacethe important part is that the ones starting with nvme0* and sd* are physically existing partitions04:15
gnarfacethe others are virtual of some sort04:15
cehtehcan mdadm.conf use labels?04:15
gnarfacedon't know for sure but i would assume so04:16
gnarfacei mean if it can use UUIDs...04:16
cehtehstil lthings look correct04:16
gnarfacei think the uuid's are NOT supposed to repeat04:17
gnarfacethat seems to be in error04:17
gnarfaceafaik the whole point of a uuid is that it's globally and temporally unique unless something has gone very wrong04:17
gnarfaceit's right there in the name04:17
cehtehi tihnk thats the UUID of the raid04:17
rrqcehteh: are you up to adding a # to line 53?04:17
gnarfacebut, that said... you can hose your UUID algorithm with bad clock times in theory... though a collision doesn't seem likely anyway04:17
cehtehwell i am too tired now continue tomorrow04:18
cehtehand yes the uuids repeats for the other array too . .thats the shared uuid for the entire array04:18
cehtehlemme check on this computer but i tihnk thats correct04:19
cehtehyes its common to repeast04:19
gnarfaceif it is common for the uuid of the raid to match one of the partitions in it, then maybe it's just got the wrong one somehow?04:20
cehtehhttps://paste.debian.net/1203865/  <<< i have more fancs working setups :D04:20
gnarfacei admit last time i did this i was not using uuid's either04:20
XenguyIt's official, you guys are off the hook, keep on rockin' in the free world04:23
cehtehrrq: line53 on what? /usr/share/initramfs-tools/scripts/local-block/mdadm ?04:24
cehtehrrq: that looks like thats the symptom of the error but that it doesnt recognize the arrays should be earlier04:25
cehtehgrr .. i suspect udev04:26
cehtehwhat is eudev vs udev from backports?04:29
gnarfaceeudev is devuan's fork04:29
gnarfaceit is supposed to be a drop-in replacement04:30
cehteh'supposed to be' :)04:30
cehtehwell i am just thinking if i can install the other udev to cross check04:30
gnarfacei had different but similar problems with both of them on ceres04:30
gnarfaceand the issues were isolated to only certain devices04:31
gnarface(mostly usb stuff)04:31
fsmithredudev in beowulf-backports probably should not be there.04:31
fsmithredWe have a udev package in beowulf, chimaera and ceres, and they all have +devuan in the version. The one in bpo does not.04:32
fsmithredalso it matches debian version numbers: 247x instead of 1:3.2x04:33
gnarfacehmm, an upgraded system would leave the debian udev in place though rather than replacing it with eudev, wouldn't it?04:33
gnarfacei thought i remember having to switch mine manually04:34
fsmithredno, we added the epoch to replace the debian one04:34
fsmithred1: makes it later than 247 or anything else04:34
fsmithredexcept 2:04:34
cehtehahh .. i had eudev an udev mixed installed, removing udev remains04:35
cehtehmhm these dont share names/collide strange ..04:37
fsmithredI think we had a dummy udev package, maybe in ascii04:38
rrqcehteh: would be worth a try with the commenting (#) + update initrd + reboot ..?04:38
cehtehnoted but i am just fooling around currently i am too tired trying that tomorrow04:39
* rrq time for lunch here :)04:39
cehtehthat also looks like it fails but is not the cause04:39
cehteh4:39am here04:39
cehtehand purging the old udev remains didnt fixed it04:40
cehtehdone for today04:40
user_reading backlog: cehteh: echo $'\x2d' -> -08:25
user_this is in bash, in sh you may have to use printf08:26
user_echo -e "\x2d" should work in ash too08:26
user_iow printf --help can be coded: `echo -e "printf \x2d\x2dhelp"`08:27
user_`` executes the string returned by echo08:27
user___I know I am late to the party, but better later than never.08:29
user___cehteh: your can't-enter-hyphen problem is a worthy support issue, you are a paying customer, should get imediate help on that from the resident admin even at night. It's probably something simple anyway.08:30
user___https://docs.hetzner.com/robot/dedicated-server/maintainance/kvm-console/ wow 8.5 EUR/hr above 3 hours?!08:31
user___Also, full instructions on how to support your keyboard in there, it is done graphically, in the kvm ui, not using loadkeys08:34
user___I must have come in too early for folks on here usually :)08:56
user___8.40 EUR/3hrs of hw kvm use is extortion. After 9 hrs of use you can buy a decent KVM of your own for the money, including shipping to Hetzner...08:59
user___Alas, I see the gaming / HDtv crowd have hijacked the KVM term again, it's impossible to find a PC-kvm by kvm alone. This after the kernel people hijacked kvm for kernel. Guys, there are 26 letters in the alphabet, spread out a bit.09:02
user___</rant>09:02
user___'kvm over ip price' search works09:06
user___Ok, I retract the above "after 3 hours you can buy...", the ip-kms got a lot more expensive lately.09:11
user___On topic: is there a specific devuan image "tested" in rack / remote vm / data center mode which is preferred? One with minimal installation, no gui loaded, and accent on manual network configuration?09:14
aloo_shuI'm hitting a wall, I got a small devuan ascii rootfs in a chroot under android (4.2.2, kernel 3.3.0), all nice, /dev /dev/pts /proc  and /sys are mounted over. When I'm trying ping 8.8.8.8, I'm getting socket: Permission denied09:14
user___Is the android rooted?09:15
user___Also, which ping -> post output09:15
aloo_shualso mounting over android's /system/bin and using androids ping from devuan's bash, works09:15
aloo_shuthe android is rooted, I'm root in- and outside the chroot09:15
user___Okay.09:16
aloo_shu/bin/ping (in devuan)09:16
user___dns queries use port 53 which is privileged, is it possible selinux or something else security related thinks that your chrooted system should not have access to root moe sockets (<1000) ?09:17
user___I mean ping uses port 0 not dns, sigh09:17
user___Also, this is really not a #devuan question, you may have more luck in #android-devel ?09:18
schillingklauswill it take again a year for devuan to move to the next major version after debian does?09:18
aloo_shuI *think* android 4.2.2 isn't yet that heavy on selinux - how could i check09:18
user___You are right, but you should ask in $above, not here09:18
* user___ notes even kvm-over-ip is hijacked by the same hdmi-extender crowd, they call it "over ip" because it uses RJ45 connected CAT6 cable ARGH09:20
user___The real pc-kvm-over-ip boxes got a lot more expensive to support crazy video modes on video cards. Back in the days when I dealt with these things, only SVGA needed to be supported, dead easy with relatively simple cheap chips.09:25
user___bbl09:32
aloo_shuanybody else? android security limiting the chroot env is pretty ruled out, I put a busybox binary into my chroot, busybox ping is working09:42
gnarfacesomething to do with failed ipv6 routing?  just guessing09:43
aloo_shuI'm specifically pinging 8.8.8.809:44
gnarfacecould just be crappy security, badly implemented, and your clever ruse completely fooled it09:47
onefangRecent Android version have toybox installed by default, but you said this was an old Android?09:49
schillingklausdo recent androids still use mksh?09:49
aloo_shuI rather think something is missing that devuan is expecting, maybe 3.3.0. kernel too old? using pung verbosely, I see it's trying to open a socket, then a raw socket after that fails, and is getting permission denied/protocol not supported errors09:50
onefangBut as was said before, take Android support stuff to an Android support channel.  This is a Devuan support channel.09:50
aloo_shudunno, this android is all-but-recent (4.2.2)09:50
aloo_shuI am trying to find out how devuan networking tools are trying to open sockets, and what they require to work09:51
schillingklausif he needs to write portable scripts for both devuan and the native android system, it is better to know the available tools on both09:51
gnarfaceafaik they should only need device permission in /dev09:51
gnarfacei assume you have that or the busybox one wouldn't work either09:52
aloo_shu^09:52
gnarfacekernel api seems plausible too09:52
onefangThe author of toybox uses Devuan, and used to be the maintainer of busybox.09:52
schillingklausRob Landley09:53
onefangYep.09:53
schillingklausbut he is also active in the musl world09:53
schillingklausfor musl is good for static compilation, which is an advantage for many intended usages of toybox09:54
aloo_shuI have had a debian wheezy chroot system with working networking on the same android before, what I cannot say, if this is due to the proximity in kernel version, or the fact that I used an app to configure the chroot that time, while I'm trying manually this time09:56
aplainzetakindI installed from netinst, but had wifi troubles (from the network) so I finished without downloading anything extra and ended up with the minimal system.13:36
aplainzetakindIs there a clean way to continue from a boot into this state or am I better off just reinstalling?13:37
fsmithredaplainzetakind, you're in the right spot to add what you want. If you want the full desktop, install task-xfce-desktop or similar.13:39
fsmithredotherwise, start installing the individual things you want.13:39
user___does he not need to make wifi work 1st?13:40
fsmithredoh, maybe13:40
aplainzetakindYeah I don't have wifi.13:40
fsmithredif there's no wire13:40
* user___ cheats and uses an openwrt flashed older router as wifi ap in client mode (!), to supply RJ-45 net to balky boxes while installing13:40
fsmithrednice13:40
user___klunky13:41
aplainzetakindNor ethernet.13:41
user___aplainzetakind: do you have a rj-45 at all? and wired ethernet nearby?13:41
fsmithredother options would be to install from the desktop dvd or server cd or one of the live isos. Any of those can give you a usable system without needing a netork connection.13:41
fsmithredand the live isos have wireless firmware already installed13:42
user___aplainzetakind: w/o wired net and wifi your usual option at this point is to d/l a bunch of relevant .deb packages to usb stick and mount that and use13:42
fsmithred(except for a couple of tricky broadcoms)13:42
user___and what fsmithred said13:42
aplainzetakinduser___: No cable nor ethernet connectivity.13:42
aplainzetakindI mean it's not too much trouble to do over with -desktop.13:42
user___aplainzetakind: ok so reinstall as fsmithred said or usb stick transfer what is needed13:42
user___wifi drivers are always a pita in linux, because of the non free allergy.13:43
fsmithredaplainzetakind, were you able to pull anything from the network, or did wifi not work at all?13:43
user___meaning, people hating to package nonfree drivers.13:43
fsmithredthe worst are the broadcoms that require a network connection to get the firmware that allows you to have a network connection.13:44
user___yes. And they're not the only ones.13:45
* user___ also has an ancient ZD1211 iirc based usb stick wifi for stubborn cases, that has a kernel driver.13:45
aplainzetakindI'm at the uni, there is a PEAP wifi which needs a username/pass pair, so that didn't seem to be an option and I logged into the Guest wifi, but that has the initial confirmation page nonsense.13:45
aplainzetakindBut, I will also need non-free drivers.13:46
aplainzetakindI think.13:46
user___aplainzetakind: oh, ok, do you have data on the phone? Open a connection sharing hotspot?13:46
aplainzetakinduser___: I can do that but I don't have an abundance of data. How much would it consume?13:47
user___Doesn't max 4GB I assume. A dvd worth. Likely much less.13:47
aplainzetakindI can create the usb from my other machine that's no problem. I think I'll be fine.13:48
aplainzetakindDoes the -desktop installer offer the proprietary driver?13:48
user___aplainzetakind: do you see ssid's in the output of iw scan ?13:48
user___now as is?13:48
fsmithredmost wireless firmware is in the iso. You can almost always ignore that first window that asks you to load firmware from usb.13:48
aplainzetakinduser___: With this installed minimal state? ifconfig doesn't show wifi.13:49
aplainzetakindNor ethernet actually, only lo.13:49
user___what about dmesg?13:49
user___interface needs to be up to show. ifconfig -a might help13:49
fsmithredbrb, reboot router13:50
user___also driver loaded for wifi and eth13:50
aplainzetakindHmm, dmesg says some iwlwifi has been loaded.13:50
user___nice, which one.13:50
aplainzetakind8265-3613:50
aplainzetakindSo why don't I have tha interface on ifconfig?13:51
user___and does it then list a device? Show the dmesg lines for this13:51
aplainzetakindFfff I forgot -a13:51
user___Even ifconfig -a does not show it if the driver is not loaded13:51
aplainzetakindIt does though.13:52
user___Do you see a wifi0: in dmesg?13:52
aplainzetakindI have wlan013:52
user___Nice.13:52
user___Now try: ifconfig wlan0 up13:52
user___iw scan wlan013:52
user___sorry iw wlan0 scan13:52
aplainzetakindwlan0 is down13:53
aplainzetakindifup wlan0 says no such interface.13:53
user___it needs a ssid to be brought up as sta13:53
user___do you know how to configure manually?13:53
aplainzetakindNot really. I probably did it once.13:54
user___https://linuxconfig.org/connect-to-wifi-from-the-linux-command-line13:54
user___do you have wpa_cli ?13:54
aplainzetakindYes.13:55
user___then follow the guide in the link13:55
aplainzetakindwpa_supplicant is not up though.13:56
user___start it from /etc/init.d13:56
user___no, manually13:57
aplainzetakinduser___: Eh. What was the command to issue?14:01
fsmithred_wpa_cli?14:03
aplainzetakindI always get tied up with wpa_supplicant14:03
fsmithred_he said to start something from /etc/init.d but wpa_supplicant is not there14:04
aplainzetakind-B -D nl80211 -i wlan0 or something I guess.14:04
fsmithred_I don't do it manually, so I have no idea.14:05
fsmithred_oh shit. I already mentioned there's no init script, but there are service files14:05
aplainzetakindYeah there should be files for interfaces I think.14:05
aplainzetakindI don't have them, but I think I should create one.14:06
fsmithred_yeah, you can put a stanza in for wlan014:06
Guest4hello friends14:09
Guest4I was away from freenode for a while.14:09
Guest4I came back to an empty house14:10
Weeezywhy are we here?14:10
fsmithred_you really don't know what happened?14:11
WeeezyNo I don't, sorry.14:11
fsmithred_google for freenode and andrew lee14:11
Weeezyok. thanks.14:12
fsmithred_hostile takeover14:12
Weeezymy registered login was deleted on freenode14:12
fsmithred_oh yeah. They all were.14:13
Weeezywhy this network tho?14:13
fsmithred_this is the fork. Almost all the staff left to fork this.14:13
Weeezyok.14:14
fsmithred_and many channels moved with the staff. (community)14:14
user___aplainzetakind: if you don't have a config for supplicant, generate one using: `wpa_passphrase networkname password > /etc/wpa_supplicant/wpa_supplicant.conf`14:16
user___aplainzetakind: this is in the manual setup link I posted above14:16
user___aplainzetakind: then start wpa_supplicant with that config14:16
user___you still need to say which interface with -i and daemon -B14:17
aplainzetakindAh alright. Thanks.14:17
user___iw wlan0 scan; should output the ssid's it sees. Try that 1st14:17
user___if you don't see ssid's there is more driver work ahead. But Intel drivers should work.14:18
user___I.e. no more driver setup needed.14:18
aplainzetakindI will write a dummy config and manually edit it to the PEAP network stanza from the other computer.14:18
user___yes14:18
user___good plan14:18
user___and it's: iw dev wlan0 scan14:20
user___https://wiki.archlinux.org/title/Network_configuration/Wireless#iw_and_wireless_tools_comparison good resource, as usual14:20
Weeezywas or is this Lee guy the one using bots to spam other networks? My quakenet channels were all being spammed with libera.chat propaganda14:22
aplainzetakindIsn't the way to specify the file -I ?14:22
user___Weeezy: that was a false flag operation done by disgruntled freenode evil geniuses14:23
user___aplainzetakind: -c14:23
Weeezythat's what I was thinking14:23
user___aplainzetakind: -i $interface -c $config_file -B14:23
Weeezythanks.14:23
user___Weeezy: freenode evil geniuse means NOT real freenode staff or users14:23
user___+s14:24
aplainzetakinduser___: Alright wifi is up.14:24
user___nice. And are you connected?14:24
user___ping?14:24
aplainzetakindNo, for some reason.14:24
aplainzetakindLet me try my phone's hotspot instead.14:25
user___right, edit /etc/resolv.conf14:25
aplainzetakindThat PEAP thing may be hairy.14:25
user___do you have dhclient running or not?14:25
aplainzetakinduser___:14:25
user___do you have an IP14:25
aplainzetakindNo.14:25
user___...14:25
aplainzetakindip r is silent.14:25
user___so maybe dhclient wlan0 ...14:25
aplainzetakindCool, can ping.14:26
aplainzetakindThanks.14:26
user___then find out the ns address (usually the router if nat'd), and edit /etc/resolv.conf14:26
aplainzetakindShould I hardcode that at this moment?14:27
aplainzetakindThis is just a momentary network.14:27
aplainzetakindAren't these manual settings to be superseded when I install more stuff?14:27
user___yes, wicd or networkmanager will do all this for you14:28
user___OR you can write /etc/init.d/wlan0net and have it linked in your runlevel by hand14:28
user___OR put it all in /etc/rc.local until you finish installing14:29
user___OR write all to a shell command file like /root/wlan0net.sh and source that as root after rebooting14:29
user___:)14:29
user___OR install systemd and write a unit file for it, and make it work for you (see you in a few weeks if you go this way)14:29
aplainzetakindHaha, no.14:30
aplainzetakindI run bare wpa_supplicant via wpa_gui on my other computer.14:30
aplainzetakindBut I want to try out connman.14:30
user___So, in a nutshell, what you did so far: wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -B; dhclient wlan0; cat my_resolv_conf >/etc/resolv.conf14:31
aplainzetakindI didn't do that last bit.14:32
user___What's the output of host google.com14:32
aplainzetakindI don't have host.14:32
aplainzetakindLet me get it.14:32
user___yes14:32
aplainzetakindHmm I don't have repos configured. Only looking at cdrom:// stuff.14:33
aplainzetakindHold on.14:33
user___aplainzetakind: you need dns before you can use named hosts...14:34
user___aplainzetakind: try this (I hate myself): cat "8.8.8.8" >/etc/resolv.conf14:34
user___in theory dhclient should configure resolv.conf so you can skip that14:34
user___you can try ping ping.com14:34
user___It will stall but show the resolved ip14:35
user___kill it with ^C14:35
user___bing.be hates the fact they got that host name >:-)14:35
aplainzetakindI can ping google.com I have name resolution.14:35
user___Ah it's no longer owned by ping.be, it's someone in Phoenix, AZ14:36
user___aplainzetakind: okay then go on and configure sources.list and continue installation14:36
user___aplainzetakind: imo, etch the steps to get networking going on linux into your dna. What you did today :)14:37
user___Note pinging external hosts may not work in some lan/walled garden situations, but name resolution should work, and ping prints that in the 1st line of output even if there is no echo reply.14:38
user___and it was: echo "nameserver 8.8.8.8" >/etc/resolv.conf above. Sigh, it's been a while since I needed this.14:40
aplainzetakindAlright, sources.list good, apt-get update'd.14:41
aplainzetakindHow do I "continue with the installation"?14:42
aplainzetakinduser___: I want to eventually startx into xmonad with no DE or anything.14:46
user___Well, install what you want to run. I don't know about xmonad.14:46
user___I use aptitude usually to search for and install packages14:47
user___Usually one needs some sort of login manager. Especially on a public machine. Like slim14:48
aplainzetakindThis is a laptop.14:48
user___Is it your or shared?14:49
user___You probably want a login there anyway.14:49
user___slim can do this for you14:49
user___Unless xmonad handles xsession logins14:49
* user___ installs xnee 14:49
aplainzetakindMine.14:50
aplainzetakindI think I'll be fine with plain startx.14:50
user___okay.14:50
aplainzetakindI'll get some screen locker.14:50
user___your machine, your rules14:50
aplainzetakindBut, with openrc, is installing connman and doing rc-update add connmand sufficient?14:52
fsmithred_aplainzetakind, I haven't used connman with openrc, but I would expect that just installing connman is enough. Openrc uses the sysvinit scripts.15:07
fsmithred_so does runit15:07
aplainzetakindfsmithred_: Turned out it is.15:08
fsmithred_I know there are runit scripts for connman that work, but I don't know about openrc.15:09
fsmithred_probably need to look at gentoo for examples15:09
Weeezythis is not a huge deal but I'm wondering why I can't get devuan to report v 3.115:11
WeeezyI've tried all the commands I know but still reports v 315:12
fsmithred_Weeezy, because we didn't update base-files to say 3.115:12
user___I see we don't have recapp in devuan? com.github.amikha1lov.RecApp ?15:12
fsmithred_if it said 3.1 and you did update/upgrade, the 3.1 would no longer reflect the true state of your system.15:13
fsmithred_not 3.2, but newer than 3.115:13
fsmithred_the point releases are releases of new isos15:13
fsmithred_the number reflects what's in the iso, not what's in your installed system15:14
user___we also seem to not have xrebind from gnu15:14
fsmithred_does debian have those packages?15:14
user___Looking. gnu certainly, recapp is in fedora and is open source.15:15
fsmithred_we don't take packages from fedora, we take them from debian15:15
fsmithred_and those two are not in debian that I can see15:16
user___xrebind seems to be 2006 vintage...15:16
user___xrebind allows keys to be bound to macros in X11, which can be useful, since macros will also issue mouse moves, etc.15:17
fsmithred_is it older than xvidtune?15:17
user___I don't know. Trying to build15:18
user___../include/xrebind.h:34:10: fatal error: X11/extensions/XTest.h: No such file or directory15:18
user___libx11-dev is installed15:19
user___libxext-dev too15:20
fsmithred_libxtst-dev: /usr/include/X11/extensions/XTest.h15:21
user___don't have that15:21
user___ok, found it, wrong re search15:22
user___+libxosd-dev15:23
user___xrebind built on beowulf with minor erros15:24
user___*errors15:24
fsmithred_time to go. bbl.15:27
user___works fine with examples15:27
user___thanks, bye15:27
user___grr gnu makefile does not install info and man pages15:29
user___it is not debianized15:30
enycuser___: what do you mean gnu-makefile --  the "Makefile" included in source-package of a particular GNU official project ???15:31
user___The output of ./configure in package xrebind-0.6.tar.gz15:42
user___I hacked a manpage from the output of xrebind --help if anyone needs it say beep15:43
user___Should bluez-tools work without pulse?15:54
GyrosGeiernon-audio bluetooth stuff should always work without pulse15:56
GyrosGeieraudio bluetooth stuff is old15:56
GyrosGeierand pulse have a userspace implementation of the protocols15:57
GyrosGeierso if you want headsets to work without pulse you need a kernel module15:57
cehtehuser___: unfortunally these docs seem to be out of date, the kvm console opens here as webapp not java (which sucked years ago, but i may try to fall back to that if possible) and there was no way to define the keyboardlayout in the webapp16:26
user___Hmm16:43
user___cehteh: open an urgent support ticket?16:43
user___cehteh: echo -e "\x2d" works aanyway as a hack16:44
cehtehnit when you cant enter \ either and | is missing too :D16:44
cehtehi opened a support ticket already16:44
user___ouch16:44
cehtehalmost any symbols are missing or out of place, and its not just german vs us layout because - gave ß16:45
user___On a US 104 key keyboard - is the key to the right of '0'16:45
user___I assume you tried all with and without shift16:46
user___Do you get underline _ ? The - is normally above it or under it (unshifted)16:46
cehtehyes o know16:46
cehtehon german keyboiard - is rightmost lowest row, besides shift16:47
cehtehand the ß is right of 016:47
cehtehahh i need coffee16:47
cehtehäöü worked surprisingly16:47
cehtehso its somehow totally messed up16:48
cehteh() where like on us16:48
user___I know where it should be on .de and .us16:48
cehtehand no " either16:48
user___wow16:48
user___Did you try another browser?16:48
cehtehnot yet i gone to sleep16:48
cehtehcan try chromium leter16:48
user___Try another browser.16:49
user___What did you try until now? ff?16:49
cehtehyes16:49
cehtehwell its not ultra-urgend, no business in danger16:49
user___ff has some problems with newer js / css. No doubt deliberate (/me shakes fist at google)16:49
user___cehteh: 8.4 EUR + VAT / 3 hrs of KVM is a strong incentive to get it going fast.16:50
cehtehfirst 3 hrs are free and you can reopen the ticket, they only charge because they dont want peopel to hoard the consoles and have them continously connected16:52
cehtehhetzner support is pretty ok, i know the drill, yesterday i asked for 2 hours (just precaution, didnt know about the problem yet) but stated "its no urgend, but can you leave the console connected as long as possible over night" .. and i had the console the whole night16:54
user___look at the next bill...16:54
cehtehwill do but i had that before, usually its no problem16:58
cehtehanyway whiel eating now, i need to think about how to find and fix this problem16:59
cehtehcan i turn initrd into some kind of verbose mode? enable  set -x from kernel command line?17:03
user___Not that I know of17:03
user___set -x inside initrd init script might work17:03
user___i.e. /init inside initrd17:04
cehtehyes but iirc (from long ago) i remmeber there was some debugging help, i just look that up now17:04
user___er /linuxrc17:05
cehteh /linuxrc ?17:07
user___ /linuxrc or /sbin/init in the initrd is what is run by the kernel.17:07
cehtehah17:08
user___it is a sh script (not bash!)17:08
user___set -x should work17:08
cehtehok got the console now .. finishing eating and then go17:09
user___remove bread crumbs from keyboard to make '-' key work...17:09
cehtehno bread, no muesli :) .. was saussage and carbagge salad :D17:10
user___careful with npm updates on devuan, you'll get scary warnings https://www.theregister.com/2021/07/09/npm_audit_security/17:12
cehtehthey didnt answer me for the keyboars stuff :/ trying chromium now17:13
user___tell us how it goes17:13
cehtehno does not work17:13
user___hmm, tell support you also tried chromium.17:13
user___cehteh: copy paste into web terminal works?17:15
cehtehnope17:15
user___wow, report that too to support17:15
user___basically you have non working kvm-via-ip now17:16
cehtehits just a vga screengrabber sending a video stream17:16
user___we are interested in the other direction17:16
cehtehyeah17:16
user___maybe there's a "security" filter they put in to prevent miscreants who gain access to cause trouble?17:17
cehtehi tihnk they just migrated/updated from java to webapp and its broken17:17
user___sounds like it, so far. Nothing this broken would ever be deployed at a vh provider.17:17
user___sounds like a filter.17:17
cehtehir maybe i need to install java here to fall back to java17:18
user___you can try that but you should wait for their answer.17:18
user___Seriously, web shells in js work perfectly, like shellinabox17:18
user___It is trivial to mod it to show a video stream instead of a real terminal.17:18
cehtehwhat is even needed to enable java from browser? havent done that in ages17:19
user___Oh you can forget about that, every warning will light up.17:19
user___Google is not on good terms with Oracle17:20
GyrosGeiernah, that's just banter between friends17:20
user___GyrosGeier: who occasionally whack each other's head with golf clubs.17:20
user___https://java.com/en/download/help/enable_browser.html difficulties? With the database? Of a database provider? Hah.17:21
cehtehmhm i suppose changing keyboard layout in the bios wont work, but i give it a try17:21
GyrosGeierwell, they might disagree on licensing, but they still won't poach each others' employees.17:21
cehtehsearching if that bios has some other remote admin stuf like ipmi or so17:21
user___GyrosGeier: of course not, that would push salaries up.17:22
user___cehteh: that may be even harder to connect to.17:22
cehtehpossibly17:23
user___cehteh: install jre and the plugin, for firefox. Then enable plugin, exit and restart the browser. In theory.17:25
cehtehok bios keyboard change didnt help (as expected)17:29
user___cehteh: I am now 90% sure it's a "security" feature.17:33
user___Ask support if they mistakenly enabled some kind of security filter on the kvm17:33
cehtehi doubt, but i opened a ticket17:33
user___Also, can you copy/wget a url from the initrd?17:33
cehtehwhat?17:34
user___You can pack single character files with what you need and unpack them there, then `cat $bar` etc17:34
user___$pipe17:34
cehtehhttps://pbs.twimg.com/media/E5z2El5WUAYRgI1?format=jpg&name=4096x4096   << thats a screenshot of the console17:35
user___this is a crazy situation imo. Support should help you asap.17:35
cehtehyes17:35
user___what does the menu tab on the left do?17:35
cehtehno keyboard layout switcher there17:36
user___busybox should have wget17:36
cehtehsome menus i check them all17:36
cehtehinitrd doesnt have network running17:36
cehtehi may setup dropbear in initrd but thats famous for other problems17:36
user___that is a crap kvm, using letterbox mode due to low resolution. Maybe ask for another kvm? I assume the nonfree ones are not buggy, else there would be hell to pay.17:36
cehtehi cna fullscreen it :D17:37
user___sigh17:37
cehtehbut 4k monitor here17:37
cehtehanyway its just vga screengrabber, no more no less17:37
cehtehand i want to focus on the problem with booting now rather debugging remote consoles17:37
cehtehi am in the rescue system again and can hack the initrd17:38
cehtehthe rescue system start the raid arrays even without knowing about them17:39
user___did you do what fsmithred_ said? About initrds from rescue/stock system?17:41
cehtehno lemme scroll up17:41
cehtehhuh what he saied a lot :D17:42
cehtehah initrd parses the commandline and has a debug mode17:44
Harzileincehteh: look into that hyper.m4 thing that used to come with consoleutils (?, kbd-utils?)17:47
cehtehwtf what?17:47
Harzileincehteh: then you should be able to enter both hex and decimal ascii codes for the keys17:47
Harzileincehteh: had it when i had a 85 key keyboard17:47
cehtehi am in initrd17:47
Harzileinbut there's a loadkeys there?17:47
cehtehtell me how to install a keyboard layout in initrd AND provide me one that fixes this borked layout17:48
cehtehthats prolly an afternoon work alone17:48
Harzileinkk17:48
Harzileinyeah, sorry for digressing17:48
cehtehanyway i am now looking at the initrd log :)17:49
cehtehall i see is that it somehow doesnt find device by UUID, but the erorr must be before that it doesnt start the raid18:05
cehtehwow i found the - !!!!18:09
cehtehok problem: mdadm --detail --scan should list all discovered raids (config)18:12
cehtehbut doesnt do any18:12
user___where was -18:13
cehtehon the #18:13
user___O.o18:13
user___You did say you tried all keys ... did you find | too?18:14
cehtehoverlooked that before, did i say that this console has half a second delay18:14
cehtehnot yet :D18:14
user___Eww18:14
cehtehand some keys generates multi key event incl delete / backspace chars18:14
user___just cat <<EOF >/dev/null and type all keys, then EOF18:15
user___ok, that is fun. Nothing from support?18:15
cehtehthen i dont know what relates to what key18:15
user___(now you know why they let you have this kvm all night with no payment)18:15
cehtehwaiting, ticket received18:15
cehtehanyway with - i can go on18:15
user___find the pipe and >< too, soon.18:15
cehtehi see that the kernel/mdamd somehow doesnt find any raid member18:16
cehtehmdadm v4.1 is correct?18:16
cehtehwasnt there a command to make the kernel to rescan/rediscover all block devices?18:18
user___4.1 is what I have on Beowulf18:18
cehtehwell blkid does list them18:20
user___cehteh: https://serverfault.com/questions/36038/reread-partition-table-without-rebooting18:20
user___speaking of block*, you do hatev the block* tools on the initrd?18:21
user___blockdev blkid etc18:21
user___blockdev --rereadpt /dev/sdX ; is rumored to work18:22
cehtehpartprobe is available worked .. but mdadm / raid doesnt pick that up wtf18:22
user___the page suggests 'systemctl restart systemd-udevd.service' <evil grin>18:23
user___so, ahem, is udevd running? eudevd perhaps?18:24
cehtehsystemd in initrd :)18:24
user___yup.18:24
user___Hey, it *is* the kernel already...18:24
user___that is from my link above. The scary part: comment: "The top accepted answer is incomplete: in the modern systemd world, THIS is the correct answer. Please note you also need to restart one of those (or both) systemd-udev-settle and systemd-udev-trigger. Restarting just systemd-udevd like Camp said was not enough for me."18:25
cehtehaah udev isnt running18:25
GyrosGeierhdparm -y /dev/sdX ?18:25
user___You have THREE daemons for ONE task? And they don't know about each other...18:25
GyrosGeierwait18:25
GyrosGeier-z18:25
GyrosGeierwrong keyboard18:25
user___cehteh: I think you need eudevd?18:25
GyrosGeierudevd is the daemon18:26
cehtehi started udevd which started as eudevd .. i guess its an alias18:26
GyrosGeierudev-trigger tells the kernel to generate hotplug events for all existing devices18:26
GyrosGeierudev-settle waits until all events are processed18:26
cehtehbut i had to likk it because its not forked in the background, whats the option forthat?18:26
GyrosGeierso the latter two are tools18:26
user___cehteh: --daemon18:27
GyrosGeierbut generally, rescanning is bus specific18:27
GyrosGeierthe udev stuff only deals with stuff that was found during a scan18:27
cehtehjust entering a command on that console is pain18:27
user___yes but he is scanning manually now GyrosGeier18:28
user___cehteh: if you type blindly, does it lose characters?18:28
cehtehit looses characters or repeats them multiple times18:29
user___wow18:29
cehtehor enters random conrol characters which are not visible and mess the command up18:29
GyrosGeierthat can happen when two shells are attached18:30
GyrosGeierand both try to consume characters18:31
user___GyrosGeier: or when the web to kvm interface has throttling and one of the parts has resend enabled18:33
user___which is crap18:33
cehtehok udevd was running .. i rebooted now to get again a clean state18:33
user___You can't win this, the hw kvm is crap18:34
user___cehteh: and all your startup settings are lost? Why do you keep rebooting?18:34
cehtehwhat startup settings?18:34
user___This is not windows, it does not fix/break itself18:34
user___If it did not boot 1st time it won't the following 99 times either.18:35
cehtehi wanted the kernel in a clean state, i tried a lot commands18:35
cehtehrediscover partitions, reload modules18:35
cehtehso replied to the ticket, they asked me whats wrong :D18:43
cehtehok clean state now, udev is running,18:45
cehtehi somehow suspect udev is at fault18:46
GyrosGeierwait18:47
GyrosGeieryour problem is mdadm --scan ?18:47
GyrosGeierthat is independent from udev18:47
cehtehmhm18:47
GyrosGeierbut it needs to happen after "udevadm settle"18:47
GyrosGeiermdadm scan will generate more udev events18:48
GyrosGeieras the MD devices come online18:48
GyrosGeierso if following code depends on them, do another udevadm settle18:49
cehtehok took me 1 minute to enter 'udevadm settle' and 'mdadm --assemble --scan'18:51
cehtehno change18:51
GyrosGeiermdadm will not assemble degraded arrays in --scan mode18:53
cehtehthe arrays are not degraded18:53
GyrosGeierwell, if one of the devices can't be found, they are18:53
cehtehthe devices are all there18:53
cehtehonly md does not start up18:53
cehtehand no entry in dmesg that it even tries18:54
cehtehda können wir leider nichts machen, das Layout wird aufgrund Ihrer lokalen Einstellungen angepasst.18:54
cehtehWas genau wollen Sie denn mit der KVM machen, gibt es etwas wobei wir behilflich sein können?18:54
cehteh^ reply :D18:54
golinuxcehteh: You might get bounced if you paste too many lines18:55
GyrosGeierneeds modprobe md?18:55
GyrosGeierand possible some of the personalities18:56
cehtehGyrosGeier: the modules are loaded18:56
GyrosGeiermdadm --assemble --scan --verbose18:58
GyrosGeierthat might also help18:58
cehtehaah18:58
cehteheww .. it does not scan paritions18:59
GyrosGeier/etc/mdadm/mdadm.conf18:59
cehtehis correct18:59
GyrosGeierthere is a DEVICE line there18:59
GyrosGeierif it's commented out or contains "partitions", then it should scan partitions19:00
cehtehah19:01
cehtehthat might be te problem (worekd before do)19:01
cehtehok rebooting into rescue and fixing that19:02
cehtehGyrosGeier: <3   ... i owe you a beer!  that was it. and thanks to everone else helped!19:09
GyrosGeier\o/19:09
cehtehmakes me wonder why it worked before ...19:10
GyrosGeierthe initrd also scans for and sets up md devices19:10
GyrosGeierin case root is one19:10
GyrosGeierso this is run twice19:10
GyrosGeierand each has its own copy of the config file19:11
cehtehyeah i usually known that it pulls up any array thats non degraded and finds, suddenly that seems to be not the case anymore19:11
GyrosGeierso I suspect something changed with the initramfs generation19:11
cehtehyes i pretty much narrowed it down to a initrd problem19:12
cehtehHarzilein: ping19:12
cehtehand surprisingly --- all containers came up :D ... just running without a hitch now19:14
user___And what fixed it?19:23
user___ok read backlog. DEVICES= eh19:26
user___re: ticket: cehteh: insist on clarifying it, tell them you can't type certain keys and about the 2 second delay19:26
rwpcehteh, For reference what was the root cause of the problem?  The "DEVICE partitions" was something different?19:27
cehtehi had all DEVICES   where raid partitons are on listed, but not as wildcard, just the main device19:27
cehtehthat woreked before obliviously19:27
cehtehnow i just changed it to DEVICES partitions19:28
cehtehwas a goodwill attempt to constrain the config .. and still a fault19:28
rwpGood deal that you now have it working!  I'll file that tidbit in the back of my head in case it happens again.19:34
rwpGood job to GyrosGeier for that root cause possibility!19:34
cehtehits almost always such an simple fuckup .. still hard to find unless you have an idea what may cause it19:38
cehtehesp no indication of any error until i did the mdadm --assemble --scan --verbose which shown the devices it scanned and failed19:39
rwpThere are an infinite number of ways for things to fail.  But usually only one way for them to work.19:41
rwpThe smallest of details can lead to the biggest problems!19:43
Harzileincehteh: pong (trying to find your message window on this here emergency irc client ;)20:18
cehtehHarzilein: server works again20:20
cehtehno1 in  1023 days, 23:34:29 | at                        Sun Apr 28 17:57:00 202420:23
cehtehuprecords  :D20:23
cehtehthat'll be some time20:23
Harzileinnice20:30
HarzileinGyrosGeier++20:30
Harzileinhi btw :)20:30
xrogaanAll the app icons in my application menu vanished20:35
xrogaanthey're still present if I use the editor though20:35
xrogaanthat's weird.20:35
xrogaanand wiskers show them too20:35
user___xrogaan: changed permissions recently? bulk changed?20:46
xrogaannah20:53
xrogaanthey're showing fine in other plugins (whisker)20:53
xrogaanit's just not showing in the regular application menu20:54
xrogaanso, probably not the icons themselves20:54
user___windows remedy: restart the desktop21:04
msiismIs there a way to display the current font and size used on the TTY?23:22
fsmithred_dpkg-reconfigure console-setup to change it23:23
fsmithred_oh, look in /etc/default/23:23
msiismOkay, thanks, will do.23:24
* msiism found it.23:24
fsmithred_you can just edit that file. Probably quicker.23:25
msiismYeah, I don't want to change the font. I really just wanted to know what I use.23:26
Centurion_DanHi23:27
Centurion_DanI came across an old installation of Devuan Jessie today on a laptop I setup probably 5 years ago.... Do you think I can still upgrade up to Beowulf and Chimera without it breaking??23:28
gnarfaceif you upgrade through ascii first it's theoretically possible23:31
gnarfaceit has worked on some of my headless machines anyway23:32
gnarfacewell, up to beowulf anyway23:32
msiismI'd collect all the delicate items from all the release notes before doing that.23:32
fluffywolfCenturion_Dan:  it should be fine to beowulf, but are you really sure you want chimaera?23:36
fluffywolfactually, never mind, chimaera should be reasonably usable now.23:36
fsmithred_check eudev version. You may have to upgrade to a lower version number.23:39
fsmithred_that's the main gotcha I can think of23:39
fsmithred_is eudev in jessie?23:40
fsmithred_Centurion_Dan, you might do well to skip past ascii23:42

Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!