What can I say, I didn't want to wait on my provider lol
It was also actually pretty fun!
It was also actually pretty fun!
Not really, I pulled it together from a bunch of random posts lol
Maybe I should write one, but in essence you:
Stop all non-OS essential services
Create a filesystem in a chunk of RAM
Pull essential OS files from the installed OS into it recreating needed directories (Though you could probably just use a tiny pre-built distro but meh)
Pivot root into it
Reload services (when they restart they'll be restarted in the context of where you pivot rooted, prior they're still running under the context of the installed OS)
Unmount the boot drive
Then do what ever you need to do
???
Profit
chroot isn't enough, you need "real" pivot_root
chroot just changes the shells root point, pivot_root actually changes the root mount point and enables this trick to work
The man page has a good example.
FYI you can use kexec and a prepared initrd to do something similar with only one command.
Imagining this process, despite having never done 50% of the steps on the list, makes my brain imagine this: 
Yeah, I'd love to see a write up on this to follow.Sounds like useful practice in the lab if nothing else.
What happens if the SSH session closes before dd finishes? Sounds pretty badass but I don't think I would trust this approach in prod lol
Well the 1 SSH session bit was for dramatic meme-effect lol, you can actually connect back without issue (at least it did for me) so worst case if you weren't working in tmux you'd just have to start dd again
Worst worst case, you'd just end up back where you were probably heading before anyways, KVM/IPMI
Sure, I suppose as long as sshd is up and running in the ramdisk environment (which you mentioned in another comment, along with all other services) you could always reconnect. Very neat and clever!
True, but I was more thinking about the issue of reconnecting in general when you just nuked sshd.
As long as you only copy off the disk, you can just reboot and the whole system in RAM vanishes and the normal system boots again for the second try.
I had a server I rented from a provider in a data center and I wanted to image (dd) the drive for archival before I decommissioned it from my infrastructure.
Normally, you can't really do that with the OS running and you would have to shut down and insert a live USB or something and temporarily boot from that. The server being a faraway rental the only option was to open a ticket so that they could enable an out-of-band management option like KVM or IPMI. Which would allow you to control a machine as if you had a physical monitor, keyboard and mouse in front of you. With that you can attach flash drives, shutdown, restart, see the POST/BIOS/UEFI screens etc remotely .
But, I didn't want to wait 6-8 hours for them to enable that so instead I put together a process that would "boot" me into another distro "installed" into a RAM disk (kinda like how live CD/USB works) from the currently installed and running OS without rebooting
From there I could unmount the boot disk and do what ever I wanted from there, I could have even wiped the disk entirely and installed a entirely different distro if I wanted
How do you do that? What minimal distro did you use? Did you make it yourself? How reliable is pivot-root? How many tries did it take you to do that successfully?
RAM is still memory, so you can put anything in there, like an OS.
With pivot_root, you can change where the root is. So you pivot into your OS in RAM, which becomes the root of the system.
At that point, you can do pretty much anything you would normally do with your OS, like unmount a HDD.
I've never personally done it, but that's a simplified explanation of what OP did.
If that happens, they just fall back to the first option of waiting for their provider to set up the IPMI.
I've never gotten pivot_root to work well / as documented. Major props for being able to pull this off. Hopefully you didn't have much time where a connection-drop would require a reboot. SSH in the new root on an alternate port before the pivot? idk.
Really? It seemed to go pretty smooth for me, SSH dropped once I think when I reloaded the services (expected), but after that I was able to connect back fine on the same port no issue
It's been a LONG time since I tried, but ISTR it complaining that it couldn't move the old_root because it was in use? But, when I'd fuser it, the only think that was using it was my shell (using it as the current directory).
Damn lol wish I would have come out sooner lmao Seems a tad more complicated though, I don't remember having to compile anything at least
Check out dattobd sometime. It's a tool that allows you to make a snapshot of a block device for backups, while the OS is running. Useful if your filesystem doesn't support snapshots.
I’ve been killing myself in my free time to get a NanoPi R6S to boot from an SD card; wish I had this expertise.
Edit: I didn’t say this very clearly, I’ve been killing myself to build and compile U-Boot, the Linux kernel then building an image to write to an SD card that’ll actually boot.
Heh, this expertise is built on hundreds of "Why tf won't X boot from Y!!" and solid research skills lol you'll get there
Oops, I realize now that my comment made it seem like I can’t figure out how to write an image to external media and boot it properly. It was actually more intense than that, so I’ve updated it.
Not sure about your hardware, but try to look up if it requires a special kernel, like the RPi 5 (which only runs on raspbian because they ship that kernel)
The manufacturer ships several tools but a lot of the necessary files are shared from Google Drive, and each time I try to download one it says “download limit exceeded.” My goal is to build something more vanilla — I can see that the hardware is supported in the mainline kernel so I’m doing this with tons of trial and error (which I don’t mind).
I’m doing everything in a Dockerfile so once I’m there the goal is to clean it up and push it to GitHub.
i had experimented with kexec and takeover.sh to install a distro that was not available by my provider.
it resulted in some scripts i now (triggered by this thread) have published (in a nonready state):
http://github.com/tobinq/goaround
the scripts may be in bad shape but i successfully changed one preinstalled ubuntu to a devuan with what is in these scripts. however i didn't work on them for month now and am not sure about that last state.. so its experimental only.