Thomas Lamprecht [Fri, 19 Apr 2024 12:20:16 +0000 (14:20 +0200)]
auto installer: rework sys-info struct layout and add more details
move the decoded DMI info into a "dmi" object to avoid crowding the
outer layer, add more info from the ISO and Product currently booted
and use the complete mocked variants for the assistant environments so
that users evaluating this have a full picture of possible keys.
Also include the NIC link name with the mach address so that one can
also configure the management interface correctly from a MAC.
Do this all in a slightly more ergonomic rust way, using separate
structs and impls for getting the info.
There could be even more details added, but that's for a future patch,
this one mostly transforms and enhances existing keys.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Fri, 19 Apr 2024 12:17:52 +0000 (14:17 +0200)]
common: add mocked variants for setup and ISO related info structs
and add necessary derives for debug and serialize so that we can use
this for the auto-installer HTTP payload that gets send to the client
to be able to determine a dynamic answer file.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Fri, 19 Apr 2024 10:59:13 +0000 (12:59 +0200)]
auto installer: make fetch print answer to stdout and explicitly run installer
Instead of executing the actual installer from inside a tool named
`fetch-answer` make it print the answer file content it queried to
stdout and use the outer "unconfigured" init process to redirect this
to a file inside run, which is then piped to the actual installer in a
separate command.
This makes the execution flow a bit easier to grasp when reading
unconfigured and allows to inspect the answer file in a debug session.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This new subcommand makes it possible to prepare an ISO to use it for an
automated installation.
It is possible to control the behavior of the resulting automated ISO
with optional parameters.
If no target file is specified, the new ISO will be named with suffixes
to indicate it as automated and additional information. This should help
to distinct between the different options that were chosen to create it.
The code for parsing an answer file is moved to its own function.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This patch switches the behavior to use the settings that can be
specified in the ISO.
This means, that it is possible to control how the answer file should be
fetched:
* auto - as usually, go through the options until one works (partition,
http)
* included - the answer file is included in the ISO
* partition - only check for an answer file in a partition called
'proxmoxinst' in lower or uppercase
* http - only fetch the answer file via an HTTP POST request.
Additionally it is possible to specify the HTTP URL directly in the ISO.
Placing the SSL fingerprint on a partition is not possible anymore. If
one wants to provide it right away (besides DHCP or DNS), it must be
incluced in the ISO itself. This reduced the need for another USB flash
drive.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
it is meant as a helper utility to prepare an installation for chroot
and clean up afterwards
It tries to determine the used FS from the previous installation, will
do what is necessary to mount/import the root FS to /target. It then
will set up all bind mounts.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
common: skip target_hd when deserializing InstallConfig
as only the 'path' property is serialized -> deserialization is
problematic. The information would be present in the 'run-env-info-json',
but for now there is no need for it in any code that deserializes the
low-level config. Therefore we are currently skipping it on
deserialization
If we need it in the future, we need to think about how to handle the
deserialization.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
[ TL: fix dependencies feature/version and indentation ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
d/control: explicitly depend on rust crate features and minimum version
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
auto installer: factor out fetch-answer and autoinst-helper
Putting proxmox-fetch-answer into it's own crate, will keep the use of
OpenSSL localized to where we need it. Otherwise building other binaries
will always depend on OpenSSL as well, even without actually needing it.
Having a dedicated crate for the proxmox-autoinst-helper should make it
easier to build it independently to have it available outside of the
install environment.
The fetch plugins have been moved to the proxmox-fetch-answer crate,
except for the 'get_nic_list' function and 'sysinfo.rs'. Since both are
also needed by the proxmox-autoinst-helper, they are kept in the
proxmox-auto-installer crate.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
auto-installer: fetch: add http plugin to fetch answer
This plugin will send a HTTP POST request with identifying sysinfo to
fetch an answer file. The provided sysinfo can be used to identify the
system and generate a matching answer file on demand.
The URL to send the request to, can be defined in two ways. Via a custom
DHCP option or a TXT record on a predefined subdomain, relative to the
search domain received via DHCP.
Additionally it is possible to specify a SHA256 SSL fingerprint. This
can be useful if a self-signed certificate is used or the URL is using
an IP address instead of an FQDN. Even with a trusted cert, it can be
used to pin this specific certificate.
The certificate fingerprint can either be placed on the `proxmoxinst`
partition and needs to be called `cert_fingerprint.txt`, or it can be
provided in a second custom DHCP option or a TXT record.
If no fingerprint is provided, we switch rustls to native-certs and
native-tls.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
auto-installer: fetch: add gathering of system identifiers and restructure code
They will be used as payload when POSTing a request for an answer file. The
idea is, that with this information, it should be possible to identify
the system and generate a matching answer file on the fly.
Many of these properties can also be found on the machine or packaging
of the machine and could therefore be scanned into a database.
Identifiers are the following properties from `dmidecode` sections 1, 2,
and 3:
* Asset Tag
* Product Name
* Serial Number
* SKU Number
* UUID
As well as a list of the MAC addresses of all the NICs and the product
type: pve, pmg, pbs.
Since we now have more than a simple utils.rs module in the fetch
plugins, it, and the additional fetch plugin utilities are placed in
their own directory.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It can parse an answer file to check against syntax errors, test match
filters against the current hardware and list properties of the current
hardware to match against.
Since this tool should be able to run outside of the installer
environment, it does not rely on the device information provided by the
low-level installer. It instead fetches the list of disks and NICs by
itself.
The rules when a device is ignored, should match how the low-level
installer handles it.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The new auto-installer is intended for unattended installations and
should be especially helpful for users with frequent and/or big
installations.
The main idea is that a answer file is provided by some mechanism,
e.g., on the ISO itself, on a separate (USB pen drive) partition, or
the network, which holds a somewhat generic config in form of a TOML
file that also supports wildcard glob'ing for selecting, e.g., target
disks for flexibility.
This is a initial implementation that will be improved and extended
over the next commits.
The installation config is currently generated through a dedicated
function (parse_answer) instead of a From implementation. This is
because for now the source data is spread over several other structs
in comparison to one in the TUI installer.
Logging is done to both stdout and a dedicated log file.
Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
[ TL: Squash initial "build-up" patches ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Fetches UDEV device properties prepended with 'E:' for NICs and disks.
The result is stored in its own JSON file.
This information is needed to filter for specific devices. Mainly for
the auto-installer for now.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
[ TL: rework Udev module to avoid external dependencies, keep those
in the Sys modules ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
tui: common: move InstallConfig struct to common crate
It describes the data structure expected by the low-level-installer.
We do this so we can use it in more than the TUI installer, for example
the planned auto installer.
Make the members public so we can easily implement a custom From method
for each dependent crate.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
low-level: write final installation config to /tmp
This helps to know how the system was set up in steps after the
installation. For example in debug mode or when using post commands in
the automatic/unattended installation.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com> Tested-by: Christoph Heiss <c.heiss@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Christoph Heiss [Tue, 26 Mar 2024 13:29:19 +0000 (14:29 +0100)]
run env: use default error message if country detection failed with empty string
Bit of perl fun again.
$err from detect_country_tracing_to() can be empty string under certain
circumstances (according to a forum post [0]). The // operator
evaluates an empty as true, thus `warn` receives an empty string to and
just prints
Warning: something wrong at /usr/share/perl5/proxmox/Install/RunEnv.pm line 305
Which isn't particular helpful. Use the || operator instead, that
evaluates an empty string as false and thus would fall back to the
generic error message.
A minimal reproducer/example for completeness sake:
#!/usr/bin/env perl
use strict;
use warnings;
warn ('' // "unable to detect country\n");
warn ('' || "unable to detect country\n");
gives
Warning: something's wrong at ./test.pl line 5.
unable to detect country
Christoph Heiss [Tue, 12 Mar 2024 11:59:12 +0000 (12:59 +0100)]
unconfigured: move terminal size setting before starting debug shell
Otherwise, when using the serial debug shell, the console size will be
0x0. This in turn breaks the TUI installer, as it cannot detect the
size properly.
It also adjust the size to the proper 80x24 instead of 80x25, as
advertised in the log message.
As while for VGA 80x25 is the standard size [0], for serial consoles
aka. VT10x emulated terminals it is actually 80x24 [1], which
basically everything uses as reference. GRUB also uses 80x24 when used
over a serial interface. It spits out 24 lines at least.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
[ TL: include further references about size from discussion ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Support for setting this option originally got introduced into the
installer in 2016 by
c7779156 ("refactor disk setup, add advanced ZFS options") [1]
where the default of 'off' was still correct.
As the installer only set the property if it was *not* explicitly set
to 'on', this actually regressed in the meantime.
Thus just remove the conditional all together, as the definedness-check
did not have any impact anyway (since $value gets set to 'on'
regardless) and the latter just causes regressions like this one.
Tested by installing once w/o the patch to confirm the report and once
with the patch applied, checking `zfs get compression` on the freshly
installed system.
Thomas Lamprecht [Mon, 26 Feb 2024 19:40:14 +0000 (20:40 +0100)]
sys/command: double wait frequency and send SIGKILL once after 0.5s
100 ms is quite plenty, while we would be better of using a event
based wait, i.e., dropping the WNOHANG, that would also mean handling
the time out via alarm, EINTR checking and quite a bit other stuff
making this more convoluted, so for now just go faster..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Christoph Heiss [Mon, 26 Feb 2024 16:50:43 +0000 (17:50 +0100)]
sys: command: wait for process exit with sub-second granularity
Using full seconds as a granularity for sleeping between waitpid()'s is
way too much and unnecessarily slows down the installation a lot. Most
processes take a few moments after closing their stdin/stdout to
actually exit fully, which means that we would sleep a second in most
cases.
Lower it to 0.1 second, which immensely improves the situation.
Some values for comparison; tui-installer on the same bog-standard
2-core, SeaBIOS, ext4, virtio VM (roughly averaged over multiple runs):
* 8.0 ISO (baseline): ~2:30 min
* w/o patch: ~9:00 min
* w/ patch: ~2:30 min
Values measured are from pressing the 'Install' button until the
autoreboot dialog (aka. install finished) popped up.
Fixes: 152bbef ("sys: command: factor out kill() + waitpid() from run_command()") Reported-by: Stoiko Ivanov <s.ivanov@proxmox.com> Reported-by: Filip Schauer <f.schauer@proxmox.com> Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Thomas Lamprecht [Mon, 26 Feb 2024 13:38:11 +0000 (14:38 +0100)]
run command: use explicit return undef in closures on call sites
To avoid a misinterpretation of the auto-return value:
> In the absence of an explicit return, a subroutine, eval, or do FILE
> automatically returns the value of the last expression evaluated.
-- https://perldoc.perl.org/functions/return
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Mon, 26 Feb 2024 13:34:09 +0000 (14:34 +0100)]
run command: avoid using 1 as special value
In Perl, the last expression of a block (e.g. of a method, eval) gets
returned if there's no explicit return statement. Quite often that is
truthy, i.e., 1.
As that was chosen as the special value for the CMD_FINISHED flag it
had quite a few false positives, causing weird effects and
installation failure.
Reserve that overly problematic value and chose 2 as new CMD_FINISHED
value, albeit it could be better to signal this even more explicitly,
like with a structured hash reference, but for now this is a good stop
gap.
Fixes: 23c5fbe ("sys: command: allow terminating the process early from log subroutine") Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sat, 24 Feb 2024 17:17:10 +0000 (18:17 +0100)]
tests: code-style and error handling fixes for ui2stdio
use modern calling style, avoid duplicate use of Test::More module,
handle fork error more visible, handle pipe creation errors and do
that all in one commit as it's just a test and I don't care.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Sat, 24 Feb 2024 16:56:26 +0000 (17:56 +0100)]
stdio connected UI: drop perl prototype definitions
The prototypes where completely circumvented by calling those two
methods by reference via &, and that probably happened as the send_msg
one was just wrong, it forced scalar context for the second parameter,
while that was a list (or well hash, but the difference there can be
blurry).
Anyhow, prototypes are not always of help, and can be a PITA with
side-effects too, and especially for such small modules it has not
that much use to declare them for privately-scoped methods, so just
drop them and fix the calling style.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht [Fri, 23 Feb 2024 16:19:15 +0000 (17:19 +0100)]
fqdn comparison: expand test scope
Add some negative tests to ensure a `return true` (exaggerated)
refactoring won't pass the suite, and add one test where a and b is
the same, just to be sure.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Christoph Heiss [Thu, 15 Feb 2024 12:39:38 +0000 (13:39 +0100)]
fix #5230: sys: net: properly escape FQDN regex
Due to interpolation, the \. sequence must be double-escaped.
Previously, this would result in a non-escaped dot, thus matching much
more liberally than it should.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
[ TL: fix bug # reference in code comments ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Christoph Heiss [Tue, 13 Feb 2024 15:14:03 +0000 (16:14 +0100)]
fix #4872: run env: use run_command() for country detection
This fixes a rather longstanding issue [0][1] with the country
detection, in that it might get completely stuck and thus hangs the
installation.
This is due how Perl, signals and line reading interacts.
A minimal reproducer, how the installer currently works, looks like
this:
```
#!/usr/bin/env perl
use strict;
use warnings;
open (my $fh, '-|', 'sleep', '1000') or die;
my $prev = alarm(2);
eval {
local $SIG{ALRM} = sub { die "timed out!\n" };
my $line;
while (defined ($line = <$fh>)) {
print "line: $line";
}
};
alarm($prev);
close($fh);
```
One might expect that this times out after 2 seconds, as specified in
`alarm(2)`. The thruth is that `$line = <$fh>` apparently prevents the
signal to go through. This then causes the installer to hang there
indefinitely, if `traceroute` never progresses - which seems to happen
on lots of (weird) networks, as evidently can be seen in the forum [1].
Proxmox::Sys::Command::run_command() handles of these weird cases, takes
care of the nitty-gritty details and - most importantly - interacts
properly with SIGALRM, so just use that instead.
This _should_ really fix that issue, but reproducing it 1:1 as part of
the installation process is _very_ hard, basically pure luck. But
rewriting the reproducer using run_command (in the exact same way that
this patch rewrites detect_country_tracing_to()) fixes the issue there,
so it's the best we can probably do.
NB: This causes that the traceroute command is now printed to the log
(as run_command() logs that by default), which we could also hide e.g.
through another parameter if wanted.
Christoph Heiss [Tue, 13 Feb 2024 15:14:01 +0000 (16:14 +0100)]
sys: command: allow terminating the process early from log subroutine
If the logging subroutine $func returns CMD_FINISHED after processing a
line, the running subprocess is killed early.
This mechanism can be used when e.g. only a certain part of the output
of a (long-running) command is needed, avoiding the extra time it would
take the command to finish properly.
This is done in a entirely backwards-compatible way, i.e. existing
usages don't need any modification.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Christoph Heiss [Tue, 13 Feb 2024 15:14:00 +0000 (16:14 +0100)]
sys: command: handle EINTR in run_command()
Previously, the I/O loop would continue endlessly until the subprocess
exited.
This explicit handling allows run_command() to be used with e.g.
alarm().
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
Christoph Heiss [Tue, 13 Feb 2024 15:13:59 +0000 (16:13 +0100)]
sys: command: factor out kill() + waitpid() from run_command()
This moves the kill() + waitpid() combo into a separate subroutine,
avoiding open-coding that sequence. wait_for_process() also handles
properly unkillable process (e.g. in D-state) and avoids completely
locking up the installer in such cases. See [0].
For the latter case, a timeout exists (with a default of 5 seconds) in
which to wait for the process to exit after sending an optional
TERM/KILL signal.
Also while at it, add a few basic tests for run_command().
Christoph Heiss [Tue, 13 Feb 2024 15:13:58 +0000 (16:13 +0100)]
low-level: initialize UI backend for 'dump-env' subcommand too
Some detection routines might try to log things and call some
Proxmox::Ui functions all the way down, so just initialize it with the
stdio backend to avoid errors.
Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
For consistency sake, all colons and trailing spaces in labels that were
followed with an entry were removed, this matches other panels such as
the password and country/timezone panels.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Tested-by: Christoph Heiss <c.heiss@proxmox.com> Tested-by: Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Using boxes causes the labels to not align correctly in certain
circumstances. In the following commits we replace the use of boxes with
grids and set the margins and spacing directly on the respective grid.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Tested-by: Christoph Heiss <c.heiss@proxmox.com> Tested-by: Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Previously the grids were inserted in a succession of boxes each with
its own set of margins and spacing. We define the margins now
exclusively in the grid and account for previous values.
Note that we match the top and bottom margins of the 'Target Harddisk'
panel which does not need to use a grid.
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com> Tested-by: Christoph Heiss <c.heiss@proxmox.com> Tested-by: Tested-by: Lukas Wagner <l.wagner@proxmox.com>
Christoph Heiss [Tue, 6 Feb 2024 13:12:26 +0000 (14:12 +0100)]
proxinstall, common: remove "off" as zfs checksum option
See also the thread at [0] for the initial discussion/idea.
Disabling checksums is considered an "extraordinarily bad idea" [1] (for
pretty obvious reason) and nobody should really ever use it.
Thus remove the option completely; just so that users cannot simply
disable checksum "for performance reasons" without knowing about the
implications of this.
As pointed out by Thomas, it can still be set to "off" after the
installation using the `zfs` tool, if really wanted.
Christoph Heiss [Fri, 22 Dec 2023 10:52:24 +0000 (11:52 +0100)]
proxinstall, common: remove deprecated fletcher2 as zfs checksum algorithm
Fletcher-2 has long been deprecated and should not be used anymore
[0][1], so we probably should not offer it anymore too. It's been
deprecated since at least over 3 years, beyond that it's hard to find
an exact date.
Stoiko Ivanov [Tue, 21 Nov 2023 13:11:18 +0000 (14:11 +0100)]
zfs: create dataset var-lib-vz for /var/lib/vz
Creating rpool/var/lib/vz and all intermediate datasets causes a
service-failure of `var.mount` upon shutdown.
creating the dataset for /var/lib/vz directly at the rpool and setting
its mountpoint property seems the most robust way to address this.
The alternative approach of setting `canmount=off` on the `var`
dataset seems a bit dangerous (users setting a zfs property and
suddenly hiding their /var contents).
The only small downside to this approach is that the setting of the
mountpoint happens quite a bit after extracting the data - but this
would probably be better addressed with a refactoring of the
lowlevel-installer code (setting the zfs-pool up under /target and
getting rid of a few special cases)
Stoiko Ivanov [Tue, 21 Nov 2023 11:09:59 +0000 (12:09 +0100)]
serial installer: add serial config for grub to target system
Matching if a serial will be needed for grub is based on the target
commandline - the speed is also read from there. The unit is based
on the ttyS device - although I'd assume that this might not always
match up.
projects / pve-installer.git / log