Yang Zhong [Sat, 29 Jan 2022 17:36:46 +0000 (09:36 -0800)]
x86/fpu/xstate: Fix the ARCH_REQ_XCOMP_PERM implementation
BugLink: https://bugs.launchpad.net/bugs/1967750
ARCH_REQ_XCOMP_PERM is supposed to add the requested feature to the
permission bitmap of thread_group_leader()->fpu. But the code overwrites
the bitmap with the requested feature bit only rather than adding it.
Fix the code to add the requested feature bit to the master bitmask.
Fixes: db8268df0983 ("x86/arch_prctl: Add controls for dynamic XSTATE components") Signed-off-by: Yang Zhong <yang.zhong@intel.com> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Cc: Paolo Bonzini <bonzini@gnu.org> Cc: stable@vger.kernel.org Link: https://lore.kernel.org/r/20220129173647.27981-2-chang.seok.bae@intel.com
(backported from commit 063452fd94d153d4eb38ad58f210f3d37a09cca4) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Jim Mattson [Thu, 3 Feb 2022 19:43:07 +0000 (11:43 -0800)]
x86/cpufeatures: Put the AMX macros in the word 18 block
BugLink: https://bugs.launchpad.net/bugs/1967750
These macros are for bits in CPUID.(EAX=7,ECX=0):EDX, not for bits in
CPUID(EAX=7,ECX=1):EAX. Put them with their brethren.
[ bp: Sort word 18 bits properly, as caught by Like Xu
<like.xu.linux@gmail.com> ]
Support for large, "dynamic" fpstates was recently merged. This
included code to ensure that sigaltstacks are sufficiently sized for
these large states. A new lock was added to remove races between
enabling large features and setting up sigaltstacks.
== Problem ==
The new lock (sigaltstack_lock()) is acquired in the sigreturn path
before restoring the old sigaltstack. Unfortunately, contention on the
new lock causes a measurable signal handling performance regression [1].
However, the common case is that no *changes* are made to the
sigaltstack state at sigreturn.
== Solution ==
do_sigaltstack() acquires sigaltstack_lock() and is used for both
sys_sigaltstack() and restoring the sigaltstack in sys_sigreturn().
Check for changes to the sigaltstack before taking the lock. If no
changes were made, return before acquiring the lock.
This removes lock contention from the common-case sigreturn path.
Marco Elver [Fri, 26 Nov 2021 12:47:46 +0000 (13:47 +0100)]
x86/fpu/signal: Initialize sw_bytes in save_xstate_epilog()
BugLink: https://bugs.launchpad.net/bugs/1967750
save_sw_bytes() did not fully initialize sw_bytes, which caused KMSAN
to report an infoleak (see below).
Initialize sw_bytes explicitly to avoid this.
Local variable sw_bytes created at:
save_xstate_epilog+0x80/0x510 arch/x86/kernel/fpu/signal.c:121
copy_fpstate_to_sigframe+0x861/0xb60 arch/x86/kernel/fpu/signal.c:245
Bytes 20-47 of 48 are uninitialized
Memory access of size 48 starts at ffff8880801d3a18
Data copied to user address 00007ffd90e2ef50
=====================================================
Chang S. Bae [Tue, 26 Oct 2021 09:11:57 +0000 (02:11 -0700)]
Documentation/x86: Add documentation for using dynamic XSTATE features
BugLink: https://bugs.launchpad.net/bugs/1967750
Explain how dynamic XSTATE features can be enabled via the
architecture-specific prctl() along with dynamic sigframe size and
first use trap handling.
Fix:
Documentation/x86/xstate.rst:15: WARNING: Title underline too short.
as reported by Stephen Rothwell <sfr@canb.auug.org.au>
Chang S. Bae [Tue, 26 Oct 2021 12:25:25 +0000 (05:25 -0700)]
selftests/x86/amx: Add context switch test
BugLink: https://bugs.launchpad.net/bugs/1967750
XSAVE state is thread-local. The kernel switches between thread
state at context switch time. Generally, running a selftest for
a while will naturally expose it to some context switching and
and will test the XSAVE code.
Instead of just hoping that the tests get context-switched at
random times, force context-switches on purpose. Spawn off a few
userspace threads and force context-switches between them.
Ensure that the kernel correctly context switches each thread's
unique AMX state.
Chang S. Bae [Tue, 26 Oct 2021 12:25:24 +0000 (05:25 -0700)]
selftests/x86/amx: Add test cases for AMX state management
BugLink: https://bugs.launchpad.net/bugs/1967750
AMX TILEDATA is a very large XSAVE feature. It could have caused
nasty XSAVE buffer space waste in two places:
* Signal stacks
* Kernel task_struct->fpu buffers
To avoid this waste, neither of these buffers have AMX state by
default. The non-default features are called "dynamic" features.
There is an arch_prctl(ARCH_REQ_XCOMP_PERM) which allows a task
to declare that it wants to use AMX or other "dynamic" XSAVE
features. This arch_prctl() ensures that sufficient sigaltstack
space is available before it will succeed. It also expands the
task_struct buffer.
Functions of this test:
* Test arch_prctl(ARCH_REQ_XCOMP_PERM). Ensure that it checks for
proper sigaltstack sizing and that the sizing is enforced for
future sigaltstack calls.
* Ensure that ARCH_REQ_XCOMP_PERM is inherited across fork()
* Ensure that TILEDATA use before the prctl() is fatal
* Ensure that TILEDATA is cleared across fork()
Note: Generally, compiler support is needed to do something with
AMX. Instead, directly load AMX state from userspace with a
plain XSAVE. Do not depend on the compiler.
Chang S. Bae [Thu, 21 Oct 2021 22:55:27 +0000 (15:55 -0700)]
x86/fpu/amx: Enable the AMX feature in 64-bit mode
BugLink: https://bugs.launchpad.net/bugs/1967750
Add the AMX state components in XFEATURE_MASK_USER_SUPPORTED and the
TILE_DATA component to the dynamic states and update the permission check
table accordingly.
This is only effective on 64 bit kernels as for 32bit kernels
XFEATURE_MASK_TILE is defined as 0.
TILE_DATA is caller-saved state and the only dynamic state. Add build time
sanity check to ensure the assumption that every dynamic feature is caller-
saved.
Make AMX state depend on XFD as it is dynamic feature.
Chang S. Bae [Thu, 21 Oct 2021 22:55:26 +0000 (15:55 -0700)]
x86/fpu: Add XFD handling for dynamic states
BugLink: https://bugs.launchpad.net/bugs/1967750
To handle the dynamic sizing of buffers on first use the XFD MSR has to be
armed. Store the delta between the maximum available and the default
feature bits in init_fpstate where it can be retrieved for task creation.
If the delta is non zero then dynamic features are enabled. This needs also
to enable the static key which guards the XFD updates. This is delayed to
an initcall because the FPU setup runs before jump labels are initialized.
Chang S. Bae [Thu, 21 Oct 2021 22:55:25 +0000 (15:55 -0700)]
x86/fpu: Calculate the default sizes independently
BugLink: https://bugs.launchpad.net/bugs/1967750
When dynamically enabled states are supported the maximum and default sizes
for the kernel buffers and user space interfaces are not longer identical.
Put the necessary calculations in place which only take the default enabled
features into account.
Chang S. Bae [Thu, 21 Oct 2021 22:55:24 +0000 (15:55 -0700)]
x86/fpu/amx: Define AMX state components and have it used for boot-time checks
BugLink: https://bugs.launchpad.net/bugs/1967750
The XSTATE initialization uses check_xstate_against_struct() to sanity
check the size of XSTATE-enabled features. AMX is a XSAVE-enabled feature,
and its size is not hard-coded but discoverable at run-time via CPUID.
The AMX state is composed of state components 17 and 18, which are all user
state components. The first component is the XTILECFG state of a 64-byte
tile-related control register. The state component 18, called XTILEDATA,
contains the actual tile data, and the state size varies on
implementations. The architectural maximum, as defined in the CPUID(0x1d,
1): EAX[15:0], is a byte less than 64KB. The first implementation supports
8KB.
Check the XTILEDATA state size dynamically. The feature introduces the new
tile register, TMM. Define one register struct only and read the number of
registers from CPUID. Cross-check the overall size with CPUID again.
Chang S. Bae [Thu, 21 Oct 2021 22:55:23 +0000 (15:55 -0700)]
x86/fpu/xstate: Prepare XSAVE feature table for gaps in state component numbers
BugLink: https://bugs.launchpad.net/bugs/1967750
The kernel checks at boot time which features are available by walking a
XSAVE feature table which contains the CPUID feature bit numbers which need
to be checked whether a feature is available on a CPU or not. So far the
feature numbers have been linear, but AMX will create a gap which the
current code cannot handle.
Make the table entries explicitly indexed and adjust the loop code
accordingly to prepare for that.
No functional change.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Len Brown <len.brown@intel.com> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lore.kernel.org/r/20211021225527.10184-20-chang.seok.bae@intel.com
(cherry picked from commit 70c3f1671b0cbc386b387f1de33b7837e276a195) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Chang S. Bae [Thu, 21 Oct 2021 22:55:22 +0000 (15:55 -0700)]
x86/fpu/xstate: Add fpstate_realloc()/free()
BugLink: https://bugs.launchpad.net/bugs/1967750
The fpstate embedded in struct fpu is the default state for storing the FPU
registers. It's sized so that the default supported features can be stored.
For dynamically enabled features the register buffer is too small.
The #NM handler detects first use of a feature which is disabled in the
XFD MSR. After handling permission checks it recalculates the size for
kernel space and user space state and invokes fpstate_realloc() which
tries to reallocate fpstate and install it.
Provide the allocator function which checks whether the current buffer size
is sufficient and if not allocates one. If allocation is successful the new
fpstate is initialized with the new features and sizes and the now enabled
features is removed from the task's XFD mask.
realloc_fpstate() uses vzalloc(). If use of this mechanism grows to
re-allocate buffers larger than 64KB, a more sophisticated allocation
scheme that includes purpose-built reclaim capability might be justified.
Chang S. Bae [Thu, 21 Oct 2021 22:55:21 +0000 (15:55 -0700)]
x86/fpu/xstate: Add XFD #NM handler
BugLink: https://bugs.launchpad.net/bugs/1967750
If the XFD MSR has feature bits set then #NM will be raised when user space
attempts to use an instruction related to one of these features.
When the task has no permissions to use that feature, raise SIGILL, which
is the same behavior as #UD.
If the task has permissions, calculate the new buffer size for the extended
feature set and allocate a larger fpstate. In the unlikely case that
vzalloc() fails, SIGSEGV is raised.
The allocation function will be added in the next step. Provide a stub
which fails for now.
Chang S. Bae [Thu, 21 Oct 2021 22:55:20 +0000 (15:55 -0700)]
x86/fpu: Update XFD state where required
BugLink: https://bugs.launchpad.net/bugs/1967750
The IA32_XFD_MSR allows to arm #NM traps for XSTATE components which are
enabled in XCR0. The register has to be restored before the tasks XSTATE is
restored. The life time rules are the same as for FPU state.
XFD is updated on return to userspace only when the FPU state of the task
is not up to date in the registers. It's updated before the XRSTORS so
that eventually enabled dynamic features are restored as well and not
brought into init state.
Also in signal handling for restoring FPU state from user space the
correctness of the XFD state has to be ensured.
Chang S. Bae [Thu, 21 Oct 2021 22:55:18 +0000 (15:55 -0700)]
x86/fpu: Add XFD state to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Add storage for XFD register state to struct fpstate. This will be used to
store the XFD MSR state. This will be used for switching the XFD MSR when
FPU content is restored.
Add a per-CPU variable to cache the current MSR value so the MSR has only
to be written when the values are different.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lkml.kernel.org/r/20211021225527.10184-15-chang.seok.bae@intel.com
(cherry picked from commit 8bf26758ca9659866b844dd51037314b4c0fa6bd) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Chang S. Bae [Thu, 21 Oct 2021 22:55:16 +0000 (15:55 -0700)]
x86/cpufeatures: Add eXtended Feature Disabling (XFD) feature bit
BugLink: https://bugs.launchpad.net/bugs/1967750
Intel's eXtended Feature Disable (XFD) feature is an extension of the XSAVE
architecture. XFD allows the kernel to enable a feature state in XCR0 and
to receive a #NM trap when a task uses instructions accessing that state.
This is going to be used to postpone the allocation of a larger XSTATE
buffer for a task to the point where it is actually using a related
instruction after the permission to use that facility has been granted.
XFD is not used by the kernel, but only applied to userspace. This is a
matter of policy as the kernel knows how a fpstate is reallocated and the
XFD state.
The compacted XSAVE format is adjustable for dynamic features. Make XFD
depend on XSAVES.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lkml.kernel.org/r/20211021225527.10184-13-chang.seok.bae@intel.com
(cherry picked from commit c351101678ce54492b6e09810ec02efc0df036a9) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Chang S. Bae [Thu, 21 Oct 2021 22:55:15 +0000 (15:55 -0700)]
x86/fpu: Reset permission and fpstate on exec()
BugLink: https://bugs.launchpad.net/bugs/1967750
On exec(), extended register states saved in the buffer is cleared. With
dynamic features, each task carries variables besides the register states.
The struct fpu has permission information and struct fpstate contains
buffer size and feature masks. They are all dynamically updated with
dynamic features.
Reset the current task's entire FPU data before an exec() so that the new
task starts with default permission and fpstate.
Rename the register state reset function because the old naming confuses as
it does not reset struct fpstate.
Thomas Gleixner [Thu, 21 Oct 2021 22:55:14 +0000 (15:55 -0700)]
x86/fpu: Prepare fpu_clone() for dynamically enabled features
BugLink: https://bugs.launchpad.net/bugs/1967750
The default portion of the parent's FPU state is saved in a child task.
With dynamic features enabled, the non-default portion is not saved in a
child's fpstate because these register states are defined to be
caller-saved. The new task's fpstate is therefore the default buffer.
Fork inherits the permission of the parent.
Also, do not use memcpy() when TIF_NEED_FPU_LOAD is set because it is
invalid when the parent has dynamic features.
Chang S. Bae [Thu, 21 Oct 2021 22:55:13 +0000 (15:55 -0700)]
x86/fpu/signal: Prepare for variable sigframe length
BugLink: https://bugs.launchpad.net/bugs/1967750
The software reserved portion of the fxsave frame in the signal frame
is copied from structures which have been set up at boot time. With
dynamically enabled features the content of these structures is no
longer correct because the xfeatures and size can be different per task.
Calculate the software reserved portion at runtime and fill in the
xfeatures and size values from the tasks active fpstate.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lkml.kernel.org/r/20211021225527.10184-10-chang.seok.bae@intel.com
(cherry picked from commit 53599b4d54b9b8dda1d537a558946869d2acbddc) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Thomas Gleixner [Thu, 21 Oct 2021 22:55:12 +0000 (15:55 -0700)]
x86/signal: Use fpu::__state_user_size for sigalt stack validation
BugLink: https://bugs.launchpad.net/bugs/1967750
Use the current->group_leader->fpu to check for pending permissions to use
extended features and validate against the resulting user space size which
is stored in the group leaders fpu struct as well.
This prevents a task from installing a too small sized sigaltstack after
permissions to use dynamically enabled features have been granted, but
the task has not (yet) used a related instruction.
Chang S. Bae [Thu, 21 Oct 2021 22:55:10 +0000 (15:55 -0700)]
x86/arch_prctl: Add controls for dynamic XSTATE components
BugLink: https://bugs.launchpad.net/bugs/1967750
Dynamically enabled XSTATE features are by default disabled for all
processes. A process has to request permission to use such a feature.
To support this implement a architecture specific prctl() with the options:
- ARCH_GET_XCOMP_SUPP
Copies the supported feature bitmap into the user space provided
u64 storage. The pointer is handed in via arg2
- ARCH_GET_XCOMP_PERM
Copies the process wide permitted feature bitmap into the user space
provided u64 storage. The pointer is handed in via arg2
- ARCH_REQ_XCOMP_PERM
Request permission for a feature set. A feature set can be mapped to a
facility, e.g. AMX, and can require one or more XSTATE components to
be enabled.
The feature argument is the number of the highest XSTATE component
which is required for a facility to work.
The request argument is not a user supplied bitmap because that makes
filtering harder (think seccomp) and even impossible because to
support 32bit tasks the argument would have to be a pointer.
The permission mechanism works this way:
Task asks for permission for a facility and kernel checks whether that's
supported. If supported it does:
1) Check whether permission has already been granted
2) Compute the size of the required kernel and user space buffer
(sigframe) size.
3) Validate that no task has a sigaltstack installed
which is smaller than the resulting sigframe size
4) Add the requested feature bit(s) to the permission bitmap of
current->group_leader->fpu and store the sizes in the group
leaders fpu struct as well.
If that is successful then the feature is still not enabled for any of the
tasks. The first usage of a related instruction will result in a #NM
trap. The trap handler validates the permission bit of the tasks group
leader and if permitted it installs a larger kernel buffer and transfers
the permission and size info to the new fpstate container which makes all
the FPU functions which require per task information aware of the extended
feature set.
[ tglx: Adopted to new base code, added missing serialization,
massaged namings, comments and changelog ]
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lkml.kernel.org/r/20211021225527.10184-7-chang.seok.bae@intel.com
(cherry picked from commit db8268df0983adc2bb1fb48c9e5f7bfbb5f617f3) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Thomas Gleixner [Thu, 21 Oct 2021 22:55:09 +0000 (15:55 -0700)]
x86/fpu: Add fpu_state_config::legacy_features
BugLink: https://bugs.launchpad.net/bugs/1967750
The upcoming prctl() which is required to request the permission for a
dynamically enabled feature will also provide an option to retrieve the
supported features. If the CPU does not support XSAVE, the supported
features would be 0 even when the CPU supports FP and SSE.
Provide separate storage for the legacy feature set to avoid that and fill
in the bits in the legacy init function.
Thomas Gleixner [Thu, 21 Oct 2021 22:55:08 +0000 (15:55 -0700)]
x86/fpu: Add members to struct fpu to cache permission information
BugLink: https://bugs.launchpad.net/bugs/1967750
Dynamically enabled features can be requested by any thread of a running
process at any time. The request does neither enable the feature nor
allocate larger buffers. It just stores the permission to use the feature
by adding the features to the permission bitmap and by calculating the
required sizes for kernel and user space.
The reallocation of the kernel buffer happens when the feature is used
for the first time which is caught by an exception. The permission
bitmap is then checked and if the feature is permitted, then it becomes
fully enabled. If not, the task dies similarly to a task which uses an
undefined instruction.
The size information is precomputed to allow proper sigaltstack size checks
once the feature is permitted, but not yet in use because otherwise this
would open race windows where too small stacks could be installed causing
a later fail on signal delivery.
Initialize them to the default feature set and sizes.
Chang S. Bae [Thu, 21 Oct 2021 22:55:07 +0000 (15:55 -0700)]
x86/fpu/xstate: Provide xstate_calculate_size()
BugLink: https://bugs.launchpad.net/bugs/1967750
Split out the size calculation from the paranoia check so it can be used
for recalculating buffer sizes when dynamically enabled features are
supported.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
[ tglx: Adopted to changed base code ] Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lkml.kernel.org/r/20211021225527.10184-4-chang.seok.bae@intel.com
(cherry picked from commit 84e4dccc8fce20b497388d756e12de5c9006eb48) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Add a mechanism to enforce strict checking of the sigaltstack size against
the real size of the FPU frame.
The strict check can be enabled via a config option and can also be
controlled via the kernel command line option 'strict_sas_size' independent
of the config switch.
Enabling it might break existing applications which allocate a too small
sigaltstack but 'work' because they never get a signal delivered. Though it
can be handy to filter out binaries which are not yet aware of
AT_MINSIGSTKSZ.
Also the upcoming support for dynamically enabled FPU features requires a
strict sanity check to ensure that:
- Enabling of a dynamic feature, which changes the sigframe size fits
into an enabled sigaltstack
- Installing a too small sigaltstack after a dynamic feature has been
added is not possible.
Implement the base check which is controlled by config and command line
options.
Thomas Gleixner [Thu, 21 Oct 2021 22:55:05 +0000 (15:55 -0700)]
signal: Add an optional check for altstack size
BugLink: https://bugs.launchpad.net/bugs/1967750
New x86 FPU features will be very large, requiring ~10k of stack in
signal handlers. These new features require a new approach called
"dynamic features".
The kernel currently tries to ensure that altstacks are reasonably
sized. Right now, on x86, sys_sigaltstack() requires a size of >=2k.
However, that 2k is a constant. Simply raising that 2k requirement
to >10k for the new features would break existing apps which have a
compiled-in size of 2k.
Instead of universally enforcing a larger stack, prohibit a process from
using dynamic features without properly-sized altstacks. This must be
enforced in two places:
* A dynamic feature can not be enabled without an large-enough altstack
for each process thread.
* Once a dynamic feature is enabled, any request to install a too-small
altstack will be rejected
The dynamic feature enabling code must examine each thread in a
process to ensure that the altstacks are large enough. Add a new lock
(sigaltstack_lock()) to ensure that threads can not race and change
their altstack after being examined.
Add the infrastructure in form of a config option and provide empty
stubs for architectures which do not need dynamic altstack size checks.
This implementation will be fleshed out for x86 in a future patch called
x86/arch_prctl: Add controls for dynamic XSTATE components
Thomas Gleixner [Fri, 22 Oct 2021 18:55:53 +0000 (20:55 +0200)]
x86/kvm: Convert FPU handling to a single swap buffer
BugLink: https://bugs.launchpad.net/bugs/1967750
For the upcoming AMX support it's necessary to do a proper integration with
KVM. Currently KVM allocates two FPU structs which are used for saving the user
state of the vCPU thread and restoring the guest state when entering
vcpu_run() and doing the reverse operation before leaving vcpu_run().
With the new fpstate mechanism this can be reduced to one extra buffer by
swapping the fpstate pointer in current::thread::fpu. This makes the
upcoming support for AMX and XFD simpler because then fpstate information
(features, sizes, xfd) are always consistent and it does not require any
nasty workarounds.
Thomas Gleixner [Fri, 22 Oct 2021 18:55:51 +0000 (20:55 +0200)]
x86/fpu: Provide infrastructure for KVM FPU cleanup
BugLink: https://bugs.launchpad.net/bugs/1967750
For the upcoming AMX support it's necessary to do a proper integration with
KVM. Currently KVM allocates two FPU structs which are used for saving the user
state of the vCPU thread and restoring the guest state when entering
vcpu_run() and doing the reverse operation before leaving vcpu_run().
With the new fpstate mechanism this can be reduced to one extra buffer by
swapping the fpstate pointer in current::thread::fpu. This makes the
upcoming support for AMX and XFD simpler because then fpstate information
(features, sizes, xfd) are always consistent and it does not require any
nasty workarounds.
Provide:
- An allocator which initializes the state properly
- A replacement for the existing FPU swap mechanim
Aside of the reduced memory footprint, this also makes state switching
more efficient when TIF_FPU_NEED_LOAD is set. It does not require a
memcpy as the state is already correct in the to be swapped out fpstate.
The existing interfaces will be removed once KVM is converted over.
Thomas Gleixner [Fri, 22 Oct 2021 18:55:49 +0000 (20:55 +0200)]
x86/fpu: Prepare for sanitizing KVM FPU code
BugLink: https://bugs.launchpad.net/bugs/1967750
For the upcoming AMX support it's necessary to do a proper integration with
KVM. To avoid more nasty hackery in KVM which violate encapsulation extend
struct fpu and fpstate so the fpstate switching can be consolidated and
simplified.
Currently KVM allocates two FPU structs which are used for saving the user
state of the vCPU thread and restoring the guest state when entering
vcpu_run() and doing the reverse operation before leaving vcpu_run().
With the new fpstate mechanism this can be reduced to one extra buffer by
swapping the fpstate pointer in current::thread::fpu. This makes the
upcoming support for AMX and XFD simpler because then fpstate information
(features, sizes, xfd) are always consistent and it does not require any
nasty workarounds.
Add fpu::__task_fpstate to save the regular fpstate pointer while the task
is inside vcpu_run(). Add some state fields to fpstate to indicate the
nature of the state.
Thomas Gleixner [Thu, 14 Oct 2021 23:09:40 +0000 (01:09 +0200)]
x86/fpu/xstate: Move remaining xfeature helpers to core
BugLink: https://bugs.launchpad.net/bugs/1967750
Now that everything is mopped up, move all the helpers and prototypes into
the core header. They are not required by the outside.
Rework restore_regs_from_fpstate() so it takes a constant mask which will
then be applied against the maximum feature set so that the restore
operation brings all features which are not in the xsave buffer xfeature
bitmap into init state.
This ensures that if the previous task used a dynamically enabled feature
that the task which restores has all unused components properly initialized.
Cleanup the last user of xfeatures_mask_fpstate() as well and remove it.
Thomas Gleixner [Thu, 14 Oct 2021 23:09:37 +0000 (01:09 +0200)]
x86/fpu: Mop up xfeatures_mask_uabi()
BugLink: https://bugs.launchpad.net/bugs/1967750
Use the new fpu_user_cfg to retrieve the information instead of
xfeatures_mask_uabi() which will be no longer correct when dynamically
enabled features become available.
Using fpu_user_cfg is appropriate when setting XCOMP_BV in the
init_fpstate since it has space allocated for "max_features". But,
normal fpstates might only have space for default xfeatures. Since
XRSTOR* derives the format of the XSAVE buffer from XCOMP_BV, this can
lead to XRSTOR reading out of bounds.
So when copying actively used fpstate, simply read the XCOMP_BV features
bits directly out of the fpstate instead.
This correction courtesy of Dave Hansen <dave.hansen@linux.intel.com>
Thomas Gleixner [Thu, 14 Oct 2021 23:09:35 +0000 (01:09 +0200)]
x86/fpu: Move xstate feature masks to fpu_*_cfg
BugLink: https://bugs.launchpad.net/bugs/1967750
Move the feature mask storage to the kernel and user config
structs. Default and maximum feature set are the same for now.
Thomas Gleixner [Thu, 14 Oct 2021 23:09:34 +0000 (01:09 +0200)]
x86/fpu: Move xstate size to fpu_*_cfg
BugLink: https://bugs.launchpad.net/bugs/1967750
Use the new kernel and user space config storage to store and retrieve the
XSTATE buffer sizes. The default and the maximum size are the same for now,
but will change when support for dynamically enabled features is added.
Thomas Gleixner [Thu, 14 Oct 2021 23:09:29 +0000 (01:09 +0200)]
x86/fpu: Provide struct fpu_config
BugLink: https://bugs.launchpad.net/bugs/1967750
Provide a struct to store information about the maximum supported and the
default feature set and buffer sizes for both user and kernel space.
This allows quick retrieval of this information for the upcoming support
for dynamically enabled features.
[ bp: Add vertical spacing between the struct members. ]
Thomas Gleixner [Thu, 14 Oct 2021 22:51:51 +0000 (00:51 +0200)]
x86/fpu/signal: Use fpstate for size and features
BugLink: https://bugs.launchpad.net/bugs/1967750
For dynamically enabled features it's required to get the features which
are enabled for that context when restoring from sigframe.
The same applies for all signal frame size calculations.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:55 +0000 (16:55 +0200)]
x86/fpu/xstate: Use fpstate for copy_uabi_to_xstate()
BugLink: https://bugs.launchpad.net/bugs/1967750
Prepare for dynamically enabled states per task. The function needs to
retrieve the features and sizes which are valid in a fpstate
context. Retrieve them from fpstate.
Move the function declarations to the core header as they are not
required anywhere else.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:54 +0000 (16:55 +0200)]
x86/fpu: Use fpstate in __copy_xstate_to_uabi_buf()
BugLink: https://bugs.launchpad.net/bugs/1967750
With dynamically enabled features the copy function must know the features
and the size which is valid for the task. Retrieve them from fpstate.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:51 +0000 (16:55 +0200)]
x86/fpu/xstate: Use fpstate for xsave_to_user_sigframe()
BugLink: https://bugs.launchpad.net/bugs/1967750
With dynamically enabled features the sigframe code must know the features
which are enabled for the task. Get them from fpstate.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:49 +0000 (16:55 +0200)]
x86/fpu/xstate: Use fpstate for os_xsave()
BugLink: https://bugs.launchpad.net/bugs/1967750
With variable feature sets XSAVE[S] requires to know the feature set for
which the buffer is valid. Retrieve it from fpstate.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:48 +0000 (16:55 +0200)]
x86/fpu: Use fpstate::size
BugLink: https://bugs.launchpad.net/bugs/1967750
Make use of fpstate::size in various places which require the buffer size
information for sanity checks or memcpy() sizing.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:46 +0000 (16:55 +0200)]
x86/fpu: Add size and mask information to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Add state size and feature mask information to the fpstate container. This
will be used for runtime checks with the upcoming support for dynamically
enabled features and dynamically sized buffers. That avoids conditionals
all over the place as the required information is accessible for both
default and extended buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:45 +0000 (16:55 +0200)]
x86/process: Move arch_thread_struct_whitelist() out of line
BugLink: https://bugs.launchpad.net/bugs/1967750
In preparation for dynamically enabled FPU features move the function
out of line as the goal is to expose less and not more information.
Not a problem right now, but later when dynamically allocated buffers
are available, keeping the pointer might result in freeing the
parent's buffer. Set it to NULL which prevents that. If fork reaches
clone_thread(), the pointer will be correctly set to the new task
context.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:40 +0000 (16:55 +0200)]
x86/math-emu: Convert to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert math emulation code to the new register storage
mechanism in preparation for dynamically sized buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:39 +0000 (16:55 +0200)]
x86/fpu/core: Convert to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert the rest of the core code to the new register storage mechanism in
preparation for dynamically sized buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:37 +0000 (16:55 +0200)]
x86/fpu/signal: Convert to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert signal related code to the new register storage mechanism in
preparation for dynamically sized buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:36 +0000 (16:55 +0200)]
x86/fpu/regset: Convert to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert regset related code to the new register storage mechanism in
preparation for dynamically sized buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:34 +0000 (16:55 +0200)]
x86/fpu: Convert tracing to fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert FPU tracing code to the new register storage mechanism in
preparation for dynamically sized buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:31 +0000 (16:55 +0200)]
x86/fpu: Replace KVMs xstate component clearing
BugLink: https://bugs.launchpad.net/bugs/1967750
In order to prepare for the support of dynamically enabled FPU features,
move the clearing of xstate components to the FPU core code.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:30 +0000 (16:55 +0200)]
x86/fpu: Convert restore_fpregs_from_fpstate() to struct fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert restore_fpregs_from_fpstate() and related code to the new
register storage mechanism in preparation for dynamically sized buffers.
Thomas Gleixner [Wed, 13 Oct 2021 14:55:28 +0000 (16:55 +0200)]
x86/fpu: Convert fpstate_init() to struct fpstate
BugLink: https://bugs.launchpad.net/bugs/1967750
Convert fpstate_init() and related code to the new register storage
mechanism in preparation for dynamically sized buffers.
The kernel will provide the default sized buffer for storing the regular
features up to AVX512 in thread_struct::fpu and if a task requests to use
one of the new features then the register storage has to be extended.
The state will be accessed via a pointer in thread_struct::fpu which
defaults to the builtin storage and can be switched when extended storage
is required.
To avoid conditionals all over the code, create a new container for the
register storage which will gain other information, e.g. size, feature
masks etc., later. For now it just contains the register storage, which
gives it exactly the same layout as the exiting fpu::state.
Stick fpu::state and the new fpu::__fpstate into an anonymous union and
initialize the pointer. Add build time checks to validate that both are
at the same place and have the same size.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:17 +0000 (03:16 +0200)]
x86/fpu: Replace KVMs home brewed FPU copy to user
BugLink: https://bugs.launchpad.net/bugs/1967750
Similar to the copy from user function the FPU core has this already
implemented with all bells and whistles.
Get rid of the duplicated code and use the core functionality.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:41 +0000 (03:16 +0200)]
x86/fpu: Provide a proper function for ex_handler_fprestore()
BugLink: https://bugs.launchpad.net/bugs/1967750
To make upcoming changes for support of dynamically enabled features
simpler, provide a proper function for the exception handler which removes
exposure of FPU internals.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:39 +0000 (03:16 +0200)]
x86/fpu: Replace the includes of fpu/internal.h
BugLink: https://bugs.launchpad.net/bugs/1967750
Now that the file is empty, fixup all references with the proper includes
and delete the former kitchen sink.
Include asm/fpu/xstate.h to fix a missing update_regset_xstate_info()
prototype, which is Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Borislav Petkov <bp@suse.de> Link: https://lkml.kernel.org/r/20211015011539.844565975@linutronix.de
(cherry picked from commit 0ae67cc34f765078a63137120e4567ad2f050b75) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Thomas Gleixner [Fri, 15 Oct 2021 01:16:15 +0000 (03:16 +0200)]
x86/fpu: Rework copy_xstate_to_uabi_buf()
BugLink: https://bugs.launchpad.net/bugs/1967750
Prepare for replacing the KVM copy xstate to user function by extending
copy_xstate_to_uabi_buf() with a pkru argument which allows the caller to
hand in the pkru value, which is required for KVM because the guest PKRU is
not accessible via current. Fixup all callsites accordingly.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:13 +0000 (03:16 +0200)]
x86/fpu: Replace KVMs home brewed FPU copy from user
BugLink: https://bugs.launchpad.net/bugs/1967750
Copying a user space buffer to the memory buffer is already available in
the FPU core. The copy mechanism in KVM lacks sanity checks and needs to
use cpuid() to lookup the offset of each component, while the FPU core has
this information cached.
Make the FPU core variant accessible for KVM and replace the home brewed
mechanism.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:12 +0000 (03:16 +0200)]
x86/fpu: Move KVMs FPU swapping to FPU core
BugLink: https://bugs.launchpad.net/bugs/1967750
Swapping the host/guest FPU is directly fiddling with FPU internals which
requires 5 exports. The upcoming support of dynamically enabled states
would even need more.
Implement a swap function in the FPU core code and export that instead.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Borislav Petkov <bp@suse.de> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Cc: kvm@vger.kernel.org Link: https://lkml.kernel.org/r/20211015011539.076072399@linutronix.de
(cherry picked from commit a0ff0611c2fbde94f6c9db8351939b08f2cb6797) Acked-by: Dimitri John Ledkov <dimitri.ledkov@canonical.com> Signed-off-by: Andrea Righi <andrea.righi@canonical.com>
Thomas Gleixner [Fri, 15 Oct 2021 01:16:09 +0000 (03:16 +0200)]
x86/fpu/xstate: Provide and use for_each_xfeature()
BugLink: https://bugs.launchpad.net/bugs/1967750
These loops evaluating xfeature bits are really hard to read. Create an
iterator and use for_each_set_bit_from() inside which already does the right
thing.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:06 +0000 (03:16 +0200)]
x86/fpu: Do not inherit FPU context for kernel and IO worker threads
BugLink: https://bugs.launchpad.net/bugs/1967750
There is no reason why kernel and IO worker threads need a full clone of
the parent's FPU state. Both are kernel threads which are not supposed to
use FPU. So copying a large state or doing XSAVE() is pointless. Just clean
out the minimally required state for those tasks.
Thomas Gleixner [Fri, 15 Oct 2021 01:16:04 +0000 (03:16 +0200)]
x86/process: Clone FPU in copy_thread()
BugLink: https://bugs.launchpad.net/bugs/1967750
There is no reason to clone FPU in arch_dup_task_struct(). Quite the
contrary - it prevents optimizations. Move it to copy_thread().
Thomas Gleixner [Fri, 15 Oct 2021 01:16:02 +0000 (03:16 +0200)]
x86/fpu: Remove pointless memset in fpu_clone()
BugLink: https://bugs.launchpad.net/bugs/1967750
Zeroing the forked task's FPU registers buffer to avoid leaking init
optimized stale data into the clone is a pointless exercise for the case
where the current task has TIF_NEED_FPU_LOAD set. In that case, the FPU
registers state is copied from current's FPU register buffer which can
contain stale init optimized data as well.
The alledged information leak is non-existant because this stale init
optimized data is used nowhere and cannot leak anywhere.
Thomas Gleixner [Fri, 15 Oct 2021 01:15:59 +0000 (03:15 +0200)]
x86/fpu: Restrict xsaves()/xrstors() to independent states
BugLink: https://bugs.launchpad.net/bugs/1967750
These interfaces are really only valid for features which are independently
managed and not part of the task context state for various reasons.
Tighten the checks and adjust the misleading comments.
Thomas Gleixner [Fri, 15 Oct 2021 01:15:57 +0000 (03:15 +0200)]
x86/pkru: Remove useless include
BugLink: https://bugs.launchpad.net/bugs/1967750
PKRU code does not need anything from FPU headers. Include cpufeature.h
instead and fixup the resulting fallout in perf.
This is a preparation for FPU changes in order to prevent recursive include
hell.
Thomas Gleixner [Fri, 15 Oct 2021 01:15:56 +0000 (03:15 +0200)]
x86/fpu: Update stale comments
BugLink: https://bugs.launchpad.net/bugs/1967750
copy_fpstate_to_sigframe() does not have a slow path anymore. Neither does
the !ia32 restore in __fpu_restore_sig().
Thomas Gleixner [Wed, 8 Sep 2021 13:29:38 +0000 (15:29 +0200)]
x86/fpu/signal: Change return type of __fpu_restore_sig() to boolean
BugLink: https://bugs.launchpad.net/bugs/1967750
Now that fpu__restore_sig() returns a boolean get rid of the individual
error codes in __fpu_restore_sig() as well.
Thomas Gleixner [Wed, 8 Sep 2021 13:29:37 +0000 (15:29 +0200)]
x86/fpu/signal: Change return type of fpu__restore_sig() to boolean
BugLink: https://bugs.launchpad.net/bugs/1967750
None of the call sites cares about the error code. All they need to know is
whether the function succeeded or not.
Thomas Gleixner [Wed, 8 Sep 2021 13:29:34 +0000 (15:29 +0200)]
x86/fpu/signal: Change return type of copy_fpregs_to_sigframe() helpers to boolean
BugLink: https://bugs.launchpad.net/bugs/1967750
Now that copy_fpregs_to_sigframe() returns boolean the individual return
codes in the related helper functions do not make sense anymore. Change
them to return boolean success/fail.
projects / mirror_ubuntu-jammy-kernel.git / log