If the userspace hasn't been prepared to ignore the unreliable "opened"
events and the unreliable initial state notification, Linux users can use
the following kernel parameters to handle the possible issues:
-A. button.lid_init_state=open:
+A. button.lid_init_state=method:
+ When this option is specified, the ACPI button driver reports the
+ initial lid state using the returning value of the _LID control method
+ and whether the "opened"/"closed" events are paired fully relies on the
+ firmware implementation.
+ This option can be used to fix some platforms where the returning value
+ of the _LID control method is reliable but the initial lid state
+ notification is missing.
+ This option is the default behavior during the period the userspace
+ isn't ready to handle the buggy AML tables.
+B. button.lid_init_state=open:
When this option is specified, the ACPI button driver always reports the
initial lid state as "opened" and whether the "opened"/"closed" events
are paired fully relies on the firmware implementation.
This may fix some platforms where the returning value of the _LID
control method is not reliable and the initial lid state notification is
missing.
- This option is the default behavior during the period the userspace
- isn't ready to handle the buggy AML tables.
If the userspace has been prepared to ignore the unreliable "opened" events
and the unreliable initial state notification, Linux users should always
use the following kernel parameter:
-B. button.lid_init_state=ignore:
+C. button.lid_init_state=ignore:
When this option is specified, the ACPI button driver never reports the
initial lid state and there is a compensation mechanism implemented to
ensure that the reliable "closed" notifications can always be delievered
.. |struct cpufreq_policy| replace:: :c:type:`struct cpufreq_policy <cpufreq_policy>`
+.. |intel_pstate| replace:: :doc:`intel_pstate <intel_pstate>`
=======================
CPU Performance Scaling
interface it comes from and may not be easily represented in an abstract,
platform-independent way. For this reason, ``CPUFreq`` allows scaling drivers
to bypass the governor layer and implement their own performance scaling
-algorithms. That is done by the ``intel_pstate`` scaling driver.
+algorithms. That is done by the |intel_pstate| scaling driver.
``CPUFreq`` Policy Objects
into account. That is achieved by invoking the governor's ``->stop`` and
``->start()`` callbacks, in this order, for the entire policy.
-As mentioned before, the ``intel_pstate`` scaling driver bypasses the scaling
+As mentioned before, the |intel_pstate| scaling driver bypasses the scaling
governor layer of ``CPUFreq`` and provides its own P-state selection algorithms.
-Consequently, if ``intel_pstate`` is used, scaling governors are not attached to
+Consequently, if |intel_pstate| is used, scaling governors are not attached to
new policy objects. Instead, the driver's ``->setpolicy()`` callback is invoked
to register per-CPU utilization update callbacks for each policy. These
callbacks are invoked by the CPU scheduler in the same way as for scaling
-governors, but in the ``intel_pstate`` case they both determine the P-state to
+governors, but in the |intel_pstate| case they both determine the P-state to
use and change the hardware configuration accordingly in one go from scheduler
context.
``scaling_available_governors``
List of ``CPUFreq`` scaling governors present in the kernel that can
- be attached to this policy or (if the ``intel_pstate`` scaling driver is
+ be attached to this policy or (if the |intel_pstate| scaling driver is
in use) list of scaling algorithms provided by the driver that can be
applied to this policy.
the CPU is actually running at (due to hardware design and other
limitations).
- Some scaling drivers (e.g. ``intel_pstate``) attempt to provide
+ Some scaling drivers (e.g. |intel_pstate|) attempt to provide
information more precisely reflecting the current CPU frequency through
this attribute, but that still may not be the exact current CPU
frequency as seen by the hardware at the moment.
``scaling_governor``
The scaling governor currently attached to this policy or (if the
- ``intel_pstate`` scaling driver is in use) the scaling algorithm
+ |intel_pstate| scaling driver is in use) the scaling algorithm
provided by the driver that is currently applied to this policy.
This attribute is read-write and writing to it will cause a new scaling
governor to be attached to this policy or a new scaling algorithm
provided by the scaling driver to be applied to it (in the
- ``intel_pstate`` case), as indicated by the string written to this
+ |intel_pstate| case), as indicated by the string written to this
attribute (which must be one of the names listed by the
``scaling_available_governors`` attribute described above).
the "boost" setting for the whole system. It is not present if the underlying
scaling driver does not support the frequency boost mechanism (or supports it,
but provides a driver-specific interface for controlling it, like
-``intel_pstate``).
+|intel_pstate|).
If the value in this file is 1, the frequency boost mechanism is enabled. This
means that either the hardware can be put into states in which it is able to
:maxdepth: 2
cpufreq
+ intel_pstate
.. only:: subproject and html
--- /dev/null
+===============================================
+``intel_pstate`` CPU Performance Scaling Driver
+===============================================
+
+::
+
+ Copyright (c) 2017 Intel Corp., Rafael J. Wysocki <rafael.j.wysocki@intel.com>
+
+
+General Information
+===================
+
+``intel_pstate`` is a part of the
+:doc:`CPU performance scaling subsystem <cpufreq>` in the Linux kernel
+(``CPUFreq``). It is a scaling driver for the Sandy Bridge and later
+generations of Intel processors. Note, however, that some of those processors
+may not be supported. [To understand ``intel_pstate`` it is necessary to know
+how ``CPUFreq`` works in general, so this is the time to read :doc:`cpufreq` if
+you have not done that yet.]
+
+For the processors supported by ``intel_pstate``, the P-state concept is broader
+than just an operating frequency or an operating performance point (see the
+`LinuxCon Europe 2015 presentation by Kristen Accardi <LCEU2015_>`_ for more
+information about that). For this reason, the representation of P-states used
+by ``intel_pstate`` internally follows the hardware specification (for details
+refer to `Intel® 64 and IA-32 Architectures Software Developer’s Manual
+Volume 3: System Programming Guide <SDM_>`_). However, the ``CPUFreq`` core
+uses frequencies for identifying operating performance points of CPUs and
+frequencies are involved in the user space interface exposed by it, so
+``intel_pstate`` maps its internal representation of P-states to frequencies too
+(fortunately, that mapping is unambiguous). At the same time, it would not be
+practical for ``intel_pstate`` to supply the ``CPUFreq`` core with a table of
+available frequencies due to the possible size of it, so the driver does not do
+that. Some functionality of the core is limited by that.
+
+Since the hardware P-state selection interface used by ``intel_pstate`` is
+available at the logical CPU level, the driver always works with individual
+CPUs. Consequently, if ``intel_pstate`` is in use, every ``CPUFreq`` policy
+object corresponds to one logical CPU and ``CPUFreq`` policies are effectively
+equivalent to CPUs. In particular, this means that they become "inactive" every
+time the corresponding CPU is taken offline and need to be re-initialized when
+it goes back online.
+
+``intel_pstate`` is not modular, so it cannot be unloaded, which means that the
+only way to pass early-configuration-time parameters to it is via the kernel
+command line. However, its configuration can be adjusted via ``sysfs`` to a
+great extent. In some configurations it even is possible to unregister it via
+``sysfs`` which allows another ``CPUFreq`` scaling driver to be loaded and
+registered (see `below <status_attr_>`_).
+
+
+Operation Modes
+===============
+
+``intel_pstate`` can operate in three different modes: in the active mode with
+or without hardware-managed P-states support and in the passive mode. Which of
+them will be in effect depends on what kernel command line options are used and
+on the capabilities of the processor.
+
+Active Mode
+-----------
+
+This is the default operation mode of ``intel_pstate``. If it works in this
+mode, the ``scaling_driver`` policy attribute in ``sysfs`` for all ``CPUFreq``
+policies contains the string "intel_pstate".
+
+In this mode the driver bypasses the scaling governors layer of ``CPUFreq`` and
+provides its own scaling algorithms for P-state selection. Those algorithms
+can be applied to ``CPUFreq`` policies in the same way as generic scaling
+governors (that is, through the ``scaling_governor`` policy attribute in
+``sysfs``). [Note that different P-state selection algorithms may be chosen for
+different policies, but that is not recommended.]
+
+They are not generic scaling governors, but their names are the same as the
+names of some of those governors. Moreover, confusingly enough, they generally
+do not work in the same way as the generic governors they share the names with.
+For example, the ``powersave`` P-state selection algorithm provided by
+``intel_pstate`` is not a counterpart of the generic ``powersave`` governor
+(roughly, it corresponds to the ``schedutil`` and ``ondemand`` governors).
+
+There are two P-state selection algorithms provided by ``intel_pstate`` in the
+active mode: ``powersave`` and ``performance``. The way they both operate
+depends on whether or not the hardware-managed P-states (HWP) feature has been
+enabled in the processor and possibly on the processor model.
+
+Which of the P-state selection algorithms is used by default depends on the
+:c:macro:`CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE` kernel configuration option.
+Namely, if that option is set, the ``performance`` algorithm will be used by
+default, and the other one will be used by default if it is not set.
+
+Active Mode With HWP
+~~~~~~~~~~~~~~~~~~~~
+
+If the processor supports the HWP feature, it will be enabled during the
+processor initialization and cannot be disabled after that. It is possible
+to avoid enabling it by passing the ``intel_pstate=no_hwp`` argument to the
+kernel in the command line.
+
+If the HWP feature has been enabled, ``intel_pstate`` relies on the processor to
+select P-states by itself, but still it can give hints to the processor's
+internal P-state selection logic. What those hints are depends on which P-state
+selection algorithm has been applied to the given policy (or to the CPU it
+corresponds to).
+
+Even though the P-state selection is carried out by the processor automatically,
+``intel_pstate`` registers utilization update callbacks with the CPU scheduler
+in this mode. However, they are not used for running a P-state selection
+algorithm, but for periodic updates of the current CPU frequency information to
+be made available from the ``scaling_cur_freq`` policy attribute in ``sysfs``.
+
+HWP + ``performance``
+.....................
+
+In this configuration ``intel_pstate`` will write 0 to the processor's
+Energy-Performance Preference (EPP) knob (if supported) or its
+Energy-Performance Bias (EPB) knob (otherwise), which means that the processor's
+internal P-state selection logic is expected to focus entirely on performance.
+
+This will override the EPP/EPB setting coming from the ``sysfs`` interface
+(see `Energy vs Performance Hints`_ below).
+
+Also, in this configuration the range of P-states available to the processor's
+internal P-state selection logic is always restricted to the upper boundary
+(that is, the maximum P-state that the driver is allowed to use).
+
+HWP + ``powersave``
+...................
+
+In this configuration ``intel_pstate`` will set the processor's
+Energy-Performance Preference (EPP) knob (if supported) or its
+Energy-Performance Bias (EPB) knob (otherwise) to whatever value it was
+previously set to via ``sysfs`` (or whatever default value it was
+set to by the platform firmware). This usually causes the processor's
+internal P-state selection logic to be less performance-focused.
+
+Active Mode Without HWP
+~~~~~~~~~~~~~~~~~~~~~~~
+
+This is the default operation mode for processors that do not support the HWP
+feature. It also is used by default with the ``intel_pstate=no_hwp`` argument
+in the kernel command line. However, in this mode ``intel_pstate`` may refuse
+to work with the given processor if it does not recognize it. [Note that
+``intel_pstate`` will never refuse to work with any processor with the HWP
+feature enabled.]
+
+In this mode ``intel_pstate`` registers utilization update callbacks with the
+CPU scheduler in order to run a P-state selection algorithm, either
+``powersave`` or ``performance``, depending on the ``scaling_cur_freq`` policy
+setting in ``sysfs``. The current CPU frequency information to be made
+available from the ``scaling_cur_freq`` policy attribute in ``sysfs`` is
+periodically updated by those utilization update callbacks too.
+
+``performance``
+...............
+
+Without HWP, this P-state selection algorithm is always the same regardless of
+the processor model and platform configuration.
+
+It selects the maximum P-state it is allowed to use, subject to limits set via
+``sysfs``, every time the P-state selection computations are carried out by the
+driver's utilization update callback for the given CPU (that does not happen
+more often than every 10 ms), but the hardware configuration will not be changed
+if the new P-state is the same as the current one.
+
+This is the default P-state selection algorithm if the
+:c:macro:`CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE` kernel configuration option
+is set.
+
+``powersave``
+.............
+
+Without HWP, this P-state selection algorithm generally depends on the
+processor model and/or the system profile setting in the ACPI tables and there
+are two variants of it.
+
+One of them is used with processors from the Atom line and (regardless of the
+processor model) on platforms with the system profile in the ACPI tables set to
+"mobile" (laptops mostly), "tablet", "appliance PC", "desktop", or
+"workstation". It is also used with processors supporting the HWP feature if
+that feature has not been enabled (that is, with the ``intel_pstate=no_hwp``
+argument in the kernel command line). It is similar to the algorithm
+implemented by the generic ``schedutil`` scaling governor except that the
+utilization metric used by it is based on numbers coming from feedback
+registers of the CPU. It generally selects P-states proportional to the
+current CPU utilization, so it is referred to as the "proportional" algorithm.
+
+The second variant of the ``powersave`` P-state selection algorithm, used in all
+of the other cases (generally, on processors from the Core line, so it is
+referred to as the "Core" algorithm), is based on the values read from the APERF
+and MPERF feedback registers and the previously requested target P-state.
+It does not really take CPU utilization into account explicitly, but as a rule
+it causes the CPU P-state to ramp up very quickly in response to increased
+utilization which is generally desirable in server environments.
+
+Regardless of the variant, this algorithm is run by the driver's utilization
+update callback for the given CPU when it is invoked by the CPU scheduler, but
+not more often than every 10 ms (that can be tweaked via ``debugfs`` in `this
+particular case <Tuning Interface in debugfs_>`_). Like in the ``performance``
+case, the hardware configuration is not touched if the new P-state turns out to
+be the same as the current one.
+
+This is the default P-state selection algorithm if the
+:c:macro:`CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE` kernel configuration option
+is not set.
+
+Passive Mode
+------------
+
+This mode is used if the ``intel_pstate=passive`` argument is passed to the
+kernel in the command line (it implies the ``intel_pstate=no_hwp`` setting too).
+Like in the active mode without HWP support, in this mode ``intel_pstate`` may
+refuse to work with the given processor if it does not recognize it.
+
+If the driver works in this mode, the ``scaling_driver`` policy attribute in
+``sysfs`` for all ``CPUFreq`` policies contains the string "intel_cpufreq".
+Then, the driver behaves like a regular ``CPUFreq`` scaling driver. That is,
+it is invoked by generic scaling governors when necessary to talk to the
+hardware in order to change the P-state of a CPU (in particular, the
+``schedutil`` governor can invoke it directly from scheduler context).
+
+While in this mode, ``intel_pstate`` can be used with all of the (generic)
+scaling governors listed by the ``scaling_available_governors`` policy attribute
+in ``sysfs`` (and the P-state selection algorithms described above are not
+used). Then, it is responsible for the configuration of policy objects
+corresponding to CPUs and provides the ``CPUFreq`` core (and the scaling
+governors attached to the policy objects) with accurate information on the
+maximum and minimum operating frequencies supported by the hardware (including
+the so-called "turbo" frequency ranges). In other words, in the passive mode
+the entire range of available P-states is exposed by ``intel_pstate`` to the
+``CPUFreq`` core. However, in this mode the driver does not register
+utilization update callbacks with the CPU scheduler and the ``scaling_cur_freq``
+information comes from the ``CPUFreq`` core (and is the last frequency selected
+by the current scaling governor for the given policy).
+
+
+.. _turbo:
+
+Turbo P-states Support
+======================
+
+In the majority of cases, the entire range of P-states available to
+``intel_pstate`` can be divided into two sub-ranges that correspond to
+different types of processor behavior, above and below a boundary that
+will be referred to as the "turbo threshold" in what follows.
+
+The P-states above the turbo threshold are referred to as "turbo P-states" and
+the whole sub-range of P-states they belong to is referred to as the "turbo
+range". These names are related to the Turbo Boost technology allowing a
+multicore processor to opportunistically increase the P-state of one or more
+cores if there is enough power to do that and if that is not going to cause the
+thermal envelope of the processor package to be exceeded.
+
+Specifically, if software sets the P-state of a CPU core within the turbo range
+(that is, above the turbo threshold), the processor is permitted to take over
+performance scaling control for that core and put it into turbo P-states of its
+choice going forward. However, that permission is interpreted differently by
+different processor generations. Namely, the Sandy Bridge generation of
+processors will never use any P-states above the last one set by software for
+the given core, even if it is within the turbo range, whereas all of the later
+processor generations will take it as a license to use any P-states from the
+turbo range, even above the one set by software. In other words, on those
+processors setting any P-state from the turbo range will enable the processor
+to put the given core into all turbo P-states up to and including the maximum
+supported one as it sees fit.
+
+One important property of turbo P-states is that they are not sustainable. More
+precisely, there is no guarantee that any CPUs will be able to stay in any of
+those states indefinitely, because the power distribution within the processor
+package may change over time or the thermal envelope it was designed for might
+be exceeded if a turbo P-state was used for too long.
+
+In turn, the P-states below the turbo threshold generally are sustainable. In
+fact, if one of them is set by software, the processor is not expected to change
+it to a lower one unless in a thermal stress or a power limit violation
+situation (a higher P-state may still be used if it is set for another CPU in
+the same package at the same time, for example).
+
+Some processors allow multiple cores to be in turbo P-states at the same time,
+but the maximum P-state that can be set for them generally depends on the number
+of cores running concurrently. The maximum turbo P-state that can be set for 3
+cores at the same time usually is lower than the analogous maximum P-state for
+2 cores, which in turn usually is lower than the maximum turbo P-state that can
+be set for 1 core. The one-core maximum turbo P-state is thus the maximum
+supported one overall.
+
+The maximum supported turbo P-state, the turbo threshold (the maximum supported
+non-turbo P-state) and the minimum supported P-state are specific to the
+processor model and can be determined by reading the processor's model-specific
+registers (MSRs). Moreover, some processors support the Configurable TDP
+(Thermal Design Power) feature and, when that feature is enabled, the turbo
+threshold effectively becomes a configurable value that can be set by the
+platform firmware.
+
+Unlike ``_PSS`` objects in the ACPI tables, ``intel_pstate`` always exposes
+the entire range of available P-states, including the whole turbo range, to the
+``CPUFreq`` core and (in the passive mode) to generic scaling governors. This
+generally causes turbo P-states to be set more often when ``intel_pstate`` is
+used relative to ACPI-based CPU performance scaling (see `below <acpi-cpufreq_>`_
+for more information).
+
+Moreover, since ``intel_pstate`` always knows what the real turbo threshold is
+(even if the Configurable TDP feature is enabled in the processor), its
+``no_turbo`` attribute in ``sysfs`` (described `below <no_turbo_attr_>`_) should
+work as expected in all cases (that is, if set to disable turbo P-states, it
+always should prevent ``intel_pstate`` from using them).
+
+
+Processor Support
+=================
+
+To handle a given processor ``intel_pstate`` requires a number of different
+pieces of information on it to be known, including:
+
+ * The minimum supported P-state.
+
+ * The maximum supported `non-turbo P-state <turbo_>`_.
+
+ * Whether or not turbo P-states are supported at all.
+
+ * The maximum supported `one-core turbo P-state <turbo_>`_ (if turbo P-states
+ are supported).
+
+ * The scaling formula to translate the driver's internal representation
+ of P-states into frequencies and the other way around.
+
+Generally, ways to obtain that information are specific to the processor model
+or family. Although it often is possible to obtain all of it from the processor
+itself (using model-specific registers), there are cases in which hardware
+manuals need to be consulted to get to it too.
+
+For this reason, there is a list of supported processors in ``intel_pstate`` and
+the driver initialization will fail if the detected processor is not in that
+list, unless it supports the `HWP feature <Active Mode_>`_. [The interface to
+obtain all of the information listed above is the same for all of the processors
+supporting the HWP feature, which is why they all are supported by
+``intel_pstate``.]
+
+
+User Space Interface in ``sysfs``
+=================================
+
+Global Attributes
+-----------------
+
+``intel_pstate`` exposes several global attributes (files) in ``sysfs`` to
+control its functionality at the system level. They are located in the
+``/sys/devices/system/cpu/cpufreq/intel_pstate/`` directory and affect all
+CPUs.
+
+Some of them are not present if the ``intel_pstate=per_cpu_perf_limits``
+argument is passed to the kernel in the command line.
+
+``max_perf_pct``
+ Maximum P-state the driver is allowed to set in percent of the
+ maximum supported performance level (the highest supported `turbo
+ P-state <turbo_>`_).
+
+ This attribute will not be exposed if the
+ ``intel_pstate=per_cpu_perf_limits`` argument is present in the kernel
+ command line.
+
+``min_perf_pct``
+ Minimum P-state the driver is allowed to set in percent of the
+ maximum supported performance level (the highest supported `turbo
+ P-state <turbo_>`_).
+
+ This attribute will not be exposed if the
+ ``intel_pstate=per_cpu_perf_limits`` argument is present in the kernel
+ command line.
+
+``num_pstates``
+ Number of P-states supported by the processor (between 0 and 255
+ inclusive) including both turbo and non-turbo P-states (see
+ `Turbo P-states Support`_).
+
+ The value of this attribute is not affected by the ``no_turbo``
+ setting described `below <no_turbo_attr_>`_.
+
+ This attribute is read-only.
+
+``turbo_pct``
+ Ratio of the `turbo range <turbo_>`_ size to the size of the entire
+ range of supported P-states, in percent.
+
+ This attribute is read-only.
+
+.. _no_turbo_attr:
+
+``no_turbo``
+ If set (equal to 1), the driver is not allowed to set any turbo P-states
+ (see `Turbo P-states Support`_). If unset (equalt to 0, which is the
+ default), turbo P-states can be set by the driver.
+ [Note that ``intel_pstate`` does not support the general ``boost``
+ attribute (supported by some other scaling drivers) which is replaced
+ by this one.]
+
+ This attrubute does not affect the maximum supported frequency value
+ supplied to the ``CPUFreq`` core and exposed via the policy interface,
+ but it affects the maximum possible value of per-policy P-state limits
+ (see `Interpretation of Policy Attributes`_ below for details).
+
+.. _status_attr:
+
+``status``
+ Operation mode of the driver: "active", "passive" or "off".
+
+ "active"
+ The driver is functional and in the `active mode
+ <Active Mode_>`_.
+
+ "passive"
+ The driver is functional and in the `passive mode
+ <Passive Mode_>`_.
+
+ "off"
+ The driver is not functional (it is not registered as a scaling
+ driver with the ``CPUFreq`` core).
+
+ This attribute can be written to in order to change the driver's
+ operation mode or to unregister it. The string written to it must be
+ one of the possible values of it and, if successful, the write will
+ cause the driver to switch over to the operation mode represented by
+ that string - or to be unregistered in the "off" case. [Actually,
+ switching over from the active mode to the passive mode or the other
+ way around causes the driver to be unregistered and registered again
+ with a different set of callbacks, so all of its settings (the global
+ as well as the per-policy ones) are then reset to their default
+ values, possibly depending on the target operation mode.]
+
+ That only is supported in some configurations, though (for example, if
+ the `HWP feature is enabled in the processor <Active Mode With HWP_>`_,
+ the operation mode of the driver cannot be changed), and if it is not
+ supported in the current configuration, writes to this attribute with
+ fail with an appropriate error.
+
+Interpretation of Policy Attributes
+-----------------------------------
+
+The interpretation of some ``CPUFreq`` policy attributes described in
+:doc:`cpufreq` is special with ``intel_pstate`` as the current scaling driver
+and it generally depends on the driver's `operation mode <Operation Modes_>`_.
+
+First of all, the values of the ``cpuinfo_max_freq``, ``cpuinfo_min_freq`` and
+``scaling_cur_freq`` attributes are produced by applying a processor-specific
+multiplier to the internal P-state representation used by ``intel_pstate``.
+Also, the values of the ``scaling_max_freq`` and ``scaling_min_freq``
+attributes are capped by the frequency corresponding to the maximum P-state that
+the driver is allowed to set.
+
+If the ``no_turbo`` `global attribute <no_turbo_attr_>`_ is set, the driver is
+not allowed to use turbo P-states, so the maximum value of ``scaling_max_freq``
+and ``scaling_min_freq`` is limited to the maximum non-turbo P-state frequency.
+Accordingly, setting ``no_turbo`` causes ``scaling_max_freq`` and
+``scaling_min_freq`` to go down to that value if they were above it before.
+However, the old values of ``scaling_max_freq`` and ``scaling_min_freq`` will be
+restored after unsetting ``no_turbo``, unless these attributes have been written
+to after ``no_turbo`` was set.
+
+If ``no_turbo`` is not set, the maximum possible value of ``scaling_max_freq``
+and ``scaling_min_freq`` corresponds to the maximum supported turbo P-state,
+which also is the value of ``cpuinfo_max_freq`` in either case.
+
+Next, the following policy attributes have special meaning if
+``intel_pstate`` works in the `active mode <Active Mode_>`_:
+
+``scaling_available_governors``
+ List of P-state selection algorithms provided by ``intel_pstate``.
+
+``scaling_governor``
+ P-state selection algorithm provided by ``intel_pstate`` currently in
+ use with the given policy.
+
+``scaling_cur_freq``
+ Frequency of the average P-state of the CPU represented by the given
+ policy for the time interval between the last two invocations of the
+ driver's utilization update callback by the CPU scheduler for that CPU.
+
+The meaning of these attributes in the `passive mode <Passive Mode_>`_ is the
+same as for other scaling drivers.
+
+Additionally, the value of the ``scaling_driver`` attribute for ``intel_pstate``
+depends on the operation mode of the driver. Namely, it is either
+"intel_pstate" (in the `active mode <Active Mode_>`_) or "intel_cpufreq" (in the
+`passive mode <Passive Mode_>`_).
+
+Coordination of P-State Limits
+------------------------------
+
+``intel_pstate`` allows P-state limits to be set in two ways: with the help of
+the ``max_perf_pct`` and ``min_perf_pct`` `global attributes
+<Global Attributes_>`_ or via the ``scaling_max_freq`` and ``scaling_min_freq``
+``CPUFreq`` policy attributes. The coordination between those limits is based
+on the following rules, regardless of the current operation mode of the driver:
+
+ 1. All CPUs are affected by the global limits (that is, none of them can be
+ requested to run faster than the global maximum and none of them can be
+ requested to run slower than the global minimum).
+
+ 2. Each individual CPU is affected by its own per-policy limits (that is, it
+ cannot be requested to run faster than its own per-policy maximum and it
+ cannot be requested to run slower than its own per-policy minimum).
+
+ 3. The global and per-policy limits can be set independently.
+
+If the `HWP feature is enabled in the processor <Active Mode With HWP_>`_, the
+resulting effective values are written into its registers whenever the limits
+change in order to request its internal P-state selection logic to always set
+P-states within these limits. Otherwise, the limits are taken into account by
+scaling governors (in the `passive mode <Passive Mode_>`_) and by the driver
+every time before setting a new P-state for a CPU.
+
+Additionally, if the ``intel_pstate=per_cpu_perf_limits`` command line argument
+is passed to the kernel, ``max_perf_pct`` and ``min_perf_pct`` are not exposed
+at all and the only way to set the limits is by using the policy attributes.
+
+
+Energy vs Performance Hints
+---------------------------
+
+If ``intel_pstate`` works in the `active mode with the HWP feature enabled
+<Active Mode With HWP_>`_ in the processor, additional attributes are present
+in every ``CPUFreq`` policy directory in ``sysfs``. They are intended to allow
+user space to help ``intel_pstate`` to adjust the processor's internal P-state
+selection logic by focusing it on performance or on energy-efficiency, or
+somewhere between the two extremes:
+
+``energy_performance_preference``
+ Current value of the energy vs performance hint for the given policy
+ (or the CPU represented by it).
+
+ The hint can be changed by writing to this attribute.
+
+``energy_performance_available_preferences``
+ List of strings that can be written to the
+ ``energy_performance_preference`` attribute.
+
+ They represent different energy vs performance hints and should be
+ self-explanatory, except that ``default`` represents whatever hint
+ value was set by the platform firmware.
+
+Strings written to the ``energy_performance_preference`` attribute are
+internally translated to integer values written to the processor's
+Energy-Performance Preference (EPP) knob (if supported) or its
+Energy-Performance Bias (EPB) knob.
+
+[Note that tasks may by migrated from one CPU to another by the scheduler's
+load-balancing algorithm and if different energy vs performance hints are
+set for those CPUs, that may lead to undesirable outcomes. To avoid such
+issues it is better to set the same energy vs performance hint for all CPUs
+or to pin every task potentially sensitive to them to a specific CPU.]
+
+.. _acpi-cpufreq:
+
+``intel_pstate`` vs ``acpi-cpufreq``
+====================================
+
+On the majority of systems supported by ``intel_pstate``, the ACPI tables
+provided by the platform firmware contain ``_PSS`` objects returning information
+that can be used for CPU performance scaling (refer to the `ACPI specification`_
+for details on the ``_PSS`` objects and the format of the information returned
+by them).
+
+The information returned by the ACPI ``_PSS`` objects is used by the
+``acpi-cpufreq`` scaling driver. On systems supported by ``intel_pstate``
+the ``acpi-cpufreq`` driver uses the same hardware CPU performance scaling
+interface, but the set of P-states it can use is limited by the ``_PSS``
+output.
+
+On those systems each ``_PSS`` object returns a list of P-states supported by
+the corresponding CPU which basically is a subset of the P-states range that can
+be used by ``intel_pstate`` on the same system, with one exception: the whole
+`turbo range <turbo_>`_ is represented by one item in it (the topmost one). By
+convention, the frequency returned by ``_PSS`` for that item is greater by 1 MHz
+than the frequency of the highest non-turbo P-state listed by it, but the
+corresponding P-state representation (following the hardware specification)
+returned for it matches the maximum supported turbo P-state (or is the
+special value 255 meaning essentially "go as high as you can get").
+
+The list of P-states returned by ``_PSS`` is reflected by the table of
+available frequencies supplied by ``acpi-cpufreq`` to the ``CPUFreq`` core and
+scaling governors and the minimum and maximum supported frequencies reported by
+it come from that list as well. In particular, given the special representation
+of the turbo range described above, this means that the maximum supported
+frequency reported by ``acpi-cpufreq`` is higher by 1 MHz than the frequency
+of the highest supported non-turbo P-state listed by ``_PSS`` which, of course,
+affects decisions made by the scaling governors, except for ``powersave`` and
+``performance``.
+
+For example, if a given governor attempts to select a frequency proportional to
+estimated CPU load and maps the load of 100% to the maximum supported frequency
+(possibly multiplied by a constant), then it will tend to choose P-states below
+the turbo threshold if ``acpi-cpufreq`` is used as the scaling driver, because
+in that case the turbo range corresponds to a small fraction of the frequency
+band it can use (1 MHz vs 1 GHz or more). In consequence, it will only go to
+the turbo range for the highest loads and the other loads above 50% that might
+benefit from running at turbo frequencies will be given non-turbo P-states
+instead.
+
+One more issue related to that may appear on systems supporting the
+`Configurable TDP feature <turbo_>`_ allowing the platform firmware to set the
+turbo threshold. Namely, if that is not coordinated with the lists of P-states
+returned by ``_PSS`` properly, there may be more than one item corresponding to
+a turbo P-state in those lists and there may be a problem with avoiding the
+turbo range (if desirable or necessary). Usually, to avoid using turbo
+P-states overall, ``acpi-cpufreq`` simply avoids using the topmost state listed
+by ``_PSS``, but that is not sufficient when there are other turbo P-states in
+the list returned by it.
+
+Apart from the above, ``acpi-cpufreq`` works like ``intel_pstate`` in the
+`passive mode <Passive Mode_>`_, except that the number of P-states it can set
+is limited to the ones listed by the ACPI ``_PSS`` objects.
+
+
+Kernel Command Line Options for ``intel_pstate``
+================================================
+
+Several kernel command line options can be used to pass early-configuration-time
+parameters to ``intel_pstate`` in order to enforce specific behavior of it. All
+of them have to be prepended with the ``intel_pstate=`` prefix.
+
+``disable``
+ Do not register ``intel_pstate`` as the scaling driver even if the
+ processor is supported by it.
+
+``passive``
+ Register ``intel_pstate`` in the `passive mode <Passive Mode_>`_ to
+ start with.
+
+ This option implies the ``no_hwp`` one described below.
+
+``force``
+ Register ``intel_pstate`` as the scaling driver instead of
+ ``acpi-cpufreq`` even if the latter is preferred on the given system.
+
+ This may prevent some platform features (such as thermal controls and
+ power capping) that rely on the availability of ACPI P-states
+ information from functioning as expected, so it should be used with
+ caution.
+
+ This option does not work with processors that are not supported by
+ ``intel_pstate`` and on platforms where the ``pcc-cpufreq`` scaling
+ driver is used instead of ``acpi-cpufreq``.
+
+``no_hwp``
+ Do not enable the `hardware-managed P-states (HWP) feature
+ <Active Mode With HWP_>`_ even if it is supported by the processor.
+
+``hwp_only``
+ Register ``intel_pstate`` as the scaling driver only if the
+ `hardware-managed P-states (HWP) feature <Active Mode With HWP_>`_ is
+ supported by the processor.
+
+``support_acpi_ppc``
+ Take ACPI ``_PPC`` performance limits into account.
+
+ If the preferred power management profile in the FADT (Fixed ACPI
+ Description Table) is set to "Enterprise Server" or "Performance
+ Server", the ACPI ``_PPC`` limits are taken into account by default
+ and this option has no effect.
+
+``per_cpu_perf_limits``
+ Use per-logical-CPU P-State limits (see `Coordination of P-state
+ Limits`_ for details).
+
+
+Diagnostics and Tuning
+======================
+
+Trace Events
+------------
+
+There are two static trace events that can be used for ``intel_pstate``
+diagnostics. One of them is the ``cpu_frequency`` trace event generally used
+by ``CPUFreq``, and the other one is the ``pstate_sample`` trace event specific
+to ``intel_pstate``. Both of them are triggered by ``intel_pstate`` only if
+it works in the `active mode <Active Mode_>`_.
+
+The following sequence of shell commands can be used to enable them and see
+their output (if the kernel is generally configured to support event tracing)::
+
+ # cd /sys/kernel/debug/tracing/
+ # echo 1 > events/power/pstate_sample/enable
+ # echo 1 > events/power/cpu_frequency/enable
+ # cat trace
+ gnome-terminal--4510 [001] ..s. 1177.680733: pstate_sample: core_busy=107 scaled=94 from=26 to=26 mperf=1143818 aperf=1230607 tsc=29838618 freq=2474476
+ cat-5235 [002] ..s. 1177.681723: cpu_frequency: state=2900000 cpu_id=2
+
+If ``intel_pstate`` works in the `passive mode <Passive Mode_>`_, the
+``cpu_frequency`` trace event will be triggered either by the ``schedutil``
+scaling governor (for the policies it is attached to), or by the ``CPUFreq``
+core (for the policies with other scaling governors).
+
+``ftrace``
+----------
+
+The ``ftrace`` interface can be used for low-level diagnostics of
+``intel_pstate``. For example, to check how often the function to set a
+P-state is called, the ``ftrace`` filter can be set to to
+:c:func:`intel_pstate_set_pstate`::
+
+ # cd /sys/kernel/debug/tracing/
+ # cat available_filter_functions | grep -i pstate
+ intel_pstate_set_pstate
+ intel_pstate_cpu_init
+ ...
+ # echo intel_pstate_set_pstate > set_ftrace_filter
+ # echo function > current_tracer
+ # cat trace | head -15
+ # tracer: function
+ #
+ # entries-in-buffer/entries-written: 80/80 #P:4
+ #
+ # _-----=> irqs-off
+ # / _----=> need-resched
+ # | / _---=> hardirq/softirq
+ # || / _--=> preempt-depth
+ # ||| / delay
+ # TASK-PID CPU# |||| TIMESTAMP FUNCTION
+ # | | | |||| | |
+ Xorg-3129 [000] ..s. 2537.644844: intel_pstate_set_pstate <-intel_pstate_timer_func
+ gnome-terminal--4510 [002] ..s. 2537.649844: intel_pstate_set_pstate <-intel_pstate_timer_func
+ gnome-shell-3409 [001] ..s. 2537.650850: intel_pstate_set_pstate <-intel_pstate_timer_func
+ <idle>-0 [000] ..s. 2537.654843: intel_pstate_set_pstate <-intel_pstate_timer_func
+
+Tuning Interface in ``debugfs``
+-------------------------------
+
+The ``powersave`` algorithm provided by ``intel_pstate`` for `the Core line of
+processors in the active mode <powersave_>`_ is based on a `PID controller`_
+whose parameters were chosen to address a number of different use cases at the
+same time. However, it still is possible to fine-tune it to a specific workload
+and the ``debugfs`` interface under ``/sys/kernel/debug/pstate_snb/`` is
+provided for this purpose. [Note that the ``pstate_snb`` directory will be
+present only if the specific P-state selection algorithm matching the interface
+in it actually is in use.]
+
+The following files present in that directory can be used to modify the PID
+controller parameters at run time:
+
+| ``deadband``
+| ``d_gain_pct``
+| ``i_gain_pct``
+| ``p_gain_pct``
+| ``sample_rate_ms``
+| ``setpoint``
+
+Note, however, that achieving desirable results this way generally requires
+expert-level understanding of the power vs performance tradeoff, so extra care
+is recommended when attempting to do that.
+
+
+.. _LCEU2015: http://events.linuxfoundation.org/sites/events/files/slides/LinuxConEurope_2015.pdf
+.. _SDM: http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32-architectures-software-developer-system-programming-manual-325384.html
+.. _ACPI specification: http://www.uefi.org/sites/default/files/resources/ACPI_6_1.pdf
+.. _PID controller: https://en.wikipedia.org/wiki/PID_controller
+++ /dev/null
-Intel P-State driver
---------------------
-
-This driver provides an interface to control the P-State selection for the
-SandyBridge+ Intel processors.
-
-The following document explains P-States:
-http://events.linuxfoundation.org/sites/events/files/slides/LinuxConEurope_2015.pdf
-As stated in the document, P-State doesn’t exactly mean a frequency. However, for
-the sake of the relationship with cpufreq, P-State and frequency are used
-interchangeably.
-
-Understanding the cpufreq core governors and policies are important before
-discussing more details about the Intel P-State driver. Based on what callbacks
-a cpufreq driver provides to the cpufreq core, it can support two types of
-drivers:
-- with target_index() callback: In this mode, the drivers using cpufreq core
-simply provide the minimum and maximum frequency limits and an additional
-interface target_index() to set the current frequency. The cpufreq subsystem
-has a number of scaling governors ("performance", "powersave", "ondemand",
-etc.). Depending on which governor is in use, cpufreq core will call for
-transitions to a specific frequency using target_index() callback.
-- setpolicy() callback: In this mode, drivers do not provide target_index()
-callback, so cpufreq core can't request a transition to a specific frequency.
-The driver provides minimum and maximum frequency limits and callbacks to set a
-policy. The policy in cpufreq sysfs is referred to as the "scaling governor".
-The cpufreq core can request the driver to operate in any of the two policies:
-"performance" and "powersave". The driver decides which frequency to use based
-on the above policy selection considering minimum and maximum frequency limits.
-
-The Intel P-State driver falls under the latter category, which implements the
-setpolicy() callback. This driver decides what P-State to use based on the
-requested policy from the cpufreq core. If the processor is capable of
-selecting its next P-State internally, then the driver will offload this
-responsibility to the processor (aka HWP: Hardware P-States). If not, the
-driver implements algorithms to select the next P-State.
-
-Since these policies are implemented in the driver, they are not same as the
-cpufreq scaling governors implementation, even if they have the same name in
-the cpufreq sysfs (scaling_governors). For example the "performance" policy is
-similar to cpufreq’s "performance" governor, but "powersave" is completely
-different than the cpufreq "powersave" governor. The strategy here is similar
-to cpufreq "ondemand", where the requested P-State is related to the system load.
-
-Sysfs Interface
-
-In addition to the frequency-controlling interfaces provided by the cpufreq
-core, the driver provides its own sysfs files to control the P-State selection.
-These files have been added to /sys/devices/system/cpu/intel_pstate/.
-Any changes made to these files are applicable to all CPUs (even in a
-multi-package system, Refer to later section on placing "Per-CPU limits").
-
- max_perf_pct: Limits the maximum P-State that will be requested by
- the driver. It states it as a percentage of the available performance. The
- available (P-State) performance may be reduced by the no_turbo
- setting described below.
-
- min_perf_pct: Limits the minimum P-State that will be requested by
- the driver. It states it as a percentage of the max (non-turbo)
- performance level.
-
- no_turbo: Limits the driver to selecting P-State below the turbo
- frequency range.
-
- turbo_pct: Displays the percentage of the total performance that
- is supported by hardware that is in the turbo range. This number
- is independent of whether turbo has been disabled or not.
-
- num_pstates: Displays the number of P-States that are supported
- by hardware. This number is independent of whether turbo has
- been disabled or not.
-
-For example, if a system has these parameters:
- Max 1 core turbo ratio: 0x21 (Max 1 core ratio is the maximum P-State)
- Max non turbo ratio: 0x17
- Minimum ratio : 0x08 (Here the ratio is called max efficiency ratio)
-
-Sysfs will show :
- max_perf_pct:100, which corresponds to 1 core ratio
- min_perf_pct:24, max_efficiency_ratio / max 1 Core ratio
- no_turbo:0, turbo is not disabled
- num_pstates:26 = (max 1 Core ratio - Max Efficiency Ratio + 1)
- turbo_pct:39 = (max 1 core ratio - max non turbo ratio) / num_pstates
-
-Refer to "Intel® 64 and IA-32 Architectures Software Developer’s Manual
-Volume 3: System Programming Guide" to understand ratios.
-
-There is one more sysfs attribute in /sys/devices/system/cpu/intel_pstate/
-that can be used for controlling the operation mode of the driver:
-
- status: Three settings are possible:
- "off" - The driver is not in use at this time.
- "active" - The driver works as a P-state governor (default).
- "passive" - The driver works as a regular cpufreq one and collaborates
- with the generic cpufreq governors (it sets P-states as
- requested by those governors).
- The current setting is returned by reads from this attribute. Writing one
- of the above strings to it changes the operation mode as indicated by that
- string, if possible. If HW-managed P-states (HWP) are enabled, it is not
- possible to change the driver's operation mode and attempts to write to
- this attribute will fail.
-
-cpufreq sysfs for Intel P-State
-
-Since this driver registers with cpufreq, cpufreq sysfs is also presented.
-There are some important differences, which need to be considered.
-
-scaling_cur_freq: This displays the real frequency which was used during
-the last sample period instead of what is requested. Some other cpufreq driver,
-like acpi-cpufreq, displays what is requested (Some changes are on the
-way to fix this for acpi-cpufreq driver). The same is true for frequencies
-displayed at /proc/cpuinfo.
-
-scaling_governor: This displays current active policy. Since each CPU has a
-cpufreq sysfs, it is possible to set a scaling governor to each CPU. But this
-is not possible with Intel P-States, as there is one common policy for all
-CPUs. Here, the last requested policy will be applicable to all CPUs. It is
-suggested that one use the cpupower utility to change policy to all CPUs at the
-same time.
-
-scaling_setspeed: This attribute can never be used with Intel P-State.
-
-scaling_max_freq/scaling_min_freq: This interface can be used similarly to
-the max_perf_pct/min_perf_pct of Intel P-State sysfs. However since frequencies
-are converted to nearest possible P-State, this is prone to rounding errors.
-This method is not preferred to limit performance.
-
-affected_cpus: Not used
-related_cpus: Not used
-
-For contemporary Intel processors, the frequency is controlled by the
-processor itself and the P-State exposed to software is related to
-performance levels. The idea that frequency can be set to a single
-frequency is fictional for Intel Core processors. Even if the scaling
-driver selects a single P-State, the actual frequency the processor
-will run at is selected by the processor itself.
-
-Per-CPU limits
-
-The kernel command line option "intel_pstate=per_cpu_perf_limits" forces
-the intel_pstate driver to use per-CPU performance limits. When it is set,
-the sysfs control interface described above is subject to limitations.
-- The following controls are not available for both read and write
- /sys/devices/system/cpu/intel_pstate/max_perf_pct
- /sys/devices/system/cpu/intel_pstate/min_perf_pct
-- The following controls can be used to set performance limits, as far as the
-architecture of the processor permits:
- /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
- /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
- /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
-- User can still observe turbo percent and number of P-States from
- /sys/devices/system/cpu/intel_pstate/turbo_pct
- /sys/devices/system/cpu/intel_pstate/num_pstates
-- User can read write system wide turbo status
- /sys/devices/system/cpu/no_turbo
-
-Support of energy performance hints
-It is possible to provide hints to the HWP algorithms in the processor
-to be more performance centric to more energy centric. When the driver
-is using HWP, two additional cpufreq sysfs attributes are presented for
-each logical CPU.
-These attributes are:
- - energy_performance_available_preferences
- - energy_performance_preference
-
-To get list of supported hints:
-$ cat energy_performance_available_preferences
- default performance balance_performance balance_power power
-
-The current preference can be read or changed via cpufreq sysfs
-attribute "energy_performance_preference". Reading from this attribute
-will display current effective setting. User can write any of the valid
-preference string to this attribute. User can always restore to power-on
-default by writing "default".
-
-Since threads can migrate to different CPUs, this is possible that the
-new CPU may have different energy performance preference than the previous
-one. To avoid such issues, either threads can be pinned to specific CPUs
-or set the same energy performance preference value to all CPUs.
-
-Tuning Intel P-State driver
-
-When the performance can be tuned using PID (Proportional Integral
-Derivative) controller, debugfs files are provided for adjusting performance.
-They are presented under:
-/sys/kernel/debug/pstate_snb/
-
-The PID tunable parameters are:
- deadband
- d_gain_pct
- i_gain_pct
- p_gain_pct
- sample_rate_ms
- setpoint
-
-To adjust these parameters, some understanding of driver implementation is
-necessary. There are some tweeks described here, but be very careful. Adjusting
-them requires expert level understanding of power and performance relationship.
-These limits are only useful when the "powersave" policy is active.
-
--To make the system more responsive to load changes, sample_rate_ms can
-be adjusted (current default is 10ms).
--To make the system use higher performance, even if the load is lower, setpoint
-can be adjusted to a lower number. This will also lead to faster ramp up time
-to reach the maximum P-State.
-If there are no derivative and integral coefficients, The next P-State will be
-equal to:
- current P-State - ((setpoint - current cpu load) * p_gain_pct)
-
-For example, if the current PID parameters are (Which are defaults for the core
-processors like SandyBridge):
- deadband = 0
- d_gain_pct = 0
- i_gain_pct = 0
- p_gain_pct = 20
- sample_rate_ms = 10
- setpoint = 97
-
-If the current P-State = 0x08 and current load = 100, this will result in the
-next P-State = 0x08 - ((97 - 100) * 0.2) = 8.6 (rounded to 9). Here the P-State
-goes up by only 1. If during next sample interval the current load doesn't
-change and still 100, then P-State goes up by one again. This process will
-continue as long as the load is more than the setpoint until the maximum P-State
-is reached.
-
-For the same load at setpoint = 60, this will result in the next P-State
-= 0x08 - ((60 - 100) * 0.2) = 16
-So by changing the setpoint from 97 to 60, there is an increase of the
-next P-State from 9 to 16. So this will make processor execute at higher
-P-State for the same CPU load. If the load continues to be more than the
-setpoint during next sample intervals, then P-State will go up again till the
-maximum P-State is reached. But the ramp up time to reach the maximum P-State
-will be much faster when the setpoint is 60 compared to 97.
-
-Debugging Intel P-State driver
-
-Event tracing
-To debug P-State transition, the Linux event tracing interface can be used.
-There are two specific events, which can be enabled (Provided the kernel
-configs related to event tracing are enabled).
-
-# cd /sys/kernel/debug/tracing/
-# echo 1 > events/power/pstate_sample/enable
-# echo 1 > events/power/cpu_frequency/enable
-# cat trace
-gnome-terminal--4510 [001] ..s. 1177.680733: pstate_sample: core_busy=107
- scaled=94 from=26 to=26 mperf=1143818 aperf=1230607 tsc=29838618
- freq=2474476
-cat-5235 [002] ..s. 1177.681723: cpu_frequency: state=2900000 cpu_id=2
-
-
-Using ftrace
-
-If function level tracing is required, the Linux ftrace interface can be used.
-For example if we want to check how often a function to set a P-State is
-called, we can set ftrace filter to intel_pstate_set_pstate.
-
-# cd /sys/kernel/debug/tracing/
-# cat available_filter_functions | grep -i pstate
-intel_pstate_set_pstate
-intel_pstate_cpu_init
-...
-
-# echo intel_pstate_set_pstate > set_ftrace_filter
-# echo function > current_tracer
-# cat trace | head -15
-# tracer: function
-#
-# entries-in-buffer/entries-written: 80/80 #P:4
-#
-# _-----=> irqs-off
-# / _----=> need-resched
-# | / _---=> hardirq/softirq
-# || / _--=> preempt-depth
-# ||| / delay
-# TASK-PID CPU# |||| TIMESTAMP FUNCTION
-# | | | |||| | |
- Xorg-3129 [000] ..s. 2537.644844: intel_pstate_set_pstate <-intel_pstate_timer_func
- gnome-terminal--4510 [002] ..s. 2537.649844: intel_pstate_set_pstate <-intel_pstate_timer_func
- gnome-shell-3409 [001] ..s. 2537.650850: intel_pstate_set_pstate <-intel_pstate_timer_func
- <idle>-0 [000] ..s. 2537.654843: intel_pstate_set_pstate <-intel_pstate_timer_func
- compatible: value should be one of:
"samsung,exynos5433-decon", "samsung,exynos5433-decon-tv";
- reg: physical base address and length of the DECON registers set.
-- interrupts: should contain a list of all DECON IP block interrupts in the
- order: VSYNC, LCD_SYSTEM. The interrupt specifier format
- depends on the interrupt controller used.
-- interrupt-names: should contain the interrupt names: "vsync", "lcd_sys"
- in the same order as they were listed in the interrupts
- property.
+- interrupt-names: should contain the interrupt names depending on mode of work:
+ video mode: "vsync",
+ command mode: "lcd_sys",
+ command mode with software trigger: "lcd_sys", "te".
+- interrupts or interrupts-extended: list of interrupt specifiers corresponding
+ to names privided in interrupt-names, as described in
+ interrupt-controller/interrupts.txt
- clocks: must include clock specifiers corresponding to entries in the
clock-names property.
- clock-names: list of clock names sorted in the same order as the clocks
control gpios
- threshold: allows setting the "click"-threshold in the range
- from 20 to 80.
+ from 0 to 80.
- gain: allows setting the sensitivity in the range from 0 to
31. Note that lower values indicate higher
- reg: Base address of PMIC on Hi6220 SoC.
- interrupt-controller: Hi655x has internal IRQs (has own IRQ domain).
- pmic-gpios: The GPIO used by PMIC IRQ.
+- #clock-cells: From common clock binding; shall be set to 0
+
+Optional properties:
+- clock-output-names: From common clock binding to override the
+ default output clock name
Example:
pmic: pmic@f8000000 {
interrupt-controller;
#interrupt-cells = <2>;
pmic-gpios = <&gpio1 2 GPIO_ACTIVE_HIGH>;
+ #clock-cells = <0>;
}
"ext_clock" (External clock provided to the card).
- post-power-on-delay-ms : Delay in ms after powering the card and
de-asserting the reset-gpios (if any)
+- power-off-delay-us : Delay in us after asserting the reset-gpios (if any)
+ during power off of the card.
Example:
- phy-reset-active-high : If present then the reset sequence using the GPIO
specified in the "phy-reset-gpios" property is reversed (H=reset state,
L=operation state).
+- phy-reset-post-delay : Post reset delay in milliseconds. If present then
+ a delay of phy-reset-post-delay milliseconds will be observed after the
+ phy-reset-gpios has been toggled. Can be omitted thus no delay is
+ observed. Delay is in range of 1ms to 1000ms. Other delays are invalid.
- phy-supply : regulator that powers the Ethernet PHY.
- phy-handle : phandle to the PHY device connected to this device.
- fixed-link : Assume a fixed link. See fixed-link.txt in the same directory.
+++ /dev/null
-Hi6220 SoC ION
-===================================================================
-Required properties:
-- compatible : "hisilicon,hi6220-ion"
-- list of the ION heaps
- - heap name : maybe heap_sys_user@0
- - heap id : id should be unique in the system.
- - heap base : base ddr address of the heap,0 means that
- it is dynamic.
- - heap size : memory size and 0 means it is dynamic.
- - heap type : the heap type of the heap, please also
- see the define in ion.h(drivers/staging/android/uapi/ion.h)
--------------------------------------------------------------------
-Example:
- hi6220-ion {
- compatible = "hisilicon,hi6220-ion";
- heap_sys_user@0 {
- heap-name = "sys_user";
- heap-id = <0x0>;
- heap-base = <0x0>;
- heap-size = <0x0>;
- heap-type = "ion_system";
- };
- heap_sys_contig@0 {
- heap-name = "sys_contig";
- heap-id = <0x1>;
- heap-base = <0x0>;
- heap-size = <0x0>;
- heap-type = "ion_system_contig";
- };
- };
implement appropriate obj->atomic_get_property() vfuncs for any
modeset objects with driver specific properties.
+DRIVER_SYNCOBJ
+ Driver support drm sync objects.
+
Major, Minor and Patchlevel
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. kernel-doc:: drivers/gpu/drm/drm_cache.c
:export:
+
+DRM Sync Objects
+===========================
+
+.. kernel-doc:: drivers/gpu/drm/drm_syncobj.c
+ :doc: Overview
+
+.. kernel-doc:: include/drm/drm_syncobj.h
+ :export:
+
+.. kernel-doc:: drivers/gpu/drm/drm_syncobj.c
+ :export:
The driver allows configuration of the touch screen via a set of sysfs files:
/sys/class/input/eventX/device/device/threshold:
- allows setting the "click"-threshold in the range from 20 to 80.
+ allows setting the "click"-threshold in the range from 0 to 80.
/sys/class/input/eventX/device/device/gain:
allows setting the sensitivity in the range from 0 to 31. Note that
6-jack in back, 2-jack in front
6stack-digout
6-jack with a SPDIF out
+6stack-automute
+ 6-jack with headphone jack detection
ALC260
======
Enables docking station I/O for some Lenovos
hp-gpio-led
GPIO LED support on HP laptops
+hp-dock-gpio-mic1-led
+ HP dock with mic LED support
dell-headset-multi
Headset jack, which can also be used as mic-in
dell-headset-dock
Combo jack sensing on ALC283
tpt440-dock
Pin configs for Lenovo Thinkpad Dock support
+tpt440
+ Lenovo Thinkpad T440s setup
+tpt460
+ Lenovo Thinkpad T460/560 setup
+dual-codecs
+ Lenovo laptops with dual codecs
ALC66x/67x/892
==============
Inverted internal mic workaround
dell-headset-multi
Headset jack, which can also be used as mic-in
+dual-codecs
+ Lenovo laptops with dual codecs
ALC680
======
Inverted internal mic workaround
no-primary-hp
VAIO Z/VGC-LN51JGB workaround (for fixed speaker DAC)
+dual-codecs
+ ALC1220 dual codecs for Gaming mobos
ALC861/660
==========
Conexant 5045
=============
-laptop-hpsense
- Laptop with HP sense (old model laptop)
-laptop-micsense
- Laptop with Mic sense (old model fujitsu)
-laptop-hpmicsense
- Laptop with HP and Mic senses
-benq
- Benq R55E
-laptop-hp530
- HP 530 laptop
-test
- for testing/debugging purpose, almost all controls can be
- adjusted. Appearing only when compiled with $CONFIG_SND_DEBUG=y
+cap-mix-amp
+ Fix max input level on mixer widget
+toshiba-p105
+ Toshiba P105 quirk
+hp-530
+ HP 530 quirk
Conexant 5047
=============
-laptop
- Basic Laptop config
-laptop-hp
- Laptop config for some HP models (subdevice 30A5)
-laptop-eapd
- Laptop config with EAPD support
-test
- for testing/debugging purpose, almost all controls can be
- adjusted. Appearing only when compiled with $CONFIG_SND_DEBUG=y
+cap-mix-amp
+ Fix max input level on mixer widget
Conexant 5051
=============
-laptop
- Basic Laptop config (default)
-hp
- HP Spartan laptop
-hp-dv6736
- HP dv6736
-hp-f700
- HP Compaq Presario F700
-ideapad
- Lenovo IdeaPad laptop
-toshiba
- Toshiba Satellite M300
+lenovo-x200
+ Lenovo X200 quirk
Conexant 5066
=============
-laptop
- Basic Laptop config (default)
-hp-laptop
- HP laptops, e g G60
-asus
- Asus K52JU, Lenovo G560
-dell-laptop
- Dell laptops
-dell-vostro
- Dell Vostro
-olpc-xo-1_5
- OLPC XO 1.5
-ideapad
- Lenovo IdeaPad U150
+stereo-dmic
+ Workaround for inverted stereo digital mic
+gpio1
+ Enable GPIO1 pin
+headphone-mic-pin
+ Enable headphone mic NID 0x18 without detection
+tp410
+ Thinkpad T400 & co quirks
thinkpad
- Lenovo Thinkpad
+ Thinkpad mute/mic LED quirk
+lemote-a1004
+ Lemote A1004 quirk
+lemote-a1205
+ Lemote A1205 quirk
+olpc-xo
+ OLPC XO quirk
+mute-led-eapd
+ Mute LED control via EAPD
+hp-dock
+ HP dock support
+mute-led-gpio
+ Mute LED control via GPIO
STAC9200
========
Dell desktops/laptops
alienware
Alienware M17x
+asus-mobo
+ Pin configs for ASUS mobo with 5.1/SPDIF out
auto
BIOS setup (default)
Pin fixup for HP Envy TS bass speaker (NID 0x10)
hp-bnb13-eq
Hardware equalizer setup for HP laptops
+hp-envy-ts-bass
+ HP Envy TS bass support
auto
BIOS setup (default)
Cirrus Logic CS4206/4207
========================
+mbp53
+ MacBook Pro 5,3
mbp55
MacBook Pro 5,5
imac27
IMac 27 Inch
+imac27_122
+ iMac 12,2
+apple
+ Generic Apple quirk
+mbp101
+ MacBookPro 10,1
+mbp81
+ MacBookPro 8,1
+mba42
+ MacBookAir 4,2
auto
BIOS setup (default)
MacBook Air 6,1 and 6,2
gpio0
Enable GPIO 0 amp
+mbp11
+ MacBookPro 11,2
+macmini
+ MacMini 7,1
auto
BIOS setup (default)
registering/unregistering cables and their plugs:
.. kernel-doc:: drivers/usb/typec/typec.c
- :functions: typec_register_cable typec_unregister_cable typec_register_plug
- typec_unregister_plug
+ :functions: typec_register_cable typec_unregister_cable typec_register_plug typec_unregister_plug
The class will provide a handle to struct typec_cable and struct typec_plug if
the registration is successful, or NULL if it isn't.
APIs to report it to the class:
.. kernel-doc:: drivers/usb/typec/typec.c
- :functions: typec_set_data_role typec_set_pwr_role typec_set_vconn_role
- typec_set_pwr_opmode
+ :functions: typec_set_data_role typec_set_pwr_role typec_set_vconn_role typec_set_pwr_opmode
Alternate Modes
~~~~~~~~~~~~~~~
-------------------------------------------------
iTCO_wdt:
heartbeat: Watchdog heartbeat in seconds.
- (2<heartbeat<39 (TCO v1) or 613 (TCO v2), default=30)
+ (5<=heartbeat<=74 (TCO v1) or 1226 (TCO v2), default=30)
nowayout: Watchdog cannot be stopped once started
(default=kernel config parameter)
-------------------------------------------------
M: Sumit Semwal <sumit.semwal@linaro.org>
L: devel@driverdev.osuosl.org
S: Supported
-F: Documentation/devicetree/bindings/staging/ion/
F: drivers/staging/android/ion
F: drivers/staging/android/uapi/ion.h
F: drivers/staging/android/uapi/ion_test.h
F: include/linux/spi/cc2520.h
F: Documentation/devicetree/bindings/net/ieee802154/cc2520.txt
+CCREE ARM TRUSTZONE CRYPTOCELL 700 REE DRIVER
+M: Gilad Ben-Yossef <gilad@benyossef.com>
+L: linux-crypto@vger.kernel.org
+L: driverdev-devel@linuxdriverproject.org
+S: Supported
+F: drivers/staging/ccree/
+W: https://developer.arm.com/products/system-ip/trustzone-cryptocell/cryptocell-700-family
+
CEC FRAMEWORK
M: Hans Verkuil <hans.verkuil@cisco.com>
L: linux-media@vger.kernel.org
M: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
S: Maintained
F: drivers/staging/greybus/
-L: greybus-dev@lists.linaro.org
+L: greybus-dev@lists.linaro.org (moderated for non-subscribers)
GREYBUS AUDIO PROTOCOLS DRIVERS
M: Vaibhav Agarwal <vaibhav.sr@gmail.com>
F: drivers/media/platform/rcar_jpu.c
JSM Neo PCI based serial card
-M: Gabriel Krisman Bertazi <krisman@linux.vnet.ibm.com>
+M: Guilherme G. Piccoli <gpiccoli@linux.vnet.ibm.com>
L: linux-serial@vger.kernel.org
S: Maintained
F: drivers/tty/serial/jsm/
OSD LIBRARY and FILESYSTEM
M: Boaz Harrosh <ooo@electrozaur.com>
-M: Benny Halevy <bhalevy@primarydata.com>
-L: osd-dev@open-osd.org
-W: http://open-osd.org
-T: git git://git.open-osd.org/open-osd.git
S: Maintained
F: drivers/scsi/osd/
F: include/scsi/osd_*
VERSION = 4
PATCHLEVEL = 12
SUBLEVEL = 0
-EXTRAVERSION = -rc1
+EXTRAVERSION = -rc3
NAME = Fearless Coyote
# *DOCUMENTATION*
PHONY += headers_check
headers_check: headers_install
$(Q)$(MAKE) $(hdr-inst)=include/uapi HDRCHECK=1
- $(Q)$(MAKE) $(hdr-inst)=arch/$(hdr-arch)/include/uapi/ $(hdr-dst) HDRCHECK=1
+ $(Q)$(MAKE) $(hdr-inst)=arch/$(hdr-arch)/include/uapi $(hdr-dst) HDRCHECK=1
# ---------------------------------------------------------------------------
# Kernel selftest
if (!access_ok(VERIFY_WRITE, ur, sizeof(*ur)))
return -EFAULT;
- err = 0;
- err |= put_user(status, ustatus);
+ err = put_user(status, ustatus);
+ if (ret < 0)
+ return err ? err : ret;
+
err |= __put_user(r.ru_utime.tv_sec, &ur->ru_utime.tv_sec);
err |= __put_user(r.ru_utime.tv_usec, &ur->ru_utime.tv_usec);
err |= __put_user(r.ru_stime.tv_sec, &ur->ru_stime.tv_sec);
/ {
aliases {
- ethernet = ðernet;
+ ethernet0 = ðernet;
};
};
/ {
aliases {
- ethernet = ðernet;
+ ethernet0 = ðernet;
};
};
brcm,pins = <0 1>;
brcm,function = <BCM2835_FSEL_ALT0>;
};
- i2c0_gpio32: i2c0_gpio32 {
- brcm,pins = <32 34>;
+ i2c0_gpio28: i2c0_gpio28 {
+ brcm,pins = <28 29>;
brcm,function = <BCM2835_FSEL_ALT0>;
};
i2c0_gpio44: i2c0_gpio44 {
/* Separate from the uart0_gpio14 group
* because it conflicts with spi1_gpio16, and
* people often run uart0 on the two pins
- * without flow contrl.
+ * without flow control.
*/
uart0_ctsrts_gpio16: uart0_ctsrts_gpio16 {
brcm,pins = <16 17>;
brcm,function = <BCM2835_FSEL_ALT3>;
};
- uart0_gpio30: uart0_gpio30 {
+ uart0_ctsrts_gpio30: uart0_ctsrts_gpio30 {
brcm,pins = <30 31>;
brcm,function = <BCM2835_FSEL_ALT3>;
};
- uart0_ctsrts_gpio32: uart0_ctsrts_gpio32 {
+ uart0_gpio32: uart0_gpio32 {
brcm,pins = <32 33>;
brcm,function = <BCM2835_FSEL_ALT3>;
};
+ uart0_gpio36: uart0_gpio36 {
+ brcm,pins = <36 37>;
+ brcm,function = <BCM2835_FSEL_ALT2>;
+ };
+ uart0_ctsrts_gpio38: uart0_ctsrts_gpio38 {
+ brcm,pins = <38 39>;
+ brcm,function = <BCM2835_FSEL_ALT2>;
+ };
uart1_gpio14: uart1_gpio14 {
brcm,pins = <14 15>;
brcm,pins = <30 31>;
brcm,function = <BCM2835_FSEL_ALT5>;
};
- uart1_gpio36: uart1_gpio36 {
- brcm,pins = <36 37 38 39>;
- brcm,function = <BCM2835_FSEL_ALT2>;
- };
uart1_gpio40: uart1_gpio40 {
brcm,pins = <40 41>;
brcm,function = <BCM2835_FSEL_ALT5>;
tps659038: tps659038@58 {
compatible = "ti,tps659038";
reg = <0x58>;
+ ti,palmas-override-powerhold;
+ ti,system-power-controller;
tps659038_pmic {
compatible = "ti,tps659038-pmic";
coefficients = <0 2000>;
};
+&cpu_crit {
+ temperature = <120000>; /* milli Celsius */
+};
+
/include/ "dra7xx-clocks.dtsi"
imx53-qsrb {
pinctrl_pmic: pmicgrp {
fsl,pins = <
- MX53_PAD_CSI0_DAT5__GPIO5_23 0x1e4 /* IRQ */
+ MX53_PAD_CSI0_DAT5__GPIO5_23 0x1c4 /* IRQ */
>;
};
};
model = "Freescale i.MX6 SoloX SDB RevB Board";
};
-&cpu0 {
- operating-points = <
- /* kHz uV */
- 996000 1250000
- 792000 1175000
- 396000 1175000
- 198000 1175000
- >;
- fsl,soc-operating-points = <
- /* ARM kHz SOC uV */
- 996000 1250000
- 792000 1175000
- 396000 1175000
- 198000 1175000
- >;
-};
-
&i2c1 {
clock-frequency = <100000>;
pinctrl-names = "default";
+++ /dev/null
-..
\ No newline at end of file
+++ /dev/null
-../../../../arm64/boot/dts
\ No newline at end of file
+++ /dev/null
-../../../../../include/dt-bindings
\ No newline at end of file
OMAP3_CORE1_IOPAD(0x2110, PIN_INPUT | MUX_MODE0) /* cam_xclka.cam_xclka */
OMAP3_CORE1_IOPAD(0x2112, PIN_INPUT | MUX_MODE0) /* cam_pclk.cam_pclk */
- OMAP3_CORE1_IOPAD(0x2114, PIN_INPUT | MUX_MODE0) /* cam_d0.cam_d0 */
- OMAP3_CORE1_IOPAD(0x2116, PIN_INPUT | MUX_MODE0) /* cam_d1.cam_d1 */
- OMAP3_CORE1_IOPAD(0x2118, PIN_INPUT | MUX_MODE0) /* cam_d2.cam_d2 */
+ OMAP3_CORE1_IOPAD(0x2116, PIN_INPUT | MUX_MODE0) /* cam_d0.cam_d0 */
+ OMAP3_CORE1_IOPAD(0x2118, PIN_INPUT | MUX_MODE0) /* cam_d1.cam_d1 */
+ OMAP3_CORE1_IOPAD(0x211a, PIN_INPUT | MUX_MODE0) /* cam_d2.cam_d2 */
OMAP3_CORE1_IOPAD(0x211c, PIN_INPUT | MUX_MODE0) /* cam_d3.cam_d3 */
OMAP3_CORE1_IOPAD(0x211e, PIN_INPUT | MUX_MODE0) /* cam_d4.cam_d4 */
OMAP3_CORE1_IOPAD(0x2120, PIN_INPUT | MUX_MODE0) /* cam_d5.cam_d5 */
<GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>,
<GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>,
<GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
+ clock-frequency = <13000000>;
+ arm,cpu-registers-not-fw-configured;
};
watchdog: watchdog@10007000 {
simple-audio-card,bitclock-master = <&telephony_link_master>;
simple-audio-card,frame-master = <&telephony_link_master>;
simple-audio-card,format = "i2s";
-
+ simple-audio-card,bitclock-inversion;
+ simple-audio-card,frame-inversion;
simple-audio-card,cpu {
sound-dai = <&mcbsp4>;
};
/* Pandaboard Rev A4+ have external pullups on SCL & SDA */
&dss_hdmi_pins {
pinctrl-single,pins = <
- OMAP4_IOPAD(0x09a, PIN_INPUT_PULLUP | MUX_MODE0) /* hdmi_cec.hdmi_cec */
+ OMAP4_IOPAD(0x09a, PIN_INPUT | MUX_MODE0) /* hdmi_cec.hdmi_cec */
OMAP4_IOPAD(0x09c, PIN_INPUT | MUX_MODE0) /* hdmi_scl.hdmi_scl */
OMAP4_IOPAD(0x09e, PIN_INPUT | MUX_MODE0) /* hdmi_sda.hdmi_sda */
>;
/* PandaboardES has external pullups on SCL & SDA */
&dss_hdmi_pins {
pinctrl-single,pins = <
- OMAP4_IOPAD(0x09a, PIN_INPUT_PULLUP | MUX_MODE0) /* hdmi_cec.hdmi_cec */
+ OMAP4_IOPAD(0x09a, PIN_INPUT | MUX_MODE0) /* hdmi_cec.hdmi_cec */
OMAP4_IOPAD(0x09c, PIN_INPUT | MUX_MODE0) /* hdmi_scl.hdmi_scl */
OMAP4_IOPAD(0x09e, PIN_INPUT | MUX_MODE0) /* hdmi_sda.hdmi_sda */
>;
--- /dev/null
+# CONFIG_LOCALVERSION_AUTO is not set
+CONFIG_SYSVIPC=y
+CONFIG_NO_HZ_IDLE=y
+CONFIG_BSD_PROCESS_ACCT=y
+CONFIG_USER_NS=y
+CONFIG_RELAY=y
+CONFIG_BLK_DEV_INITRD=y
+CONFIG_PARTITION_ADVANCED=y
+CONFIG_ARCH_MULTI_V4=y
+# CONFIG_ARCH_MULTI_V7 is not set
+CONFIG_ARCH_GEMINI=y
+CONFIG_PCI=y
+CONFIG_PREEMPT=y
+CONFIG_AEABI=y
+CONFIG_CMDLINE="console=ttyS0,115200n8"
+CONFIG_KEXEC=y
+CONFIG_BINFMT_MISC=y
+CONFIG_PM=y
+CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
+CONFIG_DEVTMPFS=y
+CONFIG_MTD=y
+CONFIG_MTD_BLOCK=y
+CONFIG_MTD_CFI=y
+CONFIG_MTD_CFI_INTELEXT=y
+CONFIG_MTD_CFI_AMDSTD=y
+CONFIG_MTD_CFI_STAA=y
+CONFIG_MTD_PHYSMAP=y
+CONFIG_MTD_PHYSMAP_OF=y
+CONFIG_BLK_DEV_RAM=y
+CONFIG_BLK_DEV_RAM_SIZE=16384
+# CONFIG_SCSI_PROC_FS is not set
+CONFIG_BLK_DEV_SD=y
+# CONFIG_SCSI_LOWLEVEL is not set
+CONFIG_ATA=y
+CONFIG_INPUT_EVDEV=y
+CONFIG_KEYBOARD_GPIO=y
+# CONFIG_INPUT_MOUSE is not set
+# CONFIG_LEGACY_PTYS is not set
+CONFIG_SERIAL_8250=y
+CONFIG_SERIAL_8250_CONSOLE=y
+CONFIG_SERIAL_8250_NR_UARTS=1
+CONFIG_SERIAL_8250_RUNTIME_UARTS=1
+CONFIG_SERIAL_OF_PLATFORM=y
+# CONFIG_HW_RANDOM is not set
+# CONFIG_HWMON is not set
+CONFIG_WATCHDOG=y
+CONFIG_GEMINI_WATCHDOG=y
+CONFIG_USB=y
+CONFIG_USB_MON=y
+CONFIG_USB_FOTG210_HCD=y
+CONFIG_USB_STORAGE=y
+CONFIG_NEW_LEDS=y
+CONFIG_LEDS_CLASS=y
+CONFIG_LEDS_GPIO=y
+CONFIG_LEDS_TRIGGERS=y
+CONFIG_LEDS_TRIGGER_HEARTBEAT=y
+CONFIG_RTC_CLASS=y
+CONFIG_RTC_DRV_GEMINI=y
+CONFIG_DMADEVICES=y
+# CONFIG_DNOTIFY is not set
+CONFIG_TMPFS=y
+CONFIG_TMPFS_POSIX_ACL=y
+CONFIG_ROMFS_FS=y
+CONFIG_NLS_CODEPAGE_437=y
+CONFIG_NLS_ISO8859_1=y
+# CONFIG_ENABLE_WARN_DEPRECATED is not set
+# CONFIG_ENABLE_MUST_CHECK is not set
+CONFIG_DEBUG_FS=y
int kvm_handle_cp10_id(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_handle_cp_0_13_access(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_handle_cp14_load_store(struct kvm_vcpu *vcpu, struct kvm_run *run);
-int kvm_handle_cp14_access(struct kvm_vcpu *vcpu, struct kvm_run *run);
+int kvm_handle_cp14_32(struct kvm_vcpu *vcpu, struct kvm_run *run);
+int kvm_handle_cp14_64(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_handle_cp15_32(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_handle_cp15_64(struct kvm_vcpu *vcpu, struct kvm_run *run);
#include <asm/vfp.h>
#include "../vfp/vfpinstr.h"
+#define CREATE_TRACE_POINTS
#include "trace.h"
#include "coproc.h"
return 1;
}
-int kvm_handle_cp14_access(struct kvm_vcpu *vcpu, struct kvm_run *run)
-{
- kvm_inject_undefined(vcpu);
- return 1;
-}
-
static void reset_mpidr(struct kvm_vcpu *vcpu, const struct coproc_reg *r)
{
/*
* must always support PMCCNTR (the cycle counter): we just RAZ/WI for
* all PM registers, which doesn't crash the guest kernel at least.
*/
-static bool pm_fake(struct kvm_vcpu *vcpu,
+static bool trap_raz_wi(struct kvm_vcpu *vcpu,
const struct coproc_params *p,
const struct coproc_reg *r)
{
return read_zero(vcpu, p);
}
-#define access_pmcr pm_fake
-#define access_pmcntenset pm_fake
-#define access_pmcntenclr pm_fake
-#define access_pmovsr pm_fake
-#define access_pmselr pm_fake
-#define access_pmceid0 pm_fake
-#define access_pmceid1 pm_fake
-#define access_pmccntr pm_fake
-#define access_pmxevtyper pm_fake
-#define access_pmxevcntr pm_fake
-#define access_pmuserenr pm_fake
-#define access_pmintenset pm_fake
-#define access_pmintenclr pm_fake
+#define access_pmcr trap_raz_wi
+#define access_pmcntenset trap_raz_wi
+#define access_pmcntenclr trap_raz_wi
+#define access_pmovsr trap_raz_wi
+#define access_pmselr trap_raz_wi
+#define access_pmceid0 trap_raz_wi
+#define access_pmceid1 trap_raz_wi
+#define access_pmccntr trap_raz_wi
+#define access_pmxevtyper trap_raz_wi
+#define access_pmxevcntr trap_raz_wi
+#define access_pmuserenr trap_raz_wi
+#define access_pmintenset trap_raz_wi
+#define access_pmintenclr trap_raz_wi
/* Architected CP15 registers.
* CRn denotes the primary register number, but is copied to the CRm in the
return 1;
}
-/**
- * kvm_handle_cp15_64 -- handles a mrrc/mcrr trap on a guest CP15 access
- * @vcpu: The VCPU pointer
- * @run: The kvm_run struct
- */
-int kvm_handle_cp15_64(struct kvm_vcpu *vcpu, struct kvm_run *run)
+static struct coproc_params decode_64bit_hsr(struct kvm_vcpu *vcpu)
{
struct coproc_params params;
params.Rt2 = (kvm_vcpu_get_hsr(vcpu) >> 10) & 0xf;
params.CRm = 0;
+ return params;
+}
+
+/**
+ * kvm_handle_cp15_64 -- handles a mrrc/mcrr trap on a guest CP15 access
+ * @vcpu: The VCPU pointer
+ * @run: The kvm_run struct
+ */
+int kvm_handle_cp15_64(struct kvm_vcpu *vcpu, struct kvm_run *run)
+{
+ struct coproc_params params = decode_64bit_hsr(vcpu);
+
return emulate_cp15(vcpu, ¶ms);
}
+/**
+ * kvm_handle_cp14_64 -- handles a mrrc/mcrr trap on a guest CP14 access
+ * @vcpu: The VCPU pointer
+ * @run: The kvm_run struct
+ */
+int kvm_handle_cp14_64(struct kvm_vcpu *vcpu, struct kvm_run *run)
+{
+ struct coproc_params params = decode_64bit_hsr(vcpu);
+
+ /* raz_wi cp14 */
+ trap_raz_wi(vcpu, ¶ms, NULL);
+
+ /* handled */
+ kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu));
+ return 1;
+}
+
static void reset_coproc_regs(struct kvm_vcpu *vcpu,
const struct coproc_reg *table, size_t num)
{
table[i].reset(vcpu, &table[i]);
}
-/**
- * kvm_handle_cp15_32 -- handles a mrc/mcr trap on a guest CP15 access
- * @vcpu: The VCPU pointer
- * @run: The kvm_run struct
- */
-int kvm_handle_cp15_32(struct kvm_vcpu *vcpu, struct kvm_run *run)
+static struct coproc_params decode_32bit_hsr(struct kvm_vcpu *vcpu)
{
struct coproc_params params;
params.Op2 = (kvm_vcpu_get_hsr(vcpu) >> 17) & 0x7;
params.Rt2 = 0;
+ return params;
+}
+
+/**
+ * kvm_handle_cp15_32 -- handles a mrc/mcr trap on a guest CP15 access
+ * @vcpu: The VCPU pointer
+ * @run: The kvm_run struct
+ */
+int kvm_handle_cp15_32(struct kvm_vcpu *vcpu, struct kvm_run *run)
+{
+ struct coproc_params params = decode_32bit_hsr(vcpu);
return emulate_cp15(vcpu, ¶ms);
}
+/**
+ * kvm_handle_cp14_32 -- handles a mrc/mcr trap on a guest CP14 access
+ * @vcpu: The VCPU pointer
+ * @run: The kvm_run struct
+ */
+int kvm_handle_cp14_32(struct kvm_vcpu *vcpu, struct kvm_run *run)
+{
+ struct coproc_params params = decode_32bit_hsr(vcpu);
+
+ /* raz_wi cp14 */
+ trap_raz_wi(vcpu, ¶ms, NULL);
+
+ /* handled */
+ kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu));
+ return 1;
+}
+
/******************************************************************************
* Userspace API
*****************************************************************************/
[HSR_EC_WFI] = kvm_handle_wfx,
[HSR_EC_CP15_32] = kvm_handle_cp15_32,
[HSR_EC_CP15_64] = kvm_handle_cp15_64,
- [HSR_EC_CP14_MR] = kvm_handle_cp14_access,
+ [HSR_EC_CP14_MR] = kvm_handle_cp14_32,
[HSR_EC_CP14_LS] = kvm_handle_cp14_load_store,
- [HSR_EC_CP14_64] = kvm_handle_cp14_access,
+ [HSR_EC_CP14_64] = kvm_handle_cp14_64,
[HSR_EC_CP_0_13] = kvm_handle_cp_0_13_access,
[HSR_EC_CP10_ID] = kvm_handle_cp10_id,
[HSR_EC_HVC] = handle_hvc,
# Makefile for Kernel-based Virtual Machine module, HYP part
#
+ccflags-y += -fno-stack-protector
+
KVM=../../../../virt/kvm
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o
write_sysreg(HSTR_T(15), HSTR);
write_sysreg(HCPTR_TTA | HCPTR_TCP(10) | HCPTR_TCP(11), HCPTR);
val = read_sysreg(HDCR);
- write_sysreg(val | HDCR_TPM | HDCR_TPMCR, HDCR);
+ val |= HDCR_TPM | HDCR_TPMCR; /* trap performance monitors */
+ val |= HDCR_TDRA | HDCR_TDOSA | HDCR_TDA; /* trap debug regs */
+ write_sysreg(val, HDCR);
}
static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu)
-#if !defined(_TRACE_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
-#define _TRACE_KVM_H
+#if !defined(_TRACE_ARM_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_ARM_KVM_H
#include <linux/tracepoint.h>
__entry->vcpu_pc, __entry->r0, __entry->imm)
);
-#endif /* _TRACE_KVM_H */
+#endif /* _TRACE_ARM_KVM_H */
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH arch/arm/kvm
+#define TRACE_INCLUDE_PATH .
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE trace
{ .idle = sama5d3_ddr_standby, .memctrl = AT91_MEMCTRL_DDRSDR},
};
-static const struct of_device_id const ramc_ids[] __initconst = {
+static const struct of_device_id ramc_ids[] __initconst = {
{ .compatible = "atmel,at91rm9200-sdramc", .data = &ramc_infos[0] },
{ .compatible = "atmel,at91sam9260-sdramc", .data = &ramc_infos[1] },
{ .compatible = "atmel,at91sam9g45-ddramc", .data = &ramc_infos[2] },
unsigned result;
};
-static const struct of_device_id const bcm_kona_smc_ids[] __initconst = {
+static const struct of_device_id bcm_kona_smc_ids[] __initconst = {
{.compatible = "brcm,kona-smc"},
{.compatible = "bcm,kona-smc"}, /* deprecated name */
{},
.power_off = csn3xxx_usb_power_off,
};
-static const struct of_dev_auxdata const cns3xxx_auxdata[] __initconst = {
+static const struct of_dev_auxdata cns3xxx_auxdata[] __initconst = {
{ "intel,usb-ehci", CNS3XXX_USB_BASE, "ehci-platform", &cns3xxx_usb_ehci_pdata },
{ "intel,usb-ohci", CNS3XXX_USB_OHCI_BASE, "ohci-platform", &cns3xxx_usb_ohci_pdata },
{ "cavium,cns3420-ahci", CNS3XXX_SATA2_BASE, "ahci", NULL },
extern const struct smp_operations omap4_smp_ops;
#endif
+extern u32 omap4_get_cpu1_ns_pa_addr(void);
+
#if defined(CONFIG_SMP) && defined(CONFIG_PM)
extern int omap4_mpuss_init(void);
extern int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state);
extern int omap4_hotplug_cpu(unsigned int cpu, unsigned int power_state);
-extern u32 omap4_get_cpu1_ns_pa_addr(void);
#else
static inline int omap4_enter_lowpower(unsigned int cpu,
unsigned int power_state)
{}
#endif
-u32 omap4_get_cpu1_ns_pa_addr(void)
-{
- return old_cpu1_ns_pa_addr;
-}
-
/**
* omap4_enter_lowpower: OMAP4 MPUSS Low Power Entry Function
* The purpose of this function is to manage low power programming
#endif
+u32 omap4_get_cpu1_ns_pa_addr(void)
+{
+ return old_cpu1_ns_pa_addr;
+}
+
/*
* For kexec, we must set CPU1_WAKEUP_NS_PA_ADDR to point to
* current kernel's secondary_startup() early before
cpu1_startup_pa = readl_relaxed(cfg.wakeupgen_base +
OMAP_AUX_CORE_BOOT_1);
- cpu1_ns_pa_addr = omap4_get_cpu1_ns_pa_addr();
/* Did the configured secondary_startup() get overwritten? */
if (!omap4_smp_cpu1_startup_valid(cpu1_startup_pa))
* If omap4 or 5 has NS_PA_ADDR configured, CPU1 may be in a
* deeper idle state in WFI and will wake to an invalid address.
*/
- if ((soc_is_omap44xx() || soc_is_omap54xx()) &&
- !omap4_smp_cpu1_startup_valid(cpu1_ns_pa_addr))
- needs_reset = true;
+ if ((soc_is_omap44xx() || soc_is_omap54xx())) {
+ cpu1_ns_pa_addr = omap4_get_cpu1_ns_pa_addr();
+ if (!omap4_smp_cpu1_startup_valid(cpu1_ns_pa_addr))
+ needs_reset = true;
+ } else {
+ cpu1_ns_pa_addr = 0;
+ }
if (!needs_reset || !c->cpu1_rstctrl_va)
return;
};
#endif
-static const struct of_device_id const omap_prcm_dt_match_table[] __initconst = {
+static const struct of_device_id omap_prcm_dt_match_table[] __initconst = {
#ifdef CONFIG_SOC_AM33XX
{ .compatible = "ti,am3-prcm", .data = &am3_prm_data },
#endif
u8 hsscll_12;
};
-static const struct i2c_init_data const omap4_i2c_timing_data[] __initconst = {
+static const struct i2c_init_data omap4_i2c_timing_data[] __initconst = {
{
.load = 50,
.loadbits = 0x3,
setup_irq(irq, &spear_timer_irq);
}
-static const struct of_device_id const timer_of_match[] __initconst = {
+static const struct of_device_id timer_of_match[] __initconst = {
{ .compatible = "st,spear-timer", },
{ },
};
select ARMADA_AP806_SYSCON
select ARMADA_CP110_SYSCON
select ARMADA_37XX_CLK
+ select GPIOLIB
+ select GPIOLIB_IRQCHIP
select MVEBU_ODMI
select MVEBU_PIC
+ select OF_GPIO
+ select PINCTRL
+ select PINCTRL_ARMADA_37XX
help
This enables support for Marvell EBU familly, including:
- Armada 3700 SoC Family
};
};
+ reg_sys_5v: regulator@0 {
+ compatible = "regulator-fixed";
+ regulator-name = "SYS_5V";
+ regulator-min-microvolt = <5000000>;
+ regulator-max-microvolt = <5000000>;
+ regulator-boot-on;
+ regulator-always-on;
+ };
+
+ reg_vdd_3v3: regulator@1 {
+ compatible = "regulator-fixed";
+ regulator-name = "VDD_3V3";
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ regulator-boot-on;
+ regulator-always-on;
+ vin-supply = <®_sys_5v>;
+ };
+
+ reg_5v_hub: regulator@2 {
+ compatible = "regulator-fixed";
+ regulator-name = "5V_HUB";
+ regulator-min-microvolt = <5000000>;
+ regulator-max-microvolt = <5000000>;
+ regulator-boot-on;
+ gpio = <&gpio0 7 0>;
+ regulator-always-on;
+ vin-supply = <®_sys_5v>;
+ };
+
+ wl1835_pwrseq: wl1835-pwrseq {
+ compatible = "mmc-pwrseq-simple";
+ /* WLAN_EN GPIO */
+ reset-gpios = <&gpio0 5 GPIO_ACTIVE_LOW>;
+ clocks = <&pmic>;
+ clock-names = "ext_clock";
+ power-off-delay-us = <10>;
+ };
+
soc {
spi0: spi@f7106000 {
status = "ok";
/* GPIO blocks 16 thru 19 do not appear to be routed to pins */
+ dwmmc_0: dwmmc0@f723d000 {
+ cap-mmc-highspeed;
+ non-removable;
+ bus-width = <0x8>;
+ vmmc-supply = <&ldo19>;
+ };
+
+ dwmmc_1: dwmmc1@f723e000 {
+ card-detect-delay = <200>;
+ cap-sd-highspeed;
+ sd-uhs-sdr12;
+ sd-uhs-sdr25;
+ sd-uhs-sdr50;
+ vqmmc-supply = <&ldo7>;
+ vmmc-supply = <&ldo10>;
+ bus-width = <0x4>;
+ disable-wp;
+ cd-gpios = <&gpio1 0 1>;
+ };
+
dwmmc_2: dwmmc2@f723f000 {
- ti,non-removable;
+ bus-width = <0x4>;
non-removable;
- /* WL_EN */
- vmmc-supply = <&wlan_en_reg>;
+ vmmc-supply = <®_vdd_3v3>;
+ mmc-pwrseq = <&wl1835_pwrseq>;
#address-cells = <0x1>;
#size-cells = <0x0>;
interrupts = <3 IRQ_TYPE_EDGE_RISING>;
};
};
-
- wlan_en_reg: regulator@1 {
- compatible = "regulator-fixed";
- regulator-name = "wlan-en-regulator";
- regulator-min-microvolt = <1800000>;
- regulator-max-microvolt = <1800000>;
- /* WLAN_EN GPIO */
- gpio = <&gpio0 5 0>;
- /* WLAN card specific delay */
- startup-delay-us = <70000>;
- enable-active-high;
- };
};
leds {
pmic: pmic@f8000000 {
compatible = "hisilicon,hi655x-pmic";
reg = <0x0 0xf8000000 0x0 0x1000>;
+ #clock-cells = <0>;
interrupt-controller;
#interrupt-cells = <2>;
pmic-gpios = <&gpio1 2 GPIO_ACTIVE_HIGH>;
status = "disabled";
};
- fixed_5v_hub: regulator@0 {
- compatible = "regulator-fixed";
- regulator-name = "fixed_5v_hub";
- regulator-min-microvolt = <5000000>;
- regulator-max-microvolt = <5000000>;
- regulator-boot-on;
- gpio = <&gpio0 7 0>;
- regulator-always-on;
- };
-
usb_phy: usbphy {
compatible = "hisilicon,hi6220-usb-phy";
#phy-cells = <0>;
- phy-supply = <&fixed_5v_hub>;
+ phy-supply = <®_5v_hub>;
hisilicon,peripheral-syscon = <&sys_ctrl>;
};
dwmmc_0: dwmmc0@f723d000 {
compatible = "hisilicon,hi6220-dw-mshc";
- num-slots = <0x1>;
- cap-mmc-highspeed;
- non-removable;
reg = <0x0 0xf723d000 0x0 0x1000>;
interrupts = <0x0 0x48 0x4>;
clocks = <&sys_ctrl 2>, <&sys_ctrl 1>;
clock-names = "ciu", "biu";
resets = <&sys_ctrl PERIPH_RSTDIS0_MMC0>;
reset-names = "reset";
- bus-width = <0x8>;
- vmmc-supply = <&ldo19>;
pinctrl-names = "default";
pinctrl-0 = <&emmc_pmx_func &emmc_clk_cfg_func
&emmc_cfg_func &emmc_rst_cfg_func>;
dwmmc_1: dwmmc1@f723e000 {
compatible = "hisilicon,hi6220-dw-mshc";
- num-slots = <0x1>;
- card-detect-delay = <200>;
hisilicon,peripheral-syscon = <&ao_ctrl>;
- cap-sd-highspeed;
- sd-uhs-sdr12;
- sd-uhs-sdr25;
- sd-uhs-sdr50;
reg = <0x0 0xf723e000 0x0 0x1000>;
interrupts = <0x0 0x49 0x4>;
#address-cells = <0x1>;
clock-names = "ciu", "biu";
resets = <&sys_ctrl PERIPH_RSTDIS0_MMC1>;
reset-names = "reset";
- vqmmc-supply = <&ldo7>;
- vmmc-supply = <&ldo10>;
- bus-width = <0x4>;
- disable-wp;
- cd-gpios = <&gpio1 0 1>;
pinctrl-names = "default", "idle";
pinctrl-0 = <&sd_pmx_func &sd_clk_cfg_func &sd_cfg_func>;
pinctrl-1 = <&sd_pmx_idle &sd_clk_cfg_idle &sd_cfg_idle>;
dwmmc_2: dwmmc2@f723f000 {
compatible = "hisilicon,hi6220-dw-mshc";
- num-slots = <0x1>;
reg = <0x0 0xf723f000 0x0 0x1000>;
interrupts = <0x0 0x4a 0x4>;
clocks = <&sys_ctrl HI6220_MMC2_CIUCLK>, <&sys_ctrl HI6220_MMC2_CLK>;
clock-names = "ciu", "biu";
resets = <&sys_ctrl PERIPH_RSTDIS0_MMC2>;
reset-names = "reset";
- bus-width = <0x4>;
- broken-cd;
pinctrl-names = "default", "idle";
pinctrl-0 = <&sdio_pmx_func &sdio_clk_cfg_func &sdio_cfg_func>;
pinctrl-1 = <&sdio_pmx_idle &sdio_clk_cfg_idle &sdio_cfg_idle>;
+++ /dev/null
-../../../../arm/boot/dts
\ No newline at end of file
+++ /dev/null
-..
\ No newline at end of file
+++ /dev/null
-../../../../../include/dt-bindings
\ No newline at end of file
};
&i2c0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&i2c1_pins>;
status = "okay";
gpio_exp: pca9555@22 {
&spi0 {
status = "okay";
+ pinctrl-names = "default";
+ pinctrl-0 = <&spi_quad_pins>;
m25p80@0 {
compatible = "jedec,spi-nor";
/* Exported on the micro USB connector CON32 through an FTDI */
&uart0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&uart1_pins>;
status = "okay";
};
};
ð0 {
+ pinctrl-names = "default";
+ pinctrl-0 = <&rgmii_pins>;
phy-mode = "rgmii-id";
phy = <&phy0>;
status = "okay";
#clock-cells = <1>;
};
- gpio1: gpio@13800 {
- compatible = "marvell,mvebu-gpio-3700",
+ pinctrl_nb: pinctrl@13800 {
+ compatible = "marvell,armada3710-nb-pinctrl",
"syscon", "simple-mfd";
- reg = <0x13800 0x500>;
+ reg = <0x13800 0x100>, <0x13C00 0x20>;
+ gpionb: gpio {
+ #gpio-cells = <2>;
+ gpio-ranges = <&pinctrl_nb 0 0 36>;
+ gpio-controller;
+ interrupts =
+ <GIC_SPI 51 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 52 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 53 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 54 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 56 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 57 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 58 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 152 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 153 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 154 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 155 IRQ_TYPE_LEVEL_HIGH>;
+
+ };
xtalclk: xtal-clk {
compatible = "marvell,armada-3700-xtal-clock";
clock-output-names = "xtal";
#clock-cells = <0>;
};
+
+ spi_quad_pins: spi-quad-pins {
+ groups = "spi_quad";
+ function = "spi";
+ };
+
+ i2c1_pins: i2c1-pins {
+ groups = "i2c1";
+ function = "i2c";
+ };
+
+ i2c2_pins: i2c2-pins {
+ groups = "i2c2";
+ function = "i2c";
+ };
+
+ uart1_pins: uart1-pins {
+ groups = "uart1";
+ function = "uart";
+ };
+
+ uart2_pins: uart2-pins {
+ groups = "uart2";
+ function = "uart";
+ };
+ };
+
+ pinctrl_sb: pinctrl@18800 {
+ compatible = "marvell,armada3710-sb-pinctrl",
+ "syscon", "simple-mfd";
+ reg = <0x18800 0x100>, <0x18C00 0x20>;
+ gpiosb: gpio {
+ #gpio-cells = <2>;
+ gpio-ranges = <&pinctrl_sb 0 0 29>;
+ gpio-controller;
+ interrupts =
+ <GIC_SPI 160 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 159 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 158 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 157 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 156 IRQ_TYPE_LEVEL_HIGH>;
+ };
+
+ rgmii_pins: mii-pins {
+ groups = "rgmii";
+ function = "mii";
+ };
+
};
eth0: ethernet@30000 {
bus-width = <8>;
max-frequency = <50000000>;
cap-mmc-highspeed;
+ mediatek,hs200-cmd-int-delay=<26>;
+ mediatek,hs400-cmd-int-delay=<14>;
+ mediatek,hs400-cmd-resp-sel-rising;
vmmc-supply = <&mt6397_vemc_3v3_reg>;
vqmmc-supply = <&mt6397_vio18_reg>;
non-removable;
/dts-v1/;
#include "rk3399-gru.dtsi"
-#include <include/dt-bindings/input/linux-event-codes.h>
+#include <dt-bindings/input/linux-event-codes.h>
/*
* Kevin-specific things
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
-# CONFIG_BLK_DEV_BSG is not set
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_ARCH_SUNXI=y
CONFIG_ARCH_ALPINE=y
CONFIG_ARCH_ZX=y
CONFIG_ARCH_ZYNQMP=y
CONFIG_PCI=y
-CONFIG_PCI_MSI=y
CONFIG_PCI_IOV=y
-CONFIG_PCI_AARDVARK=y
-CONFIG_PCIE_RCAR=y
-CONFIG_PCI_HOST_GENERIC=y
-CONFIG_PCI_XGENE=y
CONFIG_PCI_LAYERSCAPE=y
CONFIG_PCI_HISI=y
CONFIG_PCIE_QCOM=y
CONFIG_PCIE_ARMADA_8K=y
+CONFIG_PCI_AARDVARK=y
+CONFIG_PCIE_RCAR=y
+CONFIG_PCI_HOST_GENERIC=y
+CONFIG_PCI_XGENE=y
CONFIG_ARM64_VA_BITS_48=y
CONFIG_SCHED_MC=y
CONFIG_NUMA=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_CMA=y
CONFIG_SECCOMP=y
-CONFIG_XEN=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
+CONFIG_XEN=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_COMPAT=y
-CONFIG_CPU_IDLE=y
CONFIG_HIBERNATION=y
CONFIG_ARM_CPUIDLE=y
CONFIG_CPU_FREQ=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_NBD=m
CONFIG_VIRTIO_BLK=y
-CONFIG_EEPROM_AT25=m
CONFIG_SRAM=y
+CONFIG_EEPROM_AT25=m
# CONFIG_SCSI_PROC_FS is not set
CONFIG_BLK_DEV_SD=y
CONFIG_SCSI_SAS_ATA=y
CONFIG_AHCI_MVEBU=y
CONFIG_AHCI_XGENE=y
CONFIG_AHCI_QORIQ=y
-CONFIG_SATA_RCAR=y
CONFIG_SATA_SIL24=y
+CONFIG_SATA_RCAR=y
CONFIG_PATA_PLATFORM=y
CONFIG_PATA_OF_PLATFORM=y
CONFIG_NETDEVICES=y
CONFIG_E1000E=y
CONFIG_IGB=y
CONFIG_IGBVF=y
-CONFIG_MVPP2=y
CONFIG_MVNETA=y
+CONFIG_MVPP2=y
CONFIG_SKY2=y
CONFIG_RAVB=y
CONFIG_SMC91X=y
CONFIG_SMSC911X=y
CONFIG_STMMAC_ETH=m
-CONFIG_REALTEK_PHY=m
+CONFIG_MDIO_BUS_MUX_MMIOREG=y
CONFIG_MESON_GXL_PHY=m
CONFIG_MICREL_PHY=y
-CONFIG_MDIO_BUS_MUX=y
-CONFIG_MDIO_BUS_MUX_MMIOREG=y
+CONFIG_REALTEK_PHY=m
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=m
CONFIG_USB_RTL8152=m
CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
+CONFIG_SERIAL_MESON=y
+CONFIG_SERIAL_MESON_CONSOLE=y
CONFIG_SERIAL_SAMSUNG=y
CONFIG_SERIAL_SAMSUNG_CONSOLE=y
CONFIG_SERIAL_TEGRA=y
CONFIG_SERIAL_SH_SCI=y
CONFIG_SERIAL_SH_SCI_NR_UARTS=11
CONFIG_SERIAL_SH_SCI_CONSOLE=y
-CONFIG_SERIAL_MESON=y
-CONFIG_SERIAL_MESON_CONSOLE=y
CONFIG_SERIAL_MSM=y
CONFIG_SERIAL_MSM_CONSOLE=y
CONFIG_SERIAL_XILINX_PS_UART=y
CONFIG_I2C_RCAR=y
CONFIG_I2C_CROS_EC_TUNNEL=y
CONFIG_SPI=y
-CONFIG_SPI_MESON_SPIFC=m
CONFIG_SPI_BCM2835=m
CONFIG_SPI_BCM2835AUX=m
+CONFIG_SPI_MESON_SPIFC=m
CONFIG_SPI_ORION=y
CONFIG_SPI_PL022=y
CONFIG_SPI_QUP=y
-CONFIG_SPI_SPIDEV=m
CONFIG_SPI_S3C64XX=y
+CONFIG_SPI_SPIDEV=m
CONFIG_SPMI=y
CONFIG_PINCTRL_SINGLE=y
CONFIG_PINCTRL_MAX77620=y
CONFIG_GPIO_PCA953X_IRQ=y
CONFIG_GPIO_MAX77620=y
CONFIG_POWER_RESET_MSM=y
-CONFIG_BATTERY_BQ27XXX=y
CONFIG_POWER_RESET_XGENE=y
CONFIG_POWER_RESET_SYSCON=y
+CONFIG_BATTERY_BQ27XXX=y
+CONFIG_SENSORS_ARM_SCPI=y
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_INA2XX=m
-CONFIG_SENSORS_ARM_SCPI=y
-CONFIG_THERMAL=y
-CONFIG_THERMAL_EMULATION=y
CONFIG_THERMAL_GOV_POWER_ALLOCATOR=y
CONFIG_CPU_THERMAL=y
-CONFIG_BCM2835_THERMAL=y
+CONFIG_THERMAL_EMULATION=y
CONFIG_EXYNOS_THERMAL=y
CONFIG_WATCHDOG=y
-CONFIG_BCM2835_WDT=y
-CONFIG_RENESAS_WDT=y
CONFIG_S3C2410_WATCHDOG=y
CONFIG_MESON_GXBB_WATCHDOG=m
CONFIG_MESON_WATCHDOG=m
+CONFIG_RENESAS_WDT=y
+CONFIG_BCM2835_WDT=y
+CONFIG_MFD_CROS_EC=y
+CONFIG_MFD_CROS_EC_I2C=y
CONFIG_MFD_EXYNOS_LPASS=m
+CONFIG_MFD_HI655X_PMIC=y
CONFIG_MFD_MAX77620=y
-CONFIG_MFD_RK808=y
CONFIG_MFD_SPMI_PMIC=y
+CONFIG_MFD_RK808=y
CONFIG_MFD_SEC_CORE=y
-CONFIG_MFD_HI655X_PMIC=y
-CONFIG_REGULATOR=y
-CONFIG_MFD_CROS_EC=y
-CONFIG_MFD_CROS_EC_I2C=y
CONFIG_REGULATOR_FIXED_VOLTAGE=y
CONFIG_REGULATOR_GPIO=y
CONFIG_REGULATOR_HI655X=y
CONFIG_DRM_EXYNOS_HDMI=y
CONFIG_DRM_EXYNOS_MIC=y
CONFIG_DRM_RCAR_DU=m
-CONFIG_DRM_RCAR_HDMI=y
CONFIG_DRM_RCAR_LVDS=y
CONFIG_DRM_RCAR_VSP=y
CONFIG_DRM_TEGRA=m
-CONFIG_DRM_VC4=m
CONFIG_DRM_PANEL_SIMPLE=m
CONFIG_DRM_I2C_ADV7511=m
+CONFIG_DRM_VC4=m
CONFIG_DRM_HISI_KIRIN=m
CONFIG_DRM_MESON=m
CONFIG_FB=y
CONFIG_SND=y
CONFIG_SND_SOC=y
CONFIG_SND_BCM2835_SOC_I2S=m
-CONFIG_SND_SOC_RCAR=y
CONFIG_SND_SOC_SAMSUNG=y
+CONFIG_SND_SOC_RCAR=y
CONFIG_SND_SOC_AK4613=y
CONFIG_USB=y
CONFIG_USB_OTG=y
CONFIG_USB_XHCI_HCD=y
-CONFIG_USB_XHCI_PLATFORM=y
-CONFIG_USB_XHCI_RCAR=y
-CONFIG_USB_EHCI_EXYNOS=y
CONFIG_USB_XHCI_TEGRA=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_MSM=y
+CONFIG_USB_EHCI_EXYNOS=y
CONFIG_USB_EHCI_HCD_PLATFORM=y
-CONFIG_USB_OHCI_EXYNOS=y
CONFIG_USB_OHCI_HCD=y
+CONFIG_USB_OHCI_EXYNOS=y
CONFIG_USB_OHCI_HCD_PLATFORM=y
CONFIG_USB_RENESAS_USBHS=m
CONFIG_USB_STORAGE=y
-CONFIG_USB_DWC2=y
CONFIG_USB_DWC3=y
+CONFIG_USB_DWC2=y
CONFIG_USB_CHIPIDEA=y
CONFIG_USB_CHIPIDEA_UDC=y
CONFIG_USB_CHIPIDEA_HOST=y
CONFIG_USB_ISP1760=y
CONFIG_USB_HSIC_USB3503=y
CONFIG_USB_MSM_OTG=y
+CONFIG_USB_QCOM_8X16_PHY=y
CONFIG_USB_ULPI=y
CONFIG_USB_GADGET=y
CONFIG_USB_RENESAS_USBHS_UDC=m
CONFIG_MMC=y
CONFIG_MMC_BLOCK_MINORS=32
CONFIG_MMC_ARMMMCI=y
-CONFIG_MMC_MESON_GX=y
CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_ACPI=y
CONFIG_MMC_SDHCI_PLTFM=y
CONFIG_MMC_SDHCI_OF_ESDHC=y
CONFIG_MMC_SDHCI_CADENCE=y
CONFIG_MMC_SDHCI_TEGRA=y
+CONFIG_MMC_MESON_GX=y
CONFIG_MMC_SDHCI_MSM=y
CONFIG_MMC_SPI=y
CONFIG_MMC_SDHI=y
CONFIG_MMC_DW_K3=y
CONFIG_MMC_DW_ROCKCHIP=y
CONFIG_MMC_SUNXI=y
-CONFIG_MMC_SDHCI_XENON=y
CONFIG_MMC_BCM2835=y
+CONFIG_MMC_SDHCI_XENON=y
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
CONFIG_LEDS_GPIO=y
CONFIG_LEDS_PWM=y
CONFIG_LEDS_SYSCON=y
-CONFIG_LEDS_TRIGGERS=y
-CONFIG_LEDS_TRIGGER_DEFAULT_ON=y
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
CONFIG_LEDS_TRIGGER_CPU=y
+CONFIG_LEDS_TRIGGER_DEFAULT_ON=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_DRV_MAX77686=y
+CONFIG_RTC_DRV_RK808=m
CONFIG_RTC_DRV_S5M=y
CONFIG_RTC_DRV_DS3232=y
CONFIG_RTC_DRV_EFI=y
+CONFIG_RTC_DRV_S3C=y
CONFIG_RTC_DRV_PL031=y
CONFIG_RTC_DRV_SUN6I=y
-CONFIG_RTC_DRV_RK808=m
CONFIG_RTC_DRV_TEGRA=y
CONFIG_RTC_DRV_XGENE=y
-CONFIG_RTC_DRV_S3C=y
CONFIG_DMADEVICES=y
+CONFIG_DMA_BCM2835=m
CONFIG_MV_XOR_V2=y
CONFIG_PL330_DMA=y
-CONFIG_DMA_BCM2835=m
CONFIG_TEGRA20_APB_DMA=y
CONFIG_QCOM_BAM_DMA=y
CONFIG_QCOM_HIDMA_MGMT=y
CONFIG_VIRTIO_MMIO=y
CONFIG_XEN_GNTDEV=y
CONFIG_XEN_GRANT_DEV_ALLOC=y
+CONFIG_COMMON_CLK_RK808=y
CONFIG_COMMON_CLK_SCPI=y
CONFIG_COMMON_CLK_CS2000_CP=y
CONFIG_COMMON_CLK_S2MPS11=y
-CONFIG_COMMON_CLK_PWM=y
-CONFIG_COMMON_CLK_RK808=y
CONFIG_CLK_QORIQ=y
+CONFIG_COMMON_CLK_PWM=y
CONFIG_COMMON_CLK_QCOM=y
+CONFIG_QCOM_CLK_SMD_RPM=y
CONFIG_MSM_GCC_8916=y
CONFIG_MSM_GCC_8994=y
CONFIG_MSM_MMCC_8996=y
CONFIG_HWSPINLOCK_QCOM=y
-CONFIG_MAILBOX=y
CONFIG_ARM_MHU=y
CONFIG_PLATFORM_MHU=y
CONFIG_BCM2835_MBOX=y
CONFIG_HI6220_MBOX=y
CONFIG_ARM_SMMU=y
CONFIG_ARM_SMMU_V3=y
+CONFIG_RPMSG_QCOM_SMD=y
CONFIG_RASPBERRYPI_POWER=y
CONFIG_QCOM_SMEM=y
-CONFIG_QCOM_SMD=y
CONFIG_QCOM_SMD_RPM=y
+CONFIG_QCOM_SMP2P=y
+CONFIG_QCOM_SMSM=y
CONFIG_ROCKCHIP_PM_DOMAINS=y
CONFIG_ARCH_TEGRA_132_SOC=y
CONFIG_ARCH_TEGRA_210_SOC=y
CONFIG_ARCH_TEGRA_186_SOC=y
CONFIG_EXTCON_USB_GPIO=y
+CONFIG_IIO=y
+CONFIG_EXYNOS_ADC=y
CONFIG_PWM=y
CONFIG_PWM_BCM2835=m
+CONFIG_PWM_MESON=m
CONFIG_PWM_ROCKCHIP=y
+CONFIG_PWM_SAMSUNG=y
CONFIG_PWM_TEGRA=m
-CONFIG_PWM_MESON=m
-CONFIG_COMMON_RESET_HI6220=y
CONFIG_PHY_RCAR_GEN3_USB2=y
CONFIG_PHY_HI6220_USB=y
+CONFIG_PHY_SUN4I_USB=y
CONFIG_PHY_ROCKCHIP_INNO_USB2=y
CONFIG_PHY_ROCKCHIP_EMMC=y
-CONFIG_PHY_SUN4I_USB=y
CONFIG_PHY_XGENE=y
CONFIG_PHY_TEGRA_XUSB=y
CONFIG_ARM_SCPI_PROTOCOL=y
-CONFIG_ACPI=y
-CONFIG_IIO=y
-CONFIG_EXYNOS_ADC=y
-CONFIG_PWM_SAMSUNG=y
CONFIG_RASPBERRYPI_FIRMWARE=y
+CONFIG_ACPI=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_CUSE=m
CONFIG_OVERLAY_FS=m
CONFIG_VFAT_FS=y
-CONFIG_TMPFS=y
CONFIG_HUGETLBFS=y
CONFIG_CONFIGFS_FS=y
CONFIG_EFIVAR_FS=y
CONFIG_SECURITY=y
CONFIG_CRYPTO_ECHAINIV=y
CONFIG_CRYPTO_ANSI_CPRNG=y
-CONFIG_CRYPTO_DEV_SAFEXCEL=m
CONFIG_ARM64_CRYPTO=y
CONFIG_CRYPTO_SHA1_ARM64_CE=y
CONFIG_CRYPTO_SHA2_ARM64_CE=y
CONFIG_CRYPTO_GHASH_ARM64_CE=y
CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
-# CONFIG_CRYPTO_AES_ARM64_NEON_BLK is not set
" st" #rel "xr" #sz "\t%w[tmp], %" #w "[new], %[v]\n" \
" cbnz %w[tmp], 1b\n" \
" " #mb "\n" \
- " mov %" #w "[oldval], %" #w "[old]\n" \
"2:" \
: [tmp] "=&r" (tmp), [oldval] "=&r" (oldval), \
[v] "+Q" (*(unsigned long *)ptr) \
extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS];
+extern struct static_key_false arm64_const_caps_ready;
bool this_cpu_has_cap(unsigned int cap);
}
/* System capability check for constant caps */
-static inline bool cpus_have_const_cap(int num)
+static inline bool __cpus_have_const_cap(int num)
{
if (num >= ARM64_NCAPS)
return false;
return test_bit(num, cpu_hwcaps);
}
+static inline bool cpus_have_const_cap(int num)
+{
+ if (static_branch_likely(&arm64_const_caps_ready))
+ return __cpus_have_const_cap(num);
+ else
+ return cpus_have_cap(num);
+}
+
static inline void cpus_set_cap(unsigned int num)
{
if (num >= ARM64_NCAPS) {
num, ARM64_NCAPS);
} else {
__set_bit(num, cpu_hwcaps);
- static_branch_enable(&cpu_hwcap_keys[num]);
}
}
#include <linux/types.h>
#include <linux/kvm_types.h>
+#include <asm/cpufeature.h>
#include <asm/kvm.h>
#include <asm/kvm_asm.h>
#include <asm/kvm_mmio.h>
unsigned long vector_ptr)
{
/*
- * Call initialization code, and switch to the full blown
- * HYP code.
+ * Call initialization code, and switch to the full blown HYP code.
+ * If the cpucaps haven't been finalized yet, something has gone very
+ * wrong, and hyp will crash and burn when it uses any
+ * cpus_have_const_cap() wrapper.
*/
+ BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
__kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
}
*/
void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps)
{
- for (; caps->matches; caps++)
- if (caps->enable && cpus_have_cap(caps->capability))
+ for (; caps->matches; caps++) {
+ unsigned int num = caps->capability;
+
+ if (!cpus_have_cap(num))
+ continue;
+
+ /* Ensure cpus_have_const_cap(num) works */
+ static_branch_enable(&cpu_hwcap_keys[num]);
+
+ if (caps->enable) {
/*
* Use stop_machine() as it schedules the work allowing
* us to modify PSTATE, instead of on_each_cpu() which
* we return.
*/
stop_machine(caps->enable, NULL, cpu_online_mask);
+ }
+ }
}
/*
enable_cpu_capabilities(arm64_features);
}
+DEFINE_STATIC_KEY_FALSE(arm64_const_caps_ready);
+EXPORT_SYMBOL(arm64_const_caps_ready);
+
+static void __init mark_const_caps_ready(void)
+{
+ static_branch_enable(&arm64_const_caps_ready);
+}
+
/*
* Check if the current CPU has a given feature capability.
* Should be called from non-preemptible context.
/* Set the CPU feature capabilies */
setup_feature_capabilities();
enable_errata_workarounds();
+ mark_const_caps_ready();
setup_elf_hwcaps(arm64_elf_hwcaps);
if (system_supports_32bit_el0())
if (attr->exclude_idle)
return -EPERM;
- if (is_kernel_in_hyp_mode() &&
- attr->exclude_kernel != attr->exclude_hv)
- return -EINVAL;
+
+ /*
+ * If we're running in hyp mode, then we *are* the hypervisor.
+ * Therefore we ignore exclude_hv in this configuration, since
+ * there's no hypervisor to sample anyway. This is consistent
+ * with other architectures (x86 and Power).
+ */
+ if (is_kernel_in_hyp_mode()) {
+ if (!attr->exclude_kernel)
+ config_base |= ARMV8_PMU_INCLUDE_EL2;
+ } else {
+ if (attr->exclude_kernel)
+ config_base |= ARMV8_PMU_EXCLUDE_EL1;
+ if (!attr->exclude_hv)
+ config_base |= ARMV8_PMU_INCLUDE_EL2;
+ }
if (attr->exclude_user)
config_base |= ARMV8_PMU_EXCLUDE_EL0;
- if (!is_kernel_in_hyp_mode() && attr->exclude_kernel)
- config_base |= ARMV8_PMU_EXCLUDE_EL1;
- if (!attr->exclude_hv)
- config_base |= ARMV8_PMU_INCLUDE_EL2;
/*
* Install the filter into config_base as this is used to
# Makefile for Kernel-based Virtual Machine module, HYP part
#
+ccflags-y += -fno-stack-protector
+
KVM=../../../../virt/kvm
obj-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/hyp/vgic-v2-sr.o
*/
off = offsetof(struct bpf_array, ptrs);
emit_a64_mov_i64(tmp, off, ctx);
- emit(A64_LDR64(tmp, r2, tmp), ctx);
- emit(A64_LDR64(prg, tmp, r3), ctx);
+ emit(A64_ADD(1, tmp, r2, tmp), ctx);
+ emit(A64_LSL(1, prg, r3, 3), ctx);
+ emit(A64_LDR64(prg, tmp, prg), ctx);
emit(A64_CBZ(1, prg, jmp_offset), ctx);
/* goto *(prog->bpf_func + prologue_size); */
+++ /dev/null
-../../../../../include/dt-bindings
\ No newline at end of file
+++ /dev/null
-../../../../../include/dt-bindings
\ No newline at end of file
+++ /dev/null
-../../../../../include/dt-bindings
\ No newline at end of file
+++ /dev/null
-../../../../../include/dt-bindings
\ No newline at end of file
#include <asm-generic/module.h>
+#ifdef CC_USING_MPROFILE_KERNEL
+#define MODULE_ARCH_VERMAGIC "mprofile-kernel"
+#endif
+
#ifndef __powerpc64__
/*
* Thanks to Paul M for explaining this.
#define virt_to_pfn(kaddr) (__pa(kaddr) >> PAGE_SHIFT)
#define virt_to_page(kaddr) pfn_to_page(virt_to_pfn(kaddr))
#define pfn_to_kaddr(pfn) __va((pfn) << PAGE_SHIFT)
+
+#ifdef CONFIG_PPC_BOOK3S_64
+/*
+ * On hash the vmalloc and other regions alias to the kernel region when passed
+ * through __pa(), which virt_to_pfn() uses. That means virt_addr_valid() can
+ * return true for some vmalloc addresses, which is incorrect. So explicitly
+ * check that the address is in the kernel region.
+ */
+#define virt_addr_valid(kaddr) (REGION_ID(kaddr) == KERNEL_REGION_ID && \
+ pfn_valid(virt_to_pfn(kaddr)))
+#else
#define virt_addr_valid(kaddr) pfn_valid(virt_to_pfn(kaddr))
+#endif
/*
* On Book-E parts we need __va to parse the device tree and we can't
#define PPC_FEATURE2_HTM_NOSC 0x01000000
#define PPC_FEATURE2_ARCH_3_00 0x00800000 /* ISA 3.00 */
#define PPC_FEATURE2_HAS_IEEE128 0x00400000 /* VSX IEEE Binary Float 128-bit */
+#define PPC_FEATURE2_DARN 0x00200000 /* darn random number insn */
+#define PPC_FEATURE2_SCV 0x00100000 /* scv syscall */
/*
* IMPORTANT!
#define COMMON_USER_POWER9 COMMON_USER_POWER8
#define COMMON_USER2_POWER9 (COMMON_USER2_POWER8 | \
PPC_FEATURE2_ARCH_3_00 | \
- PPC_FEATURE2_HAS_IEEE128)
+ PPC_FEATURE2_HAS_IEEE128 | \
+ PPC_FEATURE2_DARN )
#ifdef CONFIG_PPC_BOOK3E_64
#define COMMON_USER_BOOKE (COMMON_USER_PPC64 | PPC_FEATURE_BOOKE)
* which needs to be restored from the stack.
*/
li r3, 1
- stb r0,PACA_NAPSTATELOST(r13)
+ stb r3,PACA_NAPSTATELOST(r13)
blr
/*
save_previous_kprobe(kcb);
set_current_kprobe(p, regs, kcb);
kprobes_inc_nmissed_count(p);
- prepare_singlestep(p, regs);
kcb->kprobe_status = KPROBE_REENTER;
if (p->ainsn.boostable >= 0) {
ret = try_to_emulate(p, regs);
if (ret > 0) {
restore_previous_kprobe(kcb);
+ preempt_enable_no_resched();
return 1;
}
}
+ prepare_singlestep(p, regs);
return 1;
} else {
if (*addr != BREAKPOINT_INSTRUCTION) {
if (!MSR_TM_SUSPENDED(mfmsr()))
return;
+ /*
+ * If we are in a transaction and FP is off then we can't have
+ * used FP inside that transaction. Hence the checkpointed
+ * state is the same as the live state. We need to copy the
+ * live state to the checkpointed state so that when the
+ * transaction is restored, the checkpointed state is correct
+ * and the aborted transaction sees the correct state. We use
+ * ckpt_regs.msr here as that's what tm_reclaim will use to
+ * determine if it's going to write the checkpointed state or
+ * not. So either this will write the checkpointed registers,
+ * or reclaim will. Similarly for VMX.
+ */
+ if ((thr->ckpt_regs.msr & MSR_FP) == 0)
+ memcpy(&thr->ckfp_state, &thr->fp_state,
+ sizeof(struct thread_fp_state));
+ if ((thr->ckpt_regs.msr & MSR_VEC) == 0)
+ memcpy(&thr->ckvr_state, &thr->vr_state,
+ sizeof(struct thread_vr_state));
+
giveup_all(container_of(thr, struct task_struct, thread));
tm_reclaim(thr, thr->ckpt_regs.msr, cause);
{ .pabyte = 0, .pabit = 3, .cpu_features = CPU_FTR_CTRL },
{ .pabyte = 0, .pabit = 6, .cpu_features = CPU_FTR_NOEXECUTE },
{ .pabyte = 1, .pabit = 2, .mmu_features = MMU_FTR_CI_LARGE_PAGE },
+#ifdef CONFIG_PPC_RADIX_MMU
{ .pabyte = 40, .pabit = 0, .mmu_features = MMU_FTR_TYPE_RADIX },
+#endif
{ .pabyte = 1, .pabit = 1, .invert = 1, .cpu_features = CPU_FTR_NODSISRALIGN },
{ .pabyte = 5, .pabit = 0, .cpu_features = CPU_FTR_REAL_LE,
.cpu_user_ftrs = PPC_FEATURE_TRUE_LE },
select KVM_BOOK3S_64_HANDLER
select KVM
select KVM_BOOK3S_PR_POSSIBLE if !KVM_BOOK3S_HV_POSSIBLE
- select SPAPR_TCE_IOMMU if IOMMU_SUPPORT
+ select SPAPR_TCE_IOMMU if IOMMU_SUPPORT && (PPC_SERIES || PPC_POWERNV)
---help---
Support running unmodified book3s_64 and book3s_32 guest kernels
in virtual machines on book3s_64 host processors.
e500_emulate.o
kvm-objs-$(CONFIG_KVM_E500MC) := $(kvm-e500mc-objs)
-kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) := \
+kvm-book3s_64-builtin-objs-$(CONFIG_SPAPR_TCE_IOMMU) := \
book3s_64_vio_hv.o
kvm-pr-y := \
book3s_xics.o
kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
+kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
kvm-book3s_64-module-objs := \
$(common-objs-y) \
book3s.o \
- book3s_64_vio.o \
book3s_rtas.o \
$(kvm-book3s_64-objs-y)
/* udbg_printf("H_PUT_TCE(): liobn=0x%lx ioba=0x%lx, tce=0x%lx\n", */
/* liobn, ioba, tce); */
+ /* For radix, we might be in virtual mode, so punt */
+ if (kvm_is_radix(vcpu->kvm))
+ return H_TOO_HARD;
+
stt = kvmppc_find_table(vcpu->kvm, liobn);
if (!stt)
return H_TOO_HARD;
bool prereg = false;
struct kvmppc_spapr_tce_iommu_table *stit;
+ /* For radix, we might be in virtual mode, so punt */
+ if (kvm_is_radix(vcpu->kvm))
+ return H_TOO_HARD;
+
stt = kvmppc_find_table(vcpu->kvm, liobn);
if (!stt)
return H_TOO_HARD;
long i, ret;
struct kvmppc_spapr_tce_iommu_table *stit;
+ /* For radix, we might be in virtual mode, so punt */
+ if (kvm_is_radix(vcpu->kvm))
+ return H_TOO_HARD;
+
stt = kvmppc_find_table(vcpu->kvm, liobn);
if (!stt)
return H_TOO_HARD;
return H_SUCCESS;
}
+/* This can be called in either virtual mode or real mode */
long kvmppc_h_get_tce(struct kvm_vcpu *vcpu, unsigned long liobn,
unsigned long ioba)
{
long kvmppc_h_random(struct kvm_vcpu *vcpu)
{
- if (powernv_get_random_real_mode(&vcpu->arch.gpr[4]))
+ int r;
+
+ /* Only need to do the expensive mfmsr() on radix */
+ if (kvm_is_radix(vcpu->kvm) && (mfmsr() & MSR_IR))
+ r = powernv_get_random_long(&vcpu->arch.gpr[4]);
+ else
+ r = powernv_get_random_real_mode(&vcpu->arch.gpr[4]);
+ if (r)
return H_SUCCESS;
return H_HARDWARE;
pteg_addr = get_pteg_addr(vcpu, pte_index);
mutex_lock(&vcpu->kvm->arch.hpt_mutex);
- copy_from_user(pteg, (void __user *)pteg_addr, sizeof(pteg));
+ ret = H_FUNCTION;
+ if (copy_from_user(pteg, (void __user *)pteg_addr, sizeof(pteg)))
+ goto done;
hpte = pteg;
ret = H_PTEG_FULL;
hpte[0] = cpu_to_be64(kvmppc_get_gpr(vcpu, 6));
hpte[1] = cpu_to_be64(kvmppc_get_gpr(vcpu, 7));
pteg_addr += i * HPTE_SIZE;
- copy_to_user((void __user *)pteg_addr, hpte, HPTE_SIZE);
+ ret = H_FUNCTION;
+ if (copy_to_user((void __user *)pteg_addr, hpte, HPTE_SIZE))
+ goto done;
kvmppc_set_gpr(vcpu, 4, pte_index | i);
ret = H_SUCCESS;
pteg = get_pteg_addr(vcpu, pte_index);
mutex_lock(&vcpu->kvm->arch.hpt_mutex);
- copy_from_user(pte, (void __user *)pteg, sizeof(pte));
+ ret = H_FUNCTION;
+ if (copy_from_user(pte, (void __user *)pteg, sizeof(pte)))
+ goto done;
pte[0] = be64_to_cpu((__force __be64)pte[0]);
pte[1] = be64_to_cpu((__force __be64)pte[1]);
((flags & H_ANDCOND) && (pte[0] & avpn) != 0))
goto done;
- copy_to_user((void __user *)pteg, &v, sizeof(v));
+ ret = H_FUNCTION;
+ if (copy_to_user((void __user *)pteg, &v, sizeof(v)))
+ goto done;
rb = compute_tlbie_rb(pte[0], pte[1], pte_index);
vcpu->arch.mmu.tlbie(vcpu, rb, rb & 1 ? true : false);
}
pteg = get_pteg_addr(vcpu, tsh & H_BULK_REMOVE_PTEX);
- copy_from_user(pte, (void __user *)pteg, sizeof(pte));
+ if (copy_from_user(pte, (void __user *)pteg, sizeof(pte))) {
+ ret = H_FUNCTION;
+ break;
+ }
pte[0] = be64_to_cpu((__force __be64)pte[0]);
pte[1] = be64_to_cpu((__force __be64)pte[1]);
tsh |= H_BULK_REMOVE_NOT_FOUND;
} else {
/* Splat the pteg in (userland) hpt */
- copy_to_user((void __user *)pteg, &v, sizeof(v));
+ if (copy_to_user((void __user *)pteg, &v, sizeof(v))) {
+ ret = H_FUNCTION;
+ break;
+ }
rb = compute_tlbie_rb(pte[0], pte[1],
tsh & H_BULK_REMOVE_PTEX);
pteg = get_pteg_addr(vcpu, pte_index);
mutex_lock(&vcpu->kvm->arch.hpt_mutex);
- copy_from_user(pte, (void __user *)pteg, sizeof(pte));
+ ret = H_FUNCTION;
+ if (copy_from_user(pte, (void __user *)pteg, sizeof(pte)))
+ goto done;
pte[0] = be64_to_cpu((__force __be64)pte[0]);
pte[1] = be64_to_cpu((__force __be64)pte[1]);
vcpu->arch.mmu.tlbie(vcpu, rb, rb & 1 ? true : false);
pte[0] = (__force u64)cpu_to_be64(pte[0]);
pte[1] = (__force u64)cpu_to_be64(pte[1]);
- copy_to_user((void __user *)pteg, pte, sizeof(pte));
+ ret = H_FUNCTION;
+ if (copy_to_user((void __user *)pteg, pte, sizeof(pte)))
+ goto done;
ret = H_SUCCESS;
done:
return EMULATE_DONE;
}
-static int kvmppc_h_pr_put_tce(struct kvm_vcpu *vcpu)
+static int kvmppc_h_pr_logical_ci_load(struct kvm_vcpu *vcpu)
{
- unsigned long liobn = kvmppc_get_gpr(vcpu, 4);
- unsigned long ioba = kvmppc_get_gpr(vcpu, 5);
- unsigned long tce = kvmppc_get_gpr(vcpu, 6);
long rc;
- rc = kvmppc_h_put_tce(vcpu, liobn, ioba, tce);
+ rc = kvmppc_h_logical_ci_load(vcpu);
if (rc == H_TOO_HARD)
return EMULATE_FAIL;
kvmppc_set_gpr(vcpu, 3, rc);
return EMULATE_DONE;
}
-static int kvmppc_h_pr_logical_ci_load(struct kvm_vcpu *vcpu)
+static int kvmppc_h_pr_logical_ci_store(struct kvm_vcpu *vcpu)
{
long rc;
- rc = kvmppc_h_logical_ci_load(vcpu);
+ rc = kvmppc_h_logical_ci_store(vcpu);
if (rc == H_TOO_HARD)
return EMULATE_FAIL;
kvmppc_set_gpr(vcpu, 3, rc);
return EMULATE_DONE;
}
-static int kvmppc_h_pr_logical_ci_store(struct kvm_vcpu *vcpu)
+#ifdef CONFIG_SPAPR_TCE_IOMMU
+static int kvmppc_h_pr_put_tce(struct kvm_vcpu *vcpu)
{
+ unsigned long liobn = kvmppc_get_gpr(vcpu, 4);
+ unsigned long ioba = kvmppc_get_gpr(vcpu, 5);
+ unsigned long tce = kvmppc_get_gpr(vcpu, 6);
long rc;
- rc = kvmppc_h_logical_ci_store(vcpu);
+ rc = kvmppc_h_put_tce(vcpu, liobn, ioba, tce);
if (rc == H_TOO_HARD)
return EMULATE_FAIL;
kvmppc_set_gpr(vcpu, 3, rc);
return EMULATE_DONE;
}
+#else /* CONFIG_SPAPR_TCE_IOMMU */
+static int kvmppc_h_pr_put_tce(struct kvm_vcpu *vcpu)
+{
+ return EMULATE_FAIL;
+}
+
+static int kvmppc_h_pr_put_tce_indirect(struct kvm_vcpu *vcpu)
+{
+ return EMULATE_FAIL;
+}
+
+static int kvmppc_h_pr_stuff_tce(struct kvm_vcpu *vcpu)
+{
+ return EMULATE_FAIL;
+}
+#endif /* CONFIG_SPAPR_TCE_IOMMU */
+
static int kvmppc_h_pr_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd)
{
long rc = kvmppc_xics_hcall(vcpu, cmd);
r = kvm_vm_ioctl_enable_cap(kvm, &cap);
break;
}
-#ifdef CONFIG_PPC_BOOK3S_64
+#ifdef CONFIG_SPAPR_TCE_IOMMU
case KVM_CREATE_SPAPR_TCE_64: {
struct kvm_create_spapr_tce_64 create_tce_64;
r = kvm_vm_ioctl_create_spapr_tce(kvm, &create_tce_64);
goto out;
}
+#endif
+#ifdef CONFIG_PPC_BOOK3S_64
case KVM_PPC_GET_SMMU_INFO: {
struct kvm_ppc_smmu_info info;
struct kvm *kvm = filp->private_data;
*/
#include <linux/debugfs.h>
#include <linux/fs.h>
+#include <linux/hugetlb.h>
#include <linux/io.h>
#include <linux/mm.h>
#include <linux/sched.h>
for (i = 0; i < PTRS_PER_PMD; i++, pmd++) {
addr = start + i * PMD_SIZE;
- if (!pmd_none(*pmd))
+ if (!pmd_none(*pmd) && !pmd_huge(*pmd))
/* pmd exists */
walk_pte(st, pmd, addr);
else
for (i = 0; i < PTRS_PER_PUD; i++, pud++) {
addr = start + i * PUD_SIZE;
- if (!pud_none(*pud))
+ if (!pud_none(*pud) && !pud_huge(*pud))
/* pud exists */
walk_pmd(st, pud, addr);
else
*/
for (i = 0; i < PTRS_PER_PGD; i++, pgd++) {
addr = KERN_VIRT_START + i * PGDIR_SIZE;
- if (!pgd_none(*pgd))
+ if (!pgd_none(*pgd) && !pgd_huge(*pgd))
/* pgd exists */
walk_pud(st, pgd, addr);
else
(REGION_ID(ea) != USER_REGION_ID)) {
spin_unlock(&spu->register_lock);
- ret = hash_page(ea, _PAGE_PRESENT | _PAGE_READ, 0x300, dsisr);
+ ret = hash_page(ea,
+ _PAGE_PRESENT | _PAGE_READ | _PAGE_PRIVILEGED,
+ 0x300, dsisr);
spin_lock(&spu->register_lock);
if (!ret) {
void pnv_npu2_destroy_context(struct npu_context *npu_context,
struct pci_dev *gpdev)
{
- struct pnv_phb *nphb, *phb;
+ struct pnv_phb *nphb;
struct npu *npu;
struct pci_dev *npdev = pnv_pci_get_npu_dev(gpdev, 0);
struct device_node *nvlink_dn;
nphb = pci_bus_to_host(npdev->bus)->private_data;
npu = &nphb->npu;
- phb = pci_bus_to_host(gpdev->bus)->private_data;
nvlink_dn = of_parse_phandle(npdev->dev.of_node, "ibm,nvlink", 0);
if (WARN_ON(of_property_read_u32(nvlink_dn, "ibm,npu-link-index",
&nvlink_index)))
return;
npu_context->npdev[npu->index][nvlink_index] = NULL;
- opal_npu_destroy_context(phb->opal_id, npu_context->mm->context.id,
+ opal_npu_destroy_context(nphb->opal_id, npu_context->mm->context.id,
PCI_DEVID(gpdev->bus->number, gpdev->devfn));
kref_put(&npu_context->kref, pnv_npu2_release_context);
}
#include <linux/spinlock.h>
#include <linux/kernel.h>
#include <linux/time.h>
+#include <linux/refcount.h>
#include <uapi/asm/debug.h>
#define DEBUG_MAX_LEVEL 6 /* debug levels range from 0 to 6 */
typedef struct debug_info {
struct debug_info* next;
struct debug_info* prev;
- atomic_t ref_count;
+ refcount_t ref_count;
spinlock_t lock;
int level;
int nr_areas;
return ((((int) code + 64) >> 7) + 1) << 1;
}
+struct pt_regs;
+
void show_code(struct pt_regs *regs);
void print_fn_code(unsigned char *code, unsigned long len);
int insn_to_mnemonic(unsigned char *instruction, char *buf, unsigned int len);
* 2005-Dec Used as a template for s390 by Mike Grundy
* <grundym@us.ibm.com>
*/
+#include <linux/types.h>
#include <asm-generic/kprobes.h>
#define BREAKPOINT_INSTRUCTION 0x0002
+#define FIXUP_PSW_NORMAL 0x08
+#define FIXUP_BRANCH_NOT_TAKEN 0x04
+#define FIXUP_RETURN_REGISTER 0x02
+#define FIXUP_NOT_REQUIRED 0x01
+
+int probe_is_prohibited_opcode(u16 *insn);
+int probe_get_fixup_type(u16 *insn);
+int probe_is_insn_relative_long(u16 *insn);
+
#ifdef CONFIG_KPROBES
-#include <linux/types.h>
#include <linux/ptrace.h>
#include <linux/percpu.h>
#include <linux/sched/task_stack.h>
#define KPROBE_SWAP_INST 0x10
-#define FIXUP_PSW_NORMAL 0x08
-#define FIXUP_BRANCH_NOT_TAKEN 0x04
-#define FIXUP_RETURN_REGISTER 0x02
-#define FIXUP_NOT_REQUIRED 0x01
-
/* Architecture specific copy of original instruction */
struct arch_specific_insn {
/* copy of original instruction */
int kprobe_exceptions_notify(struct notifier_block *self,
unsigned long val, void *data);
-int probe_is_prohibited_opcode(u16 *insn);
-int probe_get_fixup_type(u16 *insn);
-int probe_is_insn_relative_long(u16 *insn);
-
#define flush_insn_slot(p) do { } while (0)
#endif /* CONFIG_KPROBES */
* Returns the maximum nesting level supported by the cpu topology code.
* The current maximum level is 4 which is the drawer level.
*/
-static inline int topology_mnest_limit(void)
+static inline unsigned char topology_mnest_limit(void)
{
return min(topology_max_mnest, 4);
}
memset(rc->views, 0, DEBUG_MAX_VIEWS * sizeof(struct debug_view *));
memset(rc->debugfs_entries, 0 ,DEBUG_MAX_VIEWS *
sizeof(struct dentry*));
- atomic_set(&(rc->ref_count), 0);
+ refcount_set(&(rc->ref_count), 0);
return rc;
debug_area_last = rc;
rc->next = NULL;
- debug_info_get(rc);
+ refcount_set(&rc->ref_count, 1);
out:
return rc;
}
debug_info_get(debug_info_t * db_info)
{
if (db_info)
- atomic_inc(&db_info->ref_count);
+ refcount_inc(&db_info->ref_count);
}
/*
if (!db_info)
return;
- if (atomic_dec_and_test(&db_info->ref_count)) {
+ if (refcount_dec_and_test(&db_info->ref_count)) {
for (i = 0; i < DEBUG_MAX_VIEWS; i++) {
if (!db_info->views[i])
continue;
lg %r14,__LC_VDSO_PER_CPU
lmg %r0,%r10,__PT_R0(%r11)
mvc __LC_RETURN_PSW(16),__PT_PSW(%r11)
+.Lsysc_exit_timer:
stpt __LC_EXIT_TIMER
mvc __VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
lmg %r11,%r15,__PT_R11(%r11)
lg %r14,__LC_VDSO_PER_CPU
lmg %r0,%r10,__PT_R0(%r11)
mvc __LC_RETURN_PSW(16),__PT_PSW(%r11)
+.Lio_exit_timer:
stpt __LC_EXIT_TIMER
mvc __VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
lmg %r11,%r15,__PT_R11(%r11)
br %r14
.Lcleanup_sysc_restore:
+ # check if stpt has been executed
clg %r9,BASED(.Lcleanup_sysc_restore_insn)
+ jh 0f
+ mvc __LC_EXIT_TIMER(8),__LC_ASYNC_ENTER_TIMER
+ cghi %r11,__LC_SAVE_AREA_ASYNC
je 0f
+ mvc __LC_EXIT_TIMER(8),__LC_MCCK_ENTER_TIMER
+0: clg %r9,BASED(.Lcleanup_sysc_restore_insn+8)
+ je 1f
lg %r9,24(%r11) # get saved pointer to pt_regs
mvc __LC_RETURN_PSW(16),__PT_PSW(%r9)
mvc 0(64,%r11),__PT_R8(%r9)
lmg %r0,%r7,__PT_R0(%r9)
-0: lmg %r8,%r9,__LC_RETURN_PSW
+1: lmg %r8,%r9,__LC_RETURN_PSW
br %r14
.Lcleanup_sysc_restore_insn:
+ .quad .Lsysc_exit_timer
.quad .Lsysc_done - 4
.Lcleanup_io_tif:
br %r14
.Lcleanup_io_restore:
+ # check if stpt has been executed
clg %r9,BASED(.Lcleanup_io_restore_insn)
- je 0f
+ jh 0f
+ mvc __LC_EXIT_TIMER(8),__LC_MCCK_ENTER_TIMER
+0: clg %r9,BASED(.Lcleanup_io_restore_insn+8)
+ je 1f
lg %r9,24(%r11) # get saved r11 pointer to pt_regs
mvc __LC_RETURN_PSW(16),__PT_PSW(%r9)
mvc 0(64,%r11),__PT_R8(%r9)
lmg %r0,%r7,__PT_R0(%r9)
-0: lmg %r8,%r9,__LC_RETURN_PSW
+1: lmg %r8,%r9,__LC_RETURN_PSW
br %r14
.Lcleanup_io_restore_insn:
+ .quad .Lio_exit_timer
.quad .Lio_done - 4
.Lcleanup_idle:
return 0;
}
+#ifdef CONFIG_MODULES
+
static int __init ftrace_plt_init(void)
{
unsigned int *ip;
}
device_initcall(ftrace_plt_init);
+#endif /* CONFIG_MODULES */
+
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
/*
* Hook the return address and push it in the stack of return addresses
{
. = 0x00000000;
.text : {
- _text = .; /* Text and read-only data */
+ /* Text and read-only data */
HEAD_TEXT
+ /*
+ * E.g. perf doesn't like symbols starting at address zero,
+ * therefore skip the initial PSW and channel program located
+ * at address zero and let _text start at 0x200.
+ */
+ _text = 0x200;
TEXT_TEXT
SCHED_TEXT
CPUIDLE_TEXT
* Copyright IBM Corp. 2014
*/
+#include <linux/errno.h>
#include <asm/kprobes.h>
#include <asm/dis.h>
return 0;
done = 0;
do {
- offset = (size_t)src & ~PAGE_MASK;
- len = min(size - done, PAGE_SIZE - offset);
+ offset = (size_t)src & (L1_CACHE_BYTES - 1);
+ len = min(size - done, L1_CACHE_BYTES - offset);
if (copy_from_user(dst, src, len))
return -EFAULT;
len_str = strnlen(dst, len);
static inline int prepare_hugepage_range(struct file *file,
unsigned long addr, unsigned long len)
{
- if (len & ~HPAGE_MASK)
+ struct hstate *h = hstate_file(file);
+
+ if (len & ~huge_page_mask(h))
return -EINVAL;
- if (addr & ~HPAGE_MASK)
+ if (addr & ~huge_page_mask(h))
return -EINVAL;
return 0;
}
* ZERO_PAGE is a global shared page that is always zero: used
* for zero-mapped memory areas etc..
*/
-extern unsigned long empty_zero_page;
+extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
-#define ZERO_PAGE(vaddr) (virt_to_page(&empty_zero_page))
+#define ZERO_PAGE(vaddr) (virt_to_page(empty_zero_page))
/*
* In general all page table modifications should use the V8 atomic
*/
extern unsigned char boot_cpu_id;
-extern unsigned long empty_zero_page;
+extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
extern int serial_console;
static inline int con_is_present(void)
if (unlikely(atomic_read(¤t->tracing_graph_pause)))
return parent + 8UL;
- if (ftrace_push_return_trace(parent, self_addr, &trace.depth,
- frame_pointer, NULL) == -EBUSY)
- return parent + 8UL;
-
trace.func = self_addr;
+ trace.depth = current->curr_ret_stack + 1;
/* Only trace if the calling function expects to */
- if (!ftrace_graph_entry(&trace)) {
- current->curr_ret_stack--;
+ if (!ftrace_graph_entry(&trace))
+ return parent + 8UL;
+
+ if (ftrace_push_return_trace(parent, self_addr, &trace.depth,
+ frame_pointer, NULL) == -EBUSY)
return parent + 8UL;
- }
return return_hooker;
}
/* Saves us work later. */
- memset((void *)&empty_zero_page, 0, PAGE_SIZE);
+ memset((void *)empty_zero_page, 0, PAGE_SIZE);
i = last_valid_pfn >> ((20 - PAGE_SHIFT) + 5);
i += 1;
Management" code will be disabled if you say Y here.
See also <file:Documentation/x86/i386/IO-APIC.txt>,
- <file:Documentation/nmi_watchdog.txt> and the SMP-HOWTO available at
+ <file:Documentation/lockup-watchdogs.txt> and the SMP-HOWTO available at
<http://www.tldp.org/docs.html#howto>.
If you don't know what to do here, say N.
# If '-Os' is enabled, disable it and print a warning.
ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
undefine CONFIG_CC_OPTIMIZE_FOR_SIZE
- $(warning Disabling CONFIG_CC_OPTIMIZE_FOR_SIZE. Your compiler does not have -mfentry so you cannot optimize for size with CONFIG_FUNCTION_GRAPH_TRACER.)
+ $(warning Disabling CONFIG_CC_OPTIMIZE_FOR_SIZE. Your compiler does not have -mfentry so you cannot optimize for size with CONFIG_FUNCTION_GRAPH_TRACER.)
endif
endif
quiet_cmd_check_data_rel = DATAREL $@
define cmd_check_data_rel
for obj in $(filter %.o,$^); do \
- readelf -S $$obj | grep -qF .rel.local && { \
+ ${CROSS_COMPILE}readelf -S $$obj | grep -qF .rel.local && { \
echo "error: $$obj has data relocations!" >&2; \
exit 1; \
} || true; \
jmp __switch_to
END(__switch_to_asm)
+/*
+ * The unwinder expects the last frame on the stack to always be at the same
+ * offset from the end of the page, which allows it to validate the stack.
+ * Calling schedule_tail() directly would break that convention because its an
+ * asmlinkage function so its argument has to be pushed on the stack. This
+ * wrapper creates a proper "end of stack" frame header before the call.
+ */
+ENTRY(schedule_tail_wrapper)
+ FRAME_BEGIN
+
+ pushl %eax
+ call schedule_tail
+ popl %eax
+
+ FRAME_END
+ ret
+ENDPROC(schedule_tail_wrapper)
/*
* A newly forked process directly context switches into this address.
*
* edi: kernel thread arg
*/
ENTRY(ret_from_fork)
- FRAME_BEGIN /* help unwinder find end of stack */
-
- /*
- * schedule_tail() is asmlinkage so we have to put its 'prev' argument
- * on the stack.
- */
- pushl %eax
- call schedule_tail
- popl %eax
+ call schedule_tail_wrapper
testl %ebx, %ebx
jnz 1f /* kernel threads are uncommon */
2:
/* When we fork, we trace the syscall return in the child, too. */
- leal FRAME_OFFSET(%esp), %eax
+ movl %esp, %eax
call syscall_return_slowpath
- FRAME_END
jmp restore_all
/* kernel thread */
#include <asm/smap.h>
#include <asm/pgtable_types.h>
#include <asm/export.h>
-#include <asm/frame.h>
#include <linux/err.h>
.code64
* r12: kernel thread arg
*/
ENTRY(ret_from_fork)
- FRAME_BEGIN /* help unwinder find end of stack */
movq %rax, %rdi
- call schedule_tail /* rdi: 'prev' task parameter */
+ call schedule_tail /* rdi: 'prev' task parameter */
- testq %rbx, %rbx /* from kernel_thread? */
- jnz 1f /* kernel threads are uncommon */
+ testq %rbx, %rbx /* from kernel_thread? */
+ jnz 1f /* kernel threads are uncommon */
2:
- leaq FRAME_OFFSET(%rsp),%rdi /* pt_regs pointer */
+ movq %rsp, %rdi
call syscall_return_slowpath /* returns with IRQs disabled */
TRACE_IRQS_ON /* user mode is traced as IRQS on */
SWAPGS
- FRAME_END
jmp restore_regs_and_iret
1:
#define KVM_PRIVATE_MEM_SLOTS 3
#define KVM_MEM_SLOTS_NUM (KVM_USER_MEM_SLOTS + KVM_PRIVATE_MEM_SLOTS)
-#define KVM_HALT_POLL_NS_DEFAULT 400000
+#define KVM_HALT_POLL_NS_DEFAULT 200000
#define KVM_IRQCHIP_NUM_PINS KVM_IOAPIC_NUM_PINS
#endif
int mce_available(struct cpuinfo_x86 *c);
+bool mce_is_memory_error(struct mce *m);
DECLARE_PER_CPU(unsigned, mce_exception_count);
DECLARE_PER_CPU(unsigned, mce_poll_count);
#define __get_user_asm_u64(x, ptr, retval, errret) \
({ \
__typeof__(ptr) __ptr = (ptr); \
- asm volatile(ASM_STAC "\n" \
+ asm volatile("\n" \
"1: movl %2,%%eax\n" \
"2: movl %3,%%edx\n" \
- "3: " ASM_CLAC "\n" \
+ "3:\n" \
".section .fixup,\"ax\"\n" \
"4: mov %4,%0\n" \
" xorl %%eax,%%eax\n" \
".previous\n" \
_ASM_EXTABLE(1b, 4b) \
_ASM_EXTABLE(2b, 4b) \
- : "=r" (retval), "=A"(x) \
+ : "=r" (retval), "=&A"(x) \
: "m" (__m(__ptr)), "m" __m(((u32 *)(__ptr)) + 1), \
"i" (errret), "0" (retval)); \
})
#define unsafe_put_user(x, ptr, err_label) \
do { \
int __pu_err; \
- __put_user_size((x), (ptr), sizeof(*(ptr)), __pu_err, -EFAULT); \
+ __typeof__(*(ptr)) __pu_val = (x); \
+ __put_user_size(__pu_val, (ptr), sizeof(*(ptr)), __pu_err, -EFAULT); \
if (unlikely(__pu_err)) goto err_label; \
} while (0)
#define unsafe_get_user(x, ptr, err_label) \
do { \
int __gu_err; \
- unsigned long __gu_val; \
+ __inttype(*(ptr)) __gu_val; \
__get_user_size(__gu_val, (ptr), sizeof(*(ptr)), __gu_err, -EFAULT); \
(x) = (__force __typeof__(*(ptr)))__gu_val; \
if (unlikely(__gu_err)) goto err_label; \
memcpy(insnbuf, replacement, a->replacementlen);
insnbuf_sz = a->replacementlen;
- /* 0xe8 is a relative jump; fix the offset. */
- if (*insnbuf == 0xe8 && a->replacementlen == 5) {
+ /*
+ * 0xe8 is a relative jump; fix the offset.
+ *
+ * Instruction length is checked before the opcode to avoid
+ * accessing uninitialized bytes for zero-length replacements.
+ */
+ if (a->replacementlen == 5 && *insnbuf == 0xe8) {
*(s32 *)(insnbuf + 1) += replacement - instr;
DPRINTK("Fix CALL offset: 0x%x, CALL 0x%lx",
*(s32 *)(insnbuf + 1),
return 1;
}
-static bool memory_error(struct mce *m)
+bool mce_is_memory_error(struct mce *m)
{
- struct cpuinfo_x86 *c = &boot_cpu_data;
-
- if (c->x86_vendor == X86_VENDOR_AMD) {
+ if (m->cpuvendor == X86_VENDOR_AMD) {
/* ErrCodeExt[20:16] */
u8 xec = (m->status >> 16) & 0x1f;
return (xec == 0x0 || xec == 0x8);
- } else if (c->x86_vendor == X86_VENDOR_INTEL) {
+ } else if (m->cpuvendor == X86_VENDOR_INTEL) {
/*
* Intel SDM Volume 3B - 15.9.2 Compound Error Codes
*
return false;
}
+EXPORT_SYMBOL_GPL(mce_is_memory_error);
static bool cec_add_mce(struct mce *m)
{
return false;
/* We eat only correctable DRAM errors with usable addresses. */
- if (memory_error(m) &&
+ if (mce_is_memory_error(m) &&
!(m->status & MCI_STATUS_UC) &&
mce_usable_address(m))
if (!cec_add_elem(m->addr >> PAGE_SHIFT))
severity = mce_severity(&m, mca_cfg.tolerant, NULL, false);
- if (severity == MCE_DEFERRED_SEVERITY && memory_error(&m))
+ if (severity == MCE_DEFERRED_SEVERITY && mce_is_memory_error(&m))
if (m.status & MCI_STATUS_ADDRV)
m.severity = severity;
* Boot time FPU feature detection code:
*/
unsigned int mxcsr_feature_mask __read_mostly = 0xffffffffu;
+EXPORT_SYMBOL_GPL(mxcsr_feature_mask);
static void __init fpu__init_system_mxcsr(void)
{
{
return module_alloc(size);
}
-static inline void tramp_free(void *tramp)
+static inline void tramp_free(void *tramp, int size)
{
+ int npages = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+ set_memory_nx((unsigned long)tramp, npages);
+ set_memory_rw((unsigned long)tramp, npages);
module_memfree(tramp);
}
#else
{
return NULL;
}
-static inline void tramp_free(void *tramp) { }
+static inline void tramp_free(void *tramp, int size) { }
#endif
/* Defined as markers to the end of the ftrace default trampolines */
/* Copy ftrace_caller onto the trampoline memory */
ret = probe_kernel_read(trampoline, (void *)start_offset, size);
if (WARN_ON(ret < 0)) {
- tramp_free(trampoline);
+ tramp_free(trampoline, *tramp_size);
return 0;
}
/* Are we pointing to the reference? */
if (WARN_ON(memcmp(op_ptr.op, op_ref, 3) != 0)) {
- tramp_free(trampoline);
+ tramp_free(trampoline, *tramp_size);
return 0;
}
unsigned long offset;
unsigned long ip;
unsigned int size;
- int ret;
+ int ret, npages;
if (ops->trampoline) {
/*
*/
if (!(ops->flags & FTRACE_OPS_FL_ALLOC_TRAMP))
return;
+ npages = PAGE_ALIGN(ops->trampoline_size) >> PAGE_SHIFT;
+ set_memory_rw(ops->trampoline, npages);
} else {
ops->trampoline = create_trampoline(ops, &size);
if (!ops->trampoline)
return;
ops->trampoline_size = size;
+ npages = PAGE_ALIGN(size) >> PAGE_SHIFT;
}
offset = calc_trampoline_call_offset(ops->flags & FTRACE_OPS_FL_SAVE_REGS);
/* Do a safe modify in case the trampoline is executing */
new = ftrace_call_replace(ip, (unsigned long)func);
ret = update_ftrace_func(ip, new);
+ set_memory_ro(ops->trampoline, npages);
/* The update should never fail */
WARN_ON(ret);
if (!ops || !(ops->flags & FTRACE_OPS_FL_ALLOC_TRAMP))
return;
- tramp_free((void *)ops->trampoline);
+ tramp_free((void *)ops->trampoline, ops->trampoline_size);
ops->trampoline = 0;
}
#include <linux/ftrace.h>
#include <linux/frame.h>
#include <linux/kasan.h>
+#include <linux/moduleloader.h>
#include <asm/text-patching.h>
#include <asm/cacheflush.h>
}
}
+/* Recover page to RW mode before releasing it */
+void free_insn_page(void *page)
+{
+ set_memory_nx((unsigned long)page & PAGE_MASK, 1);
+ set_memory_rw((unsigned long)page & PAGE_MASK, 1);
+ module_memfree(page);
+}
+
static int arch_copy_kprobe(struct kprobe *p)
{
struct insn insn;
*/
x86_configure_nx();
- simple_udelay_calibration();
-
parse_early_param();
#ifdef CONFIG_MEMORY_HOTPLUG
*/
init_hypervisor_platform();
+ simple_udelay_calibration();
+
x86_init.resources.probe_roms();
/* after parse_early_param, so could debug it */
return (unsigned long *)task_pt_regs(state->task) - 2;
}
+static bool is_last_frame(struct unwind_state *state)
+{
+ return state->bp == last_frame(state);
+}
+
#ifdef CONFIG_X86_32
#define GCC_REALIGN_WORDS 3
#else
return last_frame(state) - GCC_REALIGN_WORDS;
}
-static bool is_last_task_frame(struct unwind_state *state)
+static bool is_last_aligned_frame(struct unwind_state *state)
{
unsigned long *last_bp = last_frame(state);
unsigned long *aligned_bp = last_aligned_frame(state);
/*
- * We have to check for the last task frame at two different locations
- * because gcc can occasionally decide to realign the stack pointer and
- * change the offset of the stack frame in the prologue of a function
- * called by head/entry code. Examples:
+ * GCC can occasionally decide to realign the stack pointer and change
+ * the offset of the stack frame in the prologue of a function called
+ * by head/entry code. Examples:
*
* <start_secondary>:
* push %edi
* push %rbp
* mov %rsp,%rbp
*
- * Note that after aligning the stack, it pushes a duplicate copy of
- * the return address before pushing the frame pointer.
+ * After aligning the stack, it pushes a duplicate copy of the return
+ * address before pushing the frame pointer.
+ */
+ return (state->bp == aligned_bp && *(aligned_bp + 1) == *(last_bp + 1));
+}
+
+static bool is_last_ftrace_frame(struct unwind_state *state)
+{
+ unsigned long *last_bp = last_frame(state);
+ unsigned long *last_ftrace_bp = last_bp - 3;
+
+ /*
+ * When unwinding from an ftrace handler of a function called by entry
+ * code, the stack layout of the last frame is:
+ *
+ * bp
+ * parent ret addr
+ * bp
+ * function ret addr
+ * parent ret addr
+ * pt_regs
+ * -----------------
*/
- return (state->bp == last_bp ||
- (state->bp == aligned_bp && *(aligned_bp+1) == *(last_bp+1)));
+ return (state->bp == last_ftrace_bp &&
+ *state->bp == *(state->bp + 2) &&
+ *(state->bp + 1) == *(state->bp + 4));
+}
+
+static bool is_last_task_frame(struct unwind_state *state)
+{
+ return is_last_frame(state) || is_last_aligned_frame(state) ||
+ is_last_ftrace_frame(state);
}
/*
static int check_svme(struct x86_emulate_ctxt *ctxt)
{
- u64 efer;
+ u64 efer = 0;
ctxt->ops->get_msr(ctxt, MSR_EFER, &efer);
pt_element_t pte;
pt_element_t __user *uninitialized_var(ptep_user);
gfn_t table_gfn;
- unsigned index, pt_access, pte_access, accessed_dirty, pte_pkey;
+ u64 pt_access, pte_access;
+ unsigned index, accessed_dirty, pte_pkey;
unsigned nested_access;
gpa_t pte_gpa;
bool have_ad;
int offset;
+ u64 walk_nx_mask = 0;
const int write_fault = access & PFERR_WRITE_MASK;
const int user_fault = access & PFERR_USER_MASK;
const int fetch_fault = access & PFERR_FETCH_MASK;
have_ad = PT_HAVE_ACCESSED_DIRTY(mmu);
#if PTTYPE == 64
+ walk_nx_mask = 1ULL << PT64_NX_SHIFT;
if (walker->level == PT32E_ROOT_LEVEL) {
pte = mmu->get_pdptr(vcpu, (addr >> 30) & 3);
trace_kvm_mmu_paging_element(pte, walker->level);
walker->max_level = walker->level;
ASSERT(!(is_long_mode(vcpu) && !is_pae(vcpu)));
- accessed_dirty = have_ad ? PT_GUEST_ACCESSED_MASK : 0;
-
/*
* FIXME: on Intel processors, loads of the PDPTE registers for PAE paging
* by the MOV to CR instruction are treated as reads and do not cause the
*/
nested_access = (have_ad ? PFERR_WRITE_MASK : 0) | PFERR_USER_MASK;
- pt_access = pte_access = ACC_ALL;
+ pte_access = ~0;
++walker->level;
do {
gfn_t real_gfn;
unsigned long host_addr;
- pt_access &= pte_access;
+ pt_access = pte_access;
--walker->level;
index = PT_INDEX(addr, walker->level);
trace_kvm_mmu_paging_element(pte, walker->level);
+ /*
+ * Inverting the NX it lets us AND it like other
+ * permission bits.
+ */
+ pte_access = pt_access & (pte ^ walk_nx_mask);
+
if (unlikely(!FNAME(is_present_gpte)(pte)))
goto error;
goto error;
}
- accessed_dirty &= pte;
- pte_access = pt_access & FNAME(gpte_access)(vcpu, pte);
-
walker->ptes[walker->level - 1] = pte;
} while (!is_last_gpte(mmu, walker->level, pte));
pte_pkey = FNAME(gpte_pkeys)(vcpu, pte);
- errcode = permission_fault(vcpu, mmu, pte_access, pte_pkey, access);
+ accessed_dirty = have_ad ? pte_access & PT_GUEST_ACCESSED_MASK : 0;
+
+ /* Convert to ACC_*_MASK flags for struct guest_walker. */
+ walker->pt_access = FNAME(gpte_access)(vcpu, pt_access ^ walk_nx_mask);
+ walker->pte_access = FNAME(gpte_access)(vcpu, pte_access ^ walk_nx_mask);
+ errcode = permission_fault(vcpu, mmu, walker->pte_access, pte_pkey, access);
if (unlikely(errcode))
goto error;
walker->gfn = real_gpa >> PAGE_SHIFT;
if (!write_fault)
- FNAME(protect_clean_gpte)(mmu, &pte_access, pte);
+ FNAME(protect_clean_gpte)(mmu, &walker->pte_access, pte);
else
/*
* On a write fault, fold the dirty bit into accessed_dirty.
goto retry_walk;
}
- walker->pt_access = pt_access;
- walker->pte_access = pte_access;
pgprintk("%s: pte %llx pte_access %x pt_access %x\n",
- __func__, (u64)pte, pte_access, pt_access);
+ __func__, (u64)pte, walker->pte_access, walker->pt_access);
return 1;
error:
*/
if (!(errcode & PFERR_RSVD_MASK)) {
vcpu->arch.exit_qualification &= 0x187;
- vcpu->arch.exit_qualification |= ((pt_access & pte) & 0x7) << 3;
+ vcpu->arch.exit_qualification |= (pte_access & 0x7) << 3;
}
#endif
walker->fault.address = addr;
((u64)1 << edx.split.bit_width_fixed) - 1;
}
- pmu->global_ctrl = ((1 << pmu->nr_arch_gp_counters) - 1) |
+ pmu->global_ctrl = ((1ull << pmu->nr_arch_gp_counters) - 1) |
(((1ull << pmu->nr_arch_fixed_counters) - 1) << INTEL_PMC_IDX_FIXED);
pmu->global_ctrl_mask = ~pmu->global_ctrl;
}
-static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu, int index)
+static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu,
+ unsigned int index)
{
u64 *avic_physical_id_table;
struct kvm_arch *vm_data = &vcpu->kvm->arch;
enable_ept_ad_bits = 0;
}
- if (!cpu_has_vmx_ept_ad_bits())
+ if (!cpu_has_vmx_ept_ad_bits() || !enable_ept)
enable_ept_ad_bits = 0;
if (!cpu_has_vmx_unrestricted_guest())
if (!nested_cpu_has_pml(vmcs12))
return 0;
- if (vmcs12->guest_pml_index > PML_ENTITY_NUM) {
+ if (vmcs12->guest_pml_index >= PML_ENTITY_NUM) {
vmx->nested.pml_full = true;
return 1;
}
{
struct kvm_arch *ka = &kvm->arch;
struct pvclock_vcpu_time_info hv_clock;
+ u64 ret;
spin_lock(&ka->pvclock_gtod_sync_lock);
if (!ka->use_master_clock) {
hv_clock.system_time = ka->master_kernel_ns + ka->kvmclock_offset;
spin_unlock(&ka->pvclock_gtod_sync_lock);
+ /* both __this_cpu_read() and rdtsc() should be on the same cpu */
+ get_cpu();
+
kvm_get_time_scale(NSEC_PER_SEC, __this_cpu_read(cpu_tsc_khz) * 1000LL,
&hv_clock.tsc_shift,
&hv_clock.tsc_to_system_mul);
- return __pvclock_read_cycles(&hv_clock, rdtsc());
+ ret = __pvclock_read_cycles(&hv_clock, rdtsc());
+
+ put_cpu();
+
+ return ret;
}
static void kvm_setup_pvclock_page(struct kvm_vcpu *v)
}
}
+#define XSAVE_MXCSR_OFFSET 24
+
static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
struct kvm_xsave *guest_xsave)
{
u64 xstate_bv =
*(u64 *)&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)];
+ u32 mxcsr = *(u32 *)&guest_xsave->region[XSAVE_MXCSR_OFFSET / sizeof(u32)];
if (boot_cpu_has(X86_FEATURE_XSAVE)) {
/*
* CPUID leaf 0xD, index 0, EDX:EAX. This is for compatibility
* with old userspace.
*/
- if (xstate_bv & ~kvm_supported_xcr0())
+ if (xstate_bv & ~kvm_supported_xcr0() ||
+ mxcsr & ~mxcsr_feature_mask)
return -EINVAL;
load_xsave(vcpu, (u8 *)guest_xsave->region);
} else {
- if (xstate_bv & ~XFEATURE_MASK_FPSSE)
+ if (xstate_bv & ~XFEATURE_MASK_FPSSE ||
+ mxcsr & ~mxcsr_feature_mask)
return -EINVAL;
memcpy(&vcpu->arch.guest_fpu.state.fxsave,
guest_xsave->region, sizeof(struct fxregs_state));
static int kernel_pio(struct kvm_vcpu *vcpu, void *pd)
{
- /* TODO: String I/O for in kernel device */
- int r;
+ int r = 0, i;
- if (vcpu->arch.pio.in)
- r = kvm_io_bus_read(vcpu, KVM_PIO_BUS, vcpu->arch.pio.port,
- vcpu->arch.pio.size, pd);
- else
- r = kvm_io_bus_write(vcpu, KVM_PIO_BUS,
- vcpu->arch.pio.port, vcpu->arch.pio.size,
- pd);
+ for (i = 0; i < vcpu->arch.pio.count; i++) {
+ if (vcpu->arch.pio.in)
+ r = kvm_io_bus_read(vcpu, KVM_PIO_BUS, vcpu->arch.pio.port,
+ vcpu->arch.pio.size, pd);
+ else
+ r = kvm_io_bus_write(vcpu, KVM_PIO_BUS,
+ vcpu->arch.pio.port, vcpu->arch.pio.size,
+ pd);
+ if (r)
+ break;
+ pd += vcpu->arch.pio.size;
+ }
return r;
}
if (vcpu->arch.pio.count)
goto data_avail;
+ memset(vcpu->arch.pio_data, 0, size * count);
+
ret = emulator_pio_in_out(vcpu, size, port, val, count, true);
if (ret) {
data_avail:
if (var.unusable) {
memset(desc, 0, sizeof(*desc));
+ if (base3)
+ *base3 = 0;
return false;
}
unsigned int i, level;
unsigned long addr;
- BUG_ON(irqs_disabled());
+ BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
WARN_ON(PAGE_ALIGN(start) != start);
on_each_cpu(__cpa_flush_range, NULL, 1);
}
early_param("nopat", nopat);
+static bool __read_mostly __pat_initialized = false;
+
bool pat_enabled(void)
{
- return !!__pat_enabled;
+ return __pat_initialized;
}
EXPORT_SYMBOL_GPL(pat_enabled);
}
wrmsrl(MSR_IA32_CR_PAT, pat);
+ __pat_initialized = true;
__init_cache_modes(pat);
}
static void pat_ap_init(u64 pat)
{
- if (!boot_cpu_has(X86_FEATURE_PAT)) {
+ if (!this_cpu_has(X86_FEATURE_PAT)) {
/*
* If this happens we are on a secondary CPU, but switched to
* PAT on the boot CPU. We have no way to undo PAT.
u64 pat;
struct cpuinfo_x86 *c = &boot_cpu_data;
- if (!pat_enabled()) {
+ if (!__pat_enabled) {
init_cache_modes();
return;
}
struct xen_extraversion extra;
HYPERVISOR_xen_version(XENVER_extraversion, &extra);
- pr_info("Booting paravirtualized kernel %son %s\n",
- xen_feature(XENFEAT_auto_translated_physmap) ?
- "with PVH extensions " : "", pv_info.name);
+ pr_info("Booting paravirtualized kernel on %s\n", pv_info.name);
printk(KERN_INFO "Xen version: %d.%d%s%s\n",
version >> 16, version & 0xffff, extra.extraversion,
xen_feature(XENFEAT_mmu_pt_update_preserve_ad) ? " (preserve-AD)" : "");
void xen_setup_shared_info(void)
{
- if (!xen_feature(XENFEAT_auto_translated_physmap)) {
- set_fixmap(FIX_PARAVIRT_BOOTMAP,
- xen_start_info->shared_info);
+ set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_start_info->shared_info);
- HYPERVISOR_shared_info =
- (struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
- } else
- HYPERVISOR_shared_info =
- (struct shared_info *)__va(xen_start_info->shared_info);
+ HYPERVISOR_shared_info =
+ (struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
#ifndef CONFIG_SMP
/* In UP this is as good a place as any to set up shared info */
}
EXPORT_SYMBOL_GPL(arbitrary_virt_to_machine);
-void xen_flush_tlb_all(void)
+static void xen_flush_tlb_all(void)
{
struct mmuext_op *op;
struct multicall_space mcs;
pteval_t flags = val & PTE_FLAGS_MASK;
unsigned long mfn;
- if (!xen_feature(XENFEAT_auto_translated_physmap))
- mfn = __pfn_to_mfn(pfn);
- else
- mfn = pfn;
+ mfn = __pfn_to_mfn(pfn);
+
/*
* If there's no mfn for the pfn, then just create an
* empty non-present pte. Unfortunately this loses
limit--;
BUG_ON(limit >= FIXADDR_TOP);
- if (xen_feature(XENFEAT_auto_translated_physmap))
- return 0;
-
/*
* 64-bit has a great big hole in the middle of the address
* space, which contains the Xen mappings. On 32-bit these
static void __init xen_pagetable_p2m_setup(void)
{
- if (xen_feature(XENFEAT_auto_translated_physmap))
- return;
-
xen_vmalloc_p2m_tree();
#ifdef CONFIG_X86_64
xen_build_mfn_list_list();
/* Remap memory freed due to conflicts with E820 map */
- if (!xen_feature(XENFEAT_auto_translated_physmap))
- xen_remap_memory();
+ xen_remap_memory();
xen_setup_shared_info();
}
/* Zap identity mapping */
init_level4_pgt[0] = __pgd(0);
- if (!xen_feature(XENFEAT_auto_translated_physmap)) {
- /* Pre-constructed entries are in pfn, so convert to mfn */
- /* L4[272] -> level3_ident_pgt
- * L4[511] -> level3_kernel_pgt */
- convert_pfn_mfn(init_level4_pgt);
+ /* Pre-constructed entries are in pfn, so convert to mfn */
+ /* L4[272] -> level3_ident_pgt */
+ /* L4[511] -> level3_kernel_pgt */
+ convert_pfn_mfn(init_level4_pgt);
- /* L3_i[0] -> level2_ident_pgt */
- convert_pfn_mfn(level3_ident_pgt);
- /* L3_k[510] -> level2_kernel_pgt
- * L3_k[511] -> level2_fixmap_pgt */
- convert_pfn_mfn(level3_kernel_pgt);
+ /* L3_i[0] -> level2_ident_pgt */
+ convert_pfn_mfn(level3_ident_pgt);
+ /* L3_k[510] -> level2_kernel_pgt */
+ /* L3_k[511] -> level2_fixmap_pgt */
+ convert_pfn_mfn(level3_kernel_pgt);
+
+ /* L3_k[511][506] -> level1_fixmap_pgt */
+ convert_pfn_mfn(level2_fixmap_pgt);
- /* L3_k[511][506] -> level1_fixmap_pgt */
- convert_pfn_mfn(level2_fixmap_pgt);
- }
/* We get [511][511] and have Xen's version of level2_kernel_pgt */
l3 = m2v(pgd[pgd_index(__START_KERNEL_map)].pgd);
l2 = m2v(l3[pud_index(__START_KERNEL_map)].pud);
if (i && i < pgd_index(__START_KERNEL_map))
init_level4_pgt[i] = ((pgd_t *)xen_start_info->pt_base)[i];
- if (!xen_feature(XENFEAT_auto_translated_physmap)) {
- /* Make pagetable pieces RO */
- set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
- set_page_prot(level2_ident_pgt, PAGE_KERNEL_RO);
- set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level2_fixmap_pgt, PAGE_KERNEL_RO);
- set_page_prot(level1_fixmap_pgt, PAGE_KERNEL_RO);
-
- /* Pin down new L4 */
- pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE,
- PFN_DOWN(__pa_symbol(init_level4_pgt)));
-
- /* Unpin Xen-provided one */
- pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, PFN_DOWN(__pa(pgd)));
+ /* Make pagetable pieces RO */
+ set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
+ set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
+ set_page_prot(level3_kernel_pgt, PAGE_KERNEL_RO);
+ set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
+ set_page_prot(level2_ident_pgt, PAGE_KERNEL_RO);
+ set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
+ set_page_prot(level2_fixmap_pgt, PAGE_KERNEL_RO);
+ set_page_prot(level1_fixmap_pgt, PAGE_KERNEL_RO);
+
+ /* Pin down new L4 */
+ pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE,
+ PFN_DOWN(__pa_symbol(init_level4_pgt)));
+
+ /* Unpin Xen-provided one */
+ pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, PFN_DOWN(__pa(pgd)));
- /*
- * At this stage there can be no user pgd, and no page
- * structure to attach it to, so make sure we just set kernel
- * pgd.
- */
- xen_mc_batch();
- __xen_write_cr3(true, __pa(init_level4_pgt));
- xen_mc_issue(PARAVIRT_LAZY_CPU);
- } else
- native_write_cr3(__pa(init_level4_pgt));
+ /*
+ * At this stage there can be no user pgd, and no page structure to
+ * attach it to, so make sure we just set kernel pgd.
+ */
+ xen_mc_batch();
+ __xen_write_cr3(true, __pa(init_level4_pgt));
+ xen_mc_issue(PARAVIRT_LAZY_CPU);
/* We can't that easily rip out L3 and L2, as the Xen pagetables are
* set out this way: [L4], [L1], [L2], [L3], [L1], [L1] ... for
static void __init xen_post_allocator_init(void)
{
- if (xen_feature(XENFEAT_auto_translated_physmap))
- return;
-
pv_mmu_ops.set_pte = xen_set_pte;
pv_mmu_ops.set_pmd = xen_set_pmd;
pv_mmu_ops.set_pud = xen_set_pud;
{
x86_init.paging.pagetable_init = xen_pagetable_init;
- if (xen_feature(XENFEAT_auto_translated_physmap))
- return;
-
pv_mmu_ops = xen_mmu_ops;
memset(dummy_mapping, 0xff, PAGE_SIZE);
* this function are redundant and can be ignored.
*/
- if (xen_feature(XENFEAT_auto_translated_physmap))
- return 0;
-
if (unlikely(order > MAX_CONTIG_ORDER))
return -ENOMEM;
int success;
unsigned long vstart;
- if (xen_feature(XENFEAT_auto_translated_physmap))
- return;
-
if (unlikely(order > MAX_CONTIG_ORDER))
return;
}
EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list);
-void blk_mq_abort_requeue_list(struct request_queue *q)
-{
- unsigned long flags;
- LIST_HEAD(rq_list);
-
- spin_lock_irqsave(&q->requeue_lock, flags);
- list_splice_init(&q->requeue_list, &rq_list);
- spin_unlock_irqrestore(&q->requeue_lock, flags);
-
- while (!list_empty(&rq_list)) {
- struct request *rq;
-
- rq = list_first_entry(&rq_list, struct request, queuelist);
- list_del_init(&rq->queuelist);
- blk_mq_end_request(rq, -EIO);
- }
-}
-EXPORT_SYMBOL(blk_mq_abort_requeue_list);
-
struct request *blk_mq_tag_to_rq(struct blk_mq_tags *tags, unsigned int tag)
{
if (tag < tags->nr_tags) {
goto unlock;
}
- if (q->mq_ops)
+ if (q->mq_ops) {
__blk_mq_register_dev(dev, q);
-
- blk_mq_debugfs_register(q);
+ blk_mq_debugfs_register(q);
+ }
kobject_uevent(&q->kobj, KOBJ_ADD);
#define DFL_THROTL_SLICE_HD (HZ / 10)
#define DFL_THROTL_SLICE_SSD (HZ / 50)
#define MAX_THROTL_SLICE (HZ)
-#define DFL_IDLE_THRESHOLD_SSD (1000L) /* 1 ms */
-#define DFL_IDLE_THRESHOLD_HD (100L * 1000) /* 100 ms */
#define MAX_IDLE_TIME (5L * 1000 * 1000) /* 5 s */
-/* default latency target is 0, eg, guarantee IO latency by default */
-#define DFL_LATENCY_TARGET (0)
+#define MIN_THROTL_BPS (320 * 1024)
+#define MIN_THROTL_IOPS (10)
+#define DFL_LATENCY_TARGET (-1L)
+#define DFL_IDLE_THRESHOLD (0)
#define SKIP_LATENCY (((u64)1) << BLK_STAT_RES_SHIFT)
unsigned long last_check_time;
unsigned long latency_target; /* us */
+ unsigned long latency_target_conf; /* us */
/* When did we start a new slice */
unsigned long slice_start[2];
unsigned long slice_end[2];
unsigned long checked_last_finish_time; /* ns / 1024 */
unsigned long avg_idletime; /* ns / 1024 */
unsigned long idletime_threshold; /* us */
+ unsigned long idletime_threshold_conf; /* us */
unsigned int bio_cnt; /* total bios */
unsigned int bad_bio_cnt; /* bios exceeding latency threshold */
unsigned int limit_index;
bool limit_valid[LIMIT_CNT];
- unsigned long dft_idletime_threshold; /* us */
-
unsigned long low_upgrade_time;
unsigned long low_downgrade_time;
td = tg->td;
ret = tg->bps[rw][td->limit_index];
- if (ret == 0 && td->limit_index == LIMIT_LOW)
- return tg->bps[rw][LIMIT_MAX];
+ if (ret == 0 && td->limit_index == LIMIT_LOW) {
+ /* intermediate node or iops isn't 0 */
+ if (!list_empty(&blkg->blkcg->css.children) ||
+ tg->iops[rw][td->limit_index])
+ return U64_MAX;
+ else
+ return MIN_THROTL_BPS;
+ }
if (td->limit_index == LIMIT_MAX && tg->bps[rw][LIMIT_LOW] &&
tg->bps[rw][LIMIT_LOW] != tg->bps[rw][LIMIT_MAX]) {
if (cgroup_subsys_on_dfl(io_cgrp_subsys) && !blkg->parent)
return UINT_MAX;
+
td = tg->td;
ret = tg->iops[rw][td->limit_index];
- if (ret == 0 && tg->td->limit_index == LIMIT_LOW)
- return tg->iops[rw][LIMIT_MAX];
+ if (ret == 0 && tg->td->limit_index == LIMIT_LOW) {
+ /* intermediate node or bps isn't 0 */
+ if (!list_empty(&blkg->blkcg->css.children) ||
+ tg->bps[rw][td->limit_index])
+ return UINT_MAX;
+ else
+ return MIN_THROTL_IOPS;
+ }
if (td->limit_index == LIMIT_MAX && tg->iops[rw][LIMIT_LOW] &&
tg->iops[rw][LIMIT_LOW] != tg->iops[rw][LIMIT_MAX]) {
/* LIMIT_LOW will have default value 0 */
tg->latency_target = DFL_LATENCY_TARGET;
+ tg->latency_target_conf = DFL_LATENCY_TARGET;
+ tg->idletime_threshold = DFL_IDLE_THRESHOLD;
+ tg->idletime_threshold_conf = DFL_IDLE_THRESHOLD;
return &tg->pd;
}
if (cgroup_subsys_on_dfl(io_cgrp_subsys) && blkg->parent)
sq->parent_sq = &blkg_to_tg(blkg->parent)->service_queue;
tg->td = td;
-
- tg->idletime_threshold = td->dft_idletime_threshold;
}
/*
return 0;
}
-static void tg_conf_updated(struct throtl_grp *tg)
+static void tg_conf_updated(struct throtl_grp *tg, bool global)
{
struct throtl_service_queue *sq = &tg->service_queue;
struct cgroup_subsys_state *pos_css;
* restrictions in the whole hierarchy and allows them to bypass
* blk-throttle.
*/
- blkg_for_each_descendant_pre(blkg, pos_css, tg_to_blkg(tg))
- tg_update_has_rules(blkg_to_tg(blkg));
+ blkg_for_each_descendant_pre(blkg, pos_css,
+ global ? tg->td->queue->root_blkg : tg_to_blkg(tg)) {
+ struct throtl_grp *this_tg = blkg_to_tg(blkg);
+ struct throtl_grp *parent_tg;
+
+ tg_update_has_rules(this_tg);
+ /* ignore root/second level */
+ if (!cgroup_subsys_on_dfl(io_cgrp_subsys) || !blkg->parent ||
+ !blkg->parent->parent)
+ continue;
+ parent_tg = blkg_to_tg(blkg->parent);
+ /*
+ * make sure all children has lower idle time threshold and
+ * higher latency target
+ */
+ this_tg->idletime_threshold = min(this_tg->idletime_threshold,
+ parent_tg->idletime_threshold);
+ this_tg->latency_target = max(this_tg->latency_target,
+ parent_tg->latency_target);
+ }
/*
* We're already holding queue_lock and know @tg is valid. Let's
else
*(unsigned int *)((void *)tg + of_cft(of)->private) = v;
- tg_conf_updated(tg);
+ tg_conf_updated(tg, false);
ret = 0;
out_finish:
blkg_conf_finish(&ctx);
tg->iops_conf[READ][off] == iops_dft &&
tg->iops_conf[WRITE][off] == iops_dft &&
(off != LIMIT_LOW ||
- (tg->idletime_threshold == tg->td->dft_idletime_threshold &&
- tg->latency_target == DFL_LATENCY_TARGET)))
+ (tg->idletime_threshold_conf == DFL_IDLE_THRESHOLD &&
+ tg->latency_target_conf == DFL_LATENCY_TARGET)))
return 0;
- if (tg->bps_conf[READ][off] != bps_dft)
+ if (tg->bps_conf[READ][off] != U64_MAX)
snprintf(bufs[0], sizeof(bufs[0]), "%llu",
tg->bps_conf[READ][off]);
- if (tg->bps_conf[WRITE][off] != bps_dft)
+ if (tg->bps_conf[WRITE][off] != U64_MAX)
snprintf(bufs[1], sizeof(bufs[1]), "%llu",
tg->bps_conf[WRITE][off]);
- if (tg->iops_conf[READ][off] != iops_dft)
+ if (tg->iops_conf[READ][off] != UINT_MAX)
snprintf(bufs[2], sizeof(bufs[2]), "%u",
tg->iops_conf[READ][off]);
- if (tg->iops_conf[WRITE][off] != iops_dft)
+ if (tg->iops_conf[WRITE][off] != UINT_MAX)
snprintf(bufs[3], sizeof(bufs[3]), "%u",
tg->iops_conf[WRITE][off]);
if (off == LIMIT_LOW) {
- if (tg->idletime_threshold == ULONG_MAX)
+ if (tg->idletime_threshold_conf == ULONG_MAX)
strcpy(idle_time, " idle=max");
else
snprintf(idle_time, sizeof(idle_time), " idle=%lu",
- tg->idletime_threshold);
+ tg->idletime_threshold_conf);
- if (tg->latency_target == ULONG_MAX)
+ if (tg->latency_target_conf == ULONG_MAX)
strcpy(latency_time, " latency=max");
else
snprintf(latency_time, sizeof(latency_time),
- " latency=%lu", tg->latency_target);
+ " latency=%lu", tg->latency_target_conf);
}
seq_printf(sf, "%s rbps=%s wbps=%s riops=%s wiops=%s%s%s\n",
v[2] = tg->iops_conf[READ][index];
v[3] = tg->iops_conf[WRITE][index];
- idle_time = tg->idletime_threshold;
- latency_time = tg->latency_target;
+ idle_time = tg->idletime_threshold_conf;
+ latency_time = tg->latency_target_conf;
while (true) {
char tok[27]; /* wiops=18446744073709551616 */
char *p;
tg->iops_conf[READ][LIMIT_MAX]);
tg->iops[WRITE][LIMIT_LOW] = min(tg->iops_conf[WRITE][LIMIT_LOW],
tg->iops_conf[WRITE][LIMIT_MAX]);
+ tg->idletime_threshold_conf = idle_time;
+ tg->latency_target_conf = latency_time;
+
+ /* force user to configure all settings for low limit */
+ if (!(tg->bps[READ][LIMIT_LOW] || tg->iops[READ][LIMIT_LOW] ||
+ tg->bps[WRITE][LIMIT_LOW] || tg->iops[WRITE][LIMIT_LOW]) ||
+ tg->idletime_threshold_conf == DFL_IDLE_THRESHOLD ||
+ tg->latency_target_conf == DFL_LATENCY_TARGET) {
+ tg->bps[READ][LIMIT_LOW] = 0;
+ tg->bps[WRITE][LIMIT_LOW] = 0;
+ tg->iops[READ][LIMIT_LOW] = 0;
+ tg->iops[WRITE][LIMIT_LOW] = 0;
+ tg->idletime_threshold = DFL_IDLE_THRESHOLD;
+ tg->latency_target = DFL_LATENCY_TARGET;
+ } else if (index == LIMIT_LOW) {
+ tg->idletime_threshold = tg->idletime_threshold_conf;
+ tg->latency_target = tg->latency_target_conf;
+ }
- if (index == LIMIT_LOW) {
- blk_throtl_update_limit_valid(tg->td);
- if (tg->td->limit_valid[LIMIT_LOW])
+ blk_throtl_update_limit_valid(tg->td);
+ if (tg->td->limit_valid[LIMIT_LOW]) {
+ if (index == LIMIT_LOW)
tg->td->limit_index = LIMIT_LOW;
- tg->idletime_threshold = (idle_time == ULONG_MAX) ?
- ULONG_MAX : idle_time;
- tg->latency_target = (latency_time == ULONG_MAX) ?
- ULONG_MAX : latency_time;
- }
- tg_conf_updated(tg);
+ } else
+ tg->td->limit_index = LIMIT_MAX;
+ tg_conf_updated(tg, index == LIMIT_LOW &&
+ tg->td->limit_valid[LIMIT_LOW]);
ret = 0;
out_finish:
blkg_conf_finish(&ctx);
/*
* cgroup is idle if:
* - single idle is too long, longer than a fixed value (in case user
- * configure a too big threshold) or 4 times of slice
+ * configure a too big threshold) or 4 times of idletime threshold
* - average think time is more than threshold
* - IO latency is largely below threshold
*/
- unsigned long time = jiffies_to_usecs(4 * tg->td->throtl_slice);
-
- time = min_t(unsigned long, MAX_IDLE_TIME, time);
- return (ktime_get_ns() >> 10) - tg->last_finish_time > time ||
- tg->avg_idletime > tg->idletime_threshold ||
- (tg->latency_target && tg->bio_cnt &&
+ unsigned long time;
+ bool ret;
+
+ time = min_t(unsigned long, MAX_IDLE_TIME, 4 * tg->idletime_threshold);
+ ret = tg->latency_target == DFL_LATENCY_TARGET ||
+ tg->idletime_threshold == DFL_IDLE_THRESHOLD ||
+ (ktime_get_ns() >> 10) - tg->last_finish_time > time ||
+ tg->avg_idletime > tg->idletime_threshold ||
+ (tg->latency_target && tg->bio_cnt &&
tg->bad_bio_cnt * 5 < tg->bio_cnt);
+ throtl_log(&tg->service_queue,
+ "avg_idle=%ld, idle_threshold=%ld, bad_bio=%d, total_bio=%d, is_idle=%d, scale=%d",
+ tg->avg_idletime, tg->idletime_threshold, tg->bad_bio_cnt,
+ tg->bio_cnt, ret, tg->td->scale);
+ return ret;
}
static bool throtl_tg_can_upgrade(struct throtl_grp *tg)
struct cgroup_subsys_state *pos_css;
struct blkcg_gq *blkg;
+ throtl_log(&td->service_queue, "upgrade to max");
td->limit_index = LIMIT_MAX;
td->low_upgrade_time = jiffies;
td->scale = 0;
{
td->scale /= 2;
+ throtl_log(&td->service_queue, "downgrade, scale %d", td->scale);
if (td->scale) {
td->low_upgrade_time = jiffies - td->scale * td->throtl_slice;
return;
td->avg_buckets[i].valid = true;
last_latency = td->avg_buckets[i].latency;
}
+
+ for (i = 0; i < LATENCY_BUCKET_SIZE; i++)
+ throtl_log(&td->service_queue,
+ "Latency bucket %d: latency=%ld, valid=%d", i,
+ td->avg_buckets[i].latency, td->avg_buckets[i].valid);
}
#else
static inline void throtl_update_latency_buckets(struct throtl_data *td)
void blk_throtl_register_queue(struct request_queue *q)
{
struct throtl_data *td;
- struct cgroup_subsys_state *pos_css;
- struct blkcg_gq *blkg;
td = q->td;
BUG_ON(!td);
- if (blk_queue_nonrot(q)) {
+ if (blk_queue_nonrot(q))
td->throtl_slice = DFL_THROTL_SLICE_SSD;
- td->dft_idletime_threshold = DFL_IDLE_THRESHOLD_SSD;
- } else {
+ else
td->throtl_slice = DFL_THROTL_SLICE_HD;
- td->dft_idletime_threshold = DFL_IDLE_THRESHOLD_HD;
- }
#ifndef CONFIG_BLK_DEV_THROTTLING_LOW
/* if no low limit, use previous default */
td->throtl_slice = DFL_THROTL_SLICE_HD;
td->track_bio_latency = !q->mq_ops && !q->request_fn;
if (!td->track_bio_latency)
blk_stat_enable_accounting(q);
-
- /*
- * some tg are created before queue is fully initialized, eg, nonrot
- * isn't initialized yet
- */
- rcu_read_lock();
- blkg_for_each_descendant_post(blkg, pos_css, q->root_blkg) {
- struct throtl_grp *tg = blkg_to_tg(blkg);
-
- tg->idletime_threshold = td->dft_idletime_threshold;
- }
- rcu_read_unlock();
}
#ifdef CONFIG_BLK_DEV_THROTTLING_LOW
if (info) {
struct partition_meta_info *pinfo = alloc_part_info(disk);
- if (!pinfo)
+ if (!pinfo) {
+ err = -ENOMEM;
goto out_free_stats;
+ }
memcpy(pinfo, info, sizeof(*info));
p->info = pinfo;
}
continue;
bsd_start = le32_to_cpu(p->p_offset);
bsd_size = le32_to_cpu(p->p_size);
+ if (memcmp(flavour, "bsd\0", 4) == 0)
+ bsd_start += offset;
if (offset == bsd_start && size == bsd_size)
/* full parent partition, we have it already */
continue;
return 0;
}
+static int skcipher_setkey_unaligned(struct crypto_skcipher *tfm,
+ const u8 *key, unsigned int keylen)
+{
+ unsigned long alignmask = crypto_skcipher_alignmask(tfm);
+ struct skcipher_alg *cipher = crypto_skcipher_alg(tfm);
+ u8 *buffer, *alignbuffer;
+ unsigned long absize;
+ int ret;
+
+ absize = keylen + alignmask;
+ buffer = kmalloc(absize, GFP_ATOMIC);
+ if (!buffer)
+ return -ENOMEM;
+
+ alignbuffer = (u8 *)ALIGN((unsigned long)buffer, alignmask + 1);
+ memcpy(alignbuffer, key, keylen);
+ ret = cipher->setkey(tfm, alignbuffer, keylen);
+ kzfree(buffer);
+ return ret;
+}
+
+static int skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct skcipher_alg *cipher = crypto_skcipher_alg(tfm);
+ unsigned long alignmask = crypto_skcipher_alignmask(tfm);
+
+ if (keylen < cipher->min_keysize || keylen > cipher->max_keysize) {
+ crypto_skcipher_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
+ }
+
+ if ((unsigned long)key & alignmask)
+ return skcipher_setkey_unaligned(tfm, key, keylen);
+
+ return cipher->setkey(tfm, key, keylen);
+}
+
static void crypto_skcipher_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_skcipher *skcipher = __crypto_skcipher_cast(tfm);
tfm->__crt_alg->cra_type == &crypto_givcipher_type)
return crypto_init_skcipher_ops_ablkcipher(tfm);
- skcipher->setkey = alg->setkey;
+ skcipher->setkey = skcipher_setkey;
skcipher->encrypt = alg->encrypt;
skcipher->decrypt = alg->decrypt;
skcipher->ivsize = alg->ivsize;
#define ACPI_BUTTON_LID_INIT_IGNORE 0x00
#define ACPI_BUTTON_LID_INIT_OPEN 0x01
+#define ACPI_BUTTON_LID_INIT_METHOD 0x02
#define _COMPONENT ACPI_BUTTON_COMPONENT
ACPI_MODULE_NAME("button");
case ACPI_BUTTON_LID_INIT_OPEN:
(void)acpi_lid_notify_state(device, 1);
break;
+ case ACPI_BUTTON_LID_INIT_METHOD:
+ (void)acpi_lid_update_state(device);
+ break;
case ACPI_BUTTON_LID_INIT_IGNORE:
default:
break;
if (!strncmp(val, "open", sizeof("open") - 1)) {
lid_init_state = ACPI_BUTTON_LID_INIT_OPEN;
pr_info("Notify initial lid state as open\n");
+ } else if (!strncmp(val, "method", sizeof("method") - 1)) {
+ lid_init_state = ACPI_BUTTON_LID_INIT_METHOD;
+ pr_info("Notify initial lid state with _LID return value\n");
} else if (!strncmp(val, "ignore", sizeof("ignore") - 1)) {
lid_init_state = ACPI_BUTTON_LID_INIT_IGNORE;
pr_info("Do not notify initial lid state\n");
switch (lid_init_state) {
case ACPI_BUTTON_LID_INIT_OPEN:
return sprintf(buffer, "open");
+ case ACPI_BUTTON_LID_INIT_METHOD:
+ return sprintf(buffer, "method");
case ACPI_BUTTON_LID_INIT_IGNORE:
return sprintf(buffer, "ignore");
default:
struct nfit_spa *nfit_spa;
/* We only care about memory errors */
- if (!(mce->status & MCACOD))
+ if (!mce_is_memory_error(mce))
return NOTIFY_DONE;
/*
/**
* wakup_source_activate - Mark given wakeup source as active.
* @ws: Wakeup source to handle.
- * @hard: If set, abort suspends in progress and wake up from suspend-to-idle.
*
* Update the @ws' statistics and, if @ws has just been activated, notify the PM
* core of the event by incrementing the counter of of wakeup events being
* processed.
*/
-static void wakeup_source_activate(struct wakeup_source *ws, bool hard)
+static void wakeup_source_activate(struct wakeup_source *ws)
{
unsigned int cec;
"unregistered wakeup source\n"))
return;
- if (hard)
- pm_system_wakeup();
-
ws->active = true;
ws->active_count++;
ws->last_time = ktime_get();
ws->wakeup_count++;
if (!ws->active)
- wakeup_source_activate(ws, hard);
+ wakeup_source_activate(ws);
+
+ if (hard)
+ pm_system_wakeup();
}
/**
}
/* still holds resource->req_lock */
-static int drbd_req_put_completion_ref(struct drbd_request *req, struct bio_and_error *m, int put)
+static void drbd_req_put_completion_ref(struct drbd_request *req, struct bio_and_error *m, int put)
{
struct drbd_device *device = req->device;
D_ASSERT(device, m || (req->rq_state & RQ_POSTPONED));
+ if (!put)
+ return;
+
if (!atomic_sub_and_test(put, &req->completion_ref))
- return 0;
+ return;
drbd_req_complete(req, m);
+ /* local completion may still come in later,
+ * we need to keep the req object around. */
+ if (req->rq_state & RQ_LOCAL_ABORTED)
+ return;
+
if (req->rq_state & RQ_POSTPONED) {
/* don't destroy the req object just yet,
* but queue it for retry */
drbd_restart_request(req);
- return 0;
+ return;
}
- return 1;
+ kref_put(&req->kref, drbd_req_destroy);
}
static void set_if_null_req_next(struct drbd_peer_device *peer_device, struct drbd_request *req)
if (req->i.waiting)
wake_up(&device->misc_wait);
- if (c_put) {
- if (drbd_req_put_completion_ref(req, m, c_put))
- kref_put(&req->kref, drbd_req_destroy);
- } else {
- kref_put(&req->kref, drbd_req_destroy);
- }
+ drbd_req_put_completion_ref(req, m, c_put);
+ kref_put(&req->kref, drbd_req_destroy);
}
static void drbd_report_io_error(struct drbd_device *device, struct drbd_request *req)
}
out:
- if (drbd_req_put_completion_ref(req, &m, 1))
- kref_put(&req->kref, drbd_req_destroy);
+ drbd_req_put_completion_ref(req, &m, 1);
spin_unlock_irq(&resource->req_lock);
/* Even though above is a kref_put(), this is safe.
dev_set_drvdata(&dev->dev, NULL);
- if (be->blkif)
+ if (be->blkif) {
xen_blkif_disconnect(be->blkif);
- /* Put the reference we set in xen_blkif_alloc(). */
- xen_blkif_put(be->blkif);
+ /* Put the reference we set in xen_blkif_alloc(). */
+ xen_blkif_put(be->blkif);
+ }
+
kfree(be->mode);
kfree(be);
return 0;
} else if (!strcmp(str, "auto")) {
parport_nr[0] = LP_PARPORT_AUTO;
} else if (!strcmp(str, "none")) {
- parport_nr[parport_ptr++] = LP_PARPORT_NONE;
+ if (parport_ptr < LP_NO)
+ parport_nr[parport_ptr++] = LP_PARPORT_NONE;
+ else
+ printk(KERN_INFO "lp: too many ports, %s ignored.\n",
+ str);
} else if (!strcmp(str, "reset")) {
reset = 1;
}
static int mmap_mem(struct file *file, struct vm_area_struct *vma)
{
size_t size = vma->vm_end - vma->vm_start;
+ phys_addr_t offset = (phys_addr_t)vma->vm_pgoff << PAGE_SHIFT;
+
+ /* It's illegal to wrap around the end of the physical address space. */
+ if (offset + (phys_addr_t)size < offset)
+ return -EINVAL;
if (!valid_mmap_phys_addr_range(vma->vm_pgoff, size))
return -EINVAL;
If in doubt, say N.
+config ARM_DB8500_CPUFREQ
+ tristate "ST-Ericsson DB8500 cpufreq" if COMPILE_TEST && !ARCH_U8500
+ default ARCH_U8500
+ depends on HAS_IOMEM
+ depends on !CPU_THERMAL || THERMAL
+ help
+ This adds the CPUFreq driver for ST-Ericsson Ux500 (DB8500) SoC
+ series.
+
config ARM_IMX6Q_CPUFREQ
tristate "Freescale i.MX6 cpufreq support"
depends on ARCH_MXC
obj-$(CONFIG_ARM_BRCMSTB_AVS_CPUFREQ) += brcmstb-avs-cpufreq.o
obj-$(CONFIG_ARCH_DAVINCI) += davinci-cpufreq.o
-obj-$(CONFIG_UX500_SOC_DB8500) += dbx500-cpufreq.o
+obj-$(CONFIG_ARM_DB8500_CPUFREQ) += dbx500-cpufreq.o
obj-$(CONFIG_ARM_EXYNOS5440_CPUFREQ) += exynos5440-cpufreq.o
obj-$(CONFIG_ARM_HIGHBANK_CPUFREQ) += highbank-cpufreq.o
obj-$(CONFIG_ARM_IMX6Q_CPUFREQ) += imx6q-cpufreq.o
}
EXPORT_SYMBOL_GPL(dax_read_unlock);
+#ifdef CONFIG_BLOCK
int bdev_dax_pgoff(struct block_device *bdev, sector_t sector, size_t size,
pgoff_t *pgoff)
{
return 0;
}
EXPORT_SYMBOL_GPL(__bdev_dax_supported);
+#endif
/**
* struct dax_device - anchor object for dax services
static void debug_display_dimm_sizes_df(struct amd64_pvt *pvt, u8 ctrl)
{
- u32 *dcsb = ctrl ? pvt->csels[1].csbases : pvt->csels[0].csbases;
- int dimm, size0, size1;
+ int dimm, size0, size1, cs0, cs1;
edac_printk(KERN_DEBUG, EDAC_MC, "UMC%d chip selects:\n", ctrl);
for (dimm = 0; dimm < 4; dimm++) {
size0 = 0;
+ cs0 = dimm * 2;
- if (dcsb[dimm*2] & DCSB_CS_ENABLE)
- size0 = pvt->ops->dbam_to_cs(pvt, ctrl, 0, dimm);
+ if (csrow_enabled(cs0, ctrl, pvt))
+ size0 = pvt->ops->dbam_to_cs(pvt, ctrl, 0, cs0);
size1 = 0;
- if (dcsb[dimm*2 + 1] & DCSB_CS_ENABLE)
- size1 = pvt->ops->dbam_to_cs(pvt, ctrl, 0, dimm);
+ cs1 = dimm * 2 + 1;
+
+ if (csrow_enabled(cs1, ctrl, pvt))
+ size1 = pvt->ops->dbam_to_cs(pvt, ctrl, 0, cs1);
amd64_info(EDAC_MC ": %d: %5dMB %d: %5dMB\n",
- dimm * 2, size0,
- dimm * 2 + 1, size1);
+ cs0, size0,
+ cs1, size1);
}
}
* encompasses
*
*/
-static u32 get_csrow_nr_pages(struct amd64_pvt *pvt, u8 dct, int csrow_nr)
+static u32 get_csrow_nr_pages(struct amd64_pvt *pvt, u8 dct, int csrow_nr_orig)
{
- u32 cs_mode, nr_pages;
u32 dbam = dct ? pvt->dbam1 : pvt->dbam0;
+ int csrow_nr = csrow_nr_orig;
+ u32 cs_mode, nr_pages;
+ if (!pvt->umc)
+ csrow_nr >>= 1;
- /*
- * The math on this doesn't look right on the surface because x/2*4 can
- * be simplified to x*2 but this expression makes use of the fact that
- * it is integral math where 1/2=0. This intermediate value becomes the
- * number of bits to shift the DBAM register to extract the proper CSROW
- * field.
- */
- cs_mode = DBAM_DIMM(csrow_nr / 2, dbam);
+ cs_mode = DBAM_DIMM(csrow_nr, dbam);
- nr_pages = pvt->ops->dbam_to_cs(pvt, dct, cs_mode, (csrow_nr / 2))
- << (20 - PAGE_SHIFT);
+ nr_pages = pvt->ops->dbam_to_cs(pvt, dct, cs_mode, csrow_nr);
+ nr_pages <<= 20 - PAGE_SHIFT;
edac_dbg(0, "csrow: %d, channel: %d, DBAM idx: %d\n",
- csrow_nr, dct, cs_mode);
+ csrow_nr_orig, dct, cs_mode);
edac_dbg(0, "nr_pages/channel: %u\n", nr_pages);
return nr_pages;
if (sscanf(name, "dump-type%u-%u-%d-%lu-%c",
&record->type, &part, &cnt, &time, &data_type) == 5) {
record->id = generic_id(time, part, cnt);
+ record->part = part;
record->count = cnt;
record->time.tv_sec = time;
record->time.tv_nsec = 0;
} else if (sscanf(name, "dump-type%u-%u-%d-%lu",
&record->type, &part, &cnt, &time) == 4) {
record->id = generic_id(time, part, cnt);
+ record->part = part;
record->count = cnt;
record->time.tv_sec = time;
record->time.tv_nsec = 0;
* multiple logs, remains.
*/
record->id = generic_id(time, part, 0);
+ record->part = part;
record->count = 0;
record->time.tv_sec = time;
record->time.tv_nsec = 0;
* efi_pstore_sysfs_entry_iter
*
* @record: pstore record to pass to callback
- * @pos: entry to begin iterating from
*
* You MUST call efivar_enter_iter_begin() before this function, and
* efivar_entry_iter_end() afterwards.
*
- * It is possible to begin iteration from an arbitrary entry within
- * the list by passing @pos. @pos is updated on return to point to
- * the next entry of the last one passed to efi_pstore_read_func().
- * To begin iterating from the beginning of the list @pos must be %NULL.
*/
-static int efi_pstore_sysfs_entry_iter(struct pstore_record *record,
- struct efivar_entry **pos)
+static int efi_pstore_sysfs_entry_iter(struct pstore_record *record)
{
+ struct efivar_entry **pos = (struct efivar_entry **)&record->psi->data;
struct efivar_entry *entry, *n;
struct list_head *head = &efivar_sysfs_list;
int size = 0;
*/
static ssize_t efi_pstore_read(struct pstore_record *record)
{
- struct efivar_entry *entry = (struct efivar_entry *)record->psi->data;
ssize_t size;
record->buf = kzalloc(EFIVARS_DATA_SIZE_MAX, GFP_KERNEL);
size = -EINTR;
goto out;
}
- size = efi_pstore_sysfs_entry_iter(record, &entry);
+ size = efi_pstore_sysfs_entry_iter(record);
efivar_entry_iter_end();
out:
efi_guid_t vendor = LINUX_EFI_CRASH_GUID;
int i, ret = 0;
+ record->time.tv_sec = get_seconds();
+ record->time.tv_nsec = 0;
+
+ record->id = generic_id(record->time.tv_sec, record->part,
+ record->count);
+
snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lu-%c",
record->type, record->part, record->count,
- get_seconds(), record->compressed ? 'C' : 'D');
+ record->time.tv_sec, record->compressed ? 'C' : 'D');
for (i = 0; i < DUMP_NAME_LEN; i++)
efi_name[i] = name[i];
if (record->reason == KMSG_DUMP_OOPS)
efivar_run_worker();
- record->id = record->part;
return ret;
};
* holding multiple logs, remains.
*/
snprintf(name_old, sizeof(name_old), "dump-type%u-%u-%lu",
- ed->record->type, (unsigned int)ed->record->id,
+ ed->record->type, ed->record->part,
ed->record->time.tv_sec);
for (i = 0; i < DUMP_NAME_LEN; i++)
char name[DUMP_NAME_LEN];
efi_char16_t efi_name[DUMP_NAME_LEN];
int found, i;
- unsigned int part;
- do_div(record->id, 1000);
- part = do_div(record->id, 100);
snprintf(name, sizeof(name), "dump-type%u-%u-%d-%lu",
record->type, record->part, record->count,
record->time.tv_sec);
return VPD_OK;
info = kzalloc(sizeof(*info), GFP_KERNEL);
- info->key = kzalloc(key_len + 1, GFP_KERNEL);
- if (!info->key)
+ if (!info)
return -ENOMEM;
+ info->key = kzalloc(key_len + 1, GFP_KERNEL);
+ if (!info->key) {
+ ret = -ENOMEM;
+ goto free_info;
+ }
memcpy(info->key, key, key_len);
list_add_tail(&info->list, &sec->attribs);
ret = sysfs_create_bin_file(sec->kobj, &info->bin_attr);
- if (ret) {
- kfree(info->key);
- return ret;
- }
+ if (ret)
+ goto free_info_key;
return 0;
+
+free_info_key:
+ kfree(info->key);
+free_info:
+ kfree(info);
+
+ return ret;
}
static void vpd_section_attrib_destroy(struct vpd_section *sec)
info->debug_buffer[info->debug_region_size] = 0;
info->d = debugfs_create_file(strncat(debug_name, dev_name(dev),
- sizeof(debug_name)),
+ sizeof(debug_name) -
+ sizeof("ti_sci_debug@")),
0444, NULL, info, &ti_sci_debug_fops);
if (IS_ERR(info->d))
return PTR_ERR(info->d);
drm_framebuffer.o drm_connector.o drm_blend.o \
drm_encoder.o drm_mode_object.o drm_property.o \
drm_plane.o drm_color_mgmt.o drm_print.o \
- drm_dumb_buffers.o drm_mode_config.o drm_vblank.o
+ drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
+ drm_syncobj.o
drm-$(CONFIG_DRM_LIB_RANDOM) += lib/drm_random.o
drm-$(CONFIG_DRM_VM) += drm_vm.o
void amdgpu_fbdev_restore_mode(struct amdgpu_device *adev)
{
- struct amdgpu_fbdev *afbdev = adev->mode_info.rfbdev;
+ struct amdgpu_fbdev *afbdev;
struct drm_fb_helper *fb_helper;
int ret;
+ if (!adev)
+ return;
+
+ afbdev = adev->mode_info.rfbdev;
+
if (!afbdev)
return;
mutex_unlock(&id_mgr->lock);
}
- if (gds_switch_needed) {
+ if (ring->funcs->emit_gds_switch && gds_switch_needed) {
id->gds_base = job->gds_base;
id->gds_size = job->gds_size;
id->gws_base = job->gws_base;
struct amdgpu_vm_id_manager *id_mgr = &adev->vm_manager.id_mgr[vmhub];
struct amdgpu_vm_id *id = &id_mgr->ids[vmid];
+ atomic64_set(&id->owner, 0);
id->gds_base = 0;
id->gds_size = 0;
id->gws_base = 0;
id->oa_size = 0;
}
+/**
+ * amdgpu_vm_reset_all_id - reset VMID to zero
+ *
+ * @adev: amdgpu device structure
+ *
+ * Reset VMID to force flush on next use
+ */
+void amdgpu_vm_reset_all_ids(struct amdgpu_device *adev)
+{
+ unsigned i, j;
+
+ for (i = 0; i < AMDGPU_MAX_VMHUBS; ++i) {
+ struct amdgpu_vm_id_manager *id_mgr =
+ &adev->vm_manager.id_mgr[i];
+
+ for (j = 1; j < id_mgr->num_ids; ++j)
+ amdgpu_vm_reset_id(adev, i, j);
+ }
+}
+
/**
* amdgpu_vm_bo_find - find the bo_va for a specific vm & bo
*
for (i = 0; i < AMDGPU_MAX_RINGS; ++i)
adev->vm_manager.seqno[i] = 0;
-
atomic_set(&adev->vm_manager.vm_pte_next_ring, 0);
atomic64_set(&adev->vm_manager.client_counter, 0);
spin_lock_init(&adev->vm_manager.prt_lock);
int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job);
void amdgpu_vm_reset_id(struct amdgpu_device *adev, unsigned vmhub,
unsigned vmid);
+void amdgpu_vm_reset_all_ids(struct amdgpu_device *adev);
int amdgpu_vm_update_directories(struct amdgpu_device *adev,
struct amdgpu_vm *vm);
int amdgpu_vm_clear_freed(struct amdgpu_device *adev,
u32 vblank_time = amdgpu_dpm_get_vblank_time(adev);
u32 switch_limit = adev->mc.vram_type == AMDGPU_VRAM_TYPE_GDDR5 ? 450 : 300;
+ /* disable mclk switching if the refresh is >120Hz, even if the
+ * blanking period would allow it
+ */
+ if (amdgpu_dpm_get_vrefresh(adev) > 120)
+ return true;
+
if (vblank_time < switch_limit)
return true;
else
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- if (adev->vm_manager.enabled) {
- gmc_v6_0_vm_fini(adev);
- adev->vm_manager.enabled = false;
- }
gmc_v6_0_hw_fini(adev);
return 0;
if (r)
return r;
- if (!adev->vm_manager.enabled) {
- r = gmc_v6_0_vm_init(adev);
- if (r) {
- dev_err(adev->dev, "vm manager initialization failed (%d).\n", r);
- return r;
- }
- adev->vm_manager.enabled = true;
- }
+ amdgpu_vm_reset_all_ids(adev);
- return r;
+ return 0;
}
static bool gmc_v6_0_is_idle(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- if (adev->vm_manager.enabled) {
- gmc_v7_0_vm_fini(adev);
- adev->vm_manager.enabled = false;
- }
gmc_v7_0_hw_fini(adev);
return 0;
if (r)
return r;
- if (!adev->vm_manager.enabled) {
- r = gmc_v7_0_vm_init(adev);
- if (r) {
- dev_err(adev->dev, "vm manager initialization failed (%d).\n", r);
- return r;
- }
- adev->vm_manager.enabled = true;
- }
+ amdgpu_vm_reset_all_ids(adev);
- return r;
+ return 0;
}
static bool gmc_v7_0_is_idle(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- if (adev->vm_manager.enabled) {
- gmc_v8_0_vm_fini(adev);
- adev->vm_manager.enabled = false;
- }
gmc_v8_0_hw_fini(adev);
return 0;
if (r)
return r;
- if (!adev->vm_manager.enabled) {
- r = gmc_v8_0_vm_init(adev);
- if (r) {
- dev_err(adev->dev, "vm manager initialization failed (%d).\n", r);
- return r;
- }
- adev->vm_manager.enabled = true;
- }
+ amdgpu_vm_reset_all_ids(adev);
- return r;
+ return 0;
}
static bool gmc_v8_0_is_idle(void *handle)
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
- if (adev->vm_manager.enabled) {
- gmc_v9_0_vm_fini(adev);
- adev->vm_manager.enabled = false;
- }
gmc_v9_0_hw_fini(adev);
return 0;
if (r)
return r;
- if (!adev->vm_manager.enabled) {
- r = gmc_v9_0_vm_init(adev);
- if (r) {
- dev_err(adev->dev,
- "vm manager initialization failed (%d).\n", r);
- return r;
- }
- adev->vm_manager.enabled = true;
- }
+ amdgpu_vm_reset_all_ids(adev);
- return r;
+ return 0;
}
static bool gmc_v9_0_is_idle(void *handle)
return sizeof(struct smu7_power_state);
}
+static int smu7_vblank_too_short(struct pp_hwmgr *hwmgr,
+ uint32_t vblank_time_us)
+{
+ struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend);
+ uint32_t switch_limit_us;
+
+ switch (hwmgr->chip_id) {
+ case CHIP_POLARIS10:
+ case CHIP_POLARIS11:
+ case CHIP_POLARIS12:
+ switch_limit_us = data->is_memory_gddr5 ? 190 : 150;
+ break;
+ default:
+ switch_limit_us = data->is_memory_gddr5 ? 450 : 150;
+ break;
+ }
+
+ if (vblank_time_us < switch_limit_us)
+ return true;
+ else
+ return false;
+}
static int smu7_apply_state_adjust_rules(struct pp_hwmgr *hwmgr,
struct pp_power_state *request_ps,
bool disable_mclk_switching;
bool disable_mclk_switching_for_frame_lock;
struct cgs_display_info info = {0};
+ struct cgs_mode_info mode_info = {0};
const struct phm_clock_and_voltage_limits *max_limits;
uint32_t i;
struct smu7_hwmgr *data = (struct smu7_hwmgr *)(hwmgr->backend);
int32_t count;
int32_t stable_pstate_sclk = 0, stable_pstate_mclk = 0;
+ info.mode_info = &mode_info;
data->battery_state = (PP_StateUILabel_Battery ==
request_ps->classification.ui_label);
cgs_get_active_displays_info(hwmgr->device, &info);
- /*TO DO result = PHM_CheckVBlankTime(hwmgr, &vblankTooShort);*/
-
minimum_clocks.engineClock = hwmgr->display_config.min_core_set_clock;
minimum_clocks.memoryClock = hwmgr->display_config.min_mem_set_clock;
PHM_PlatformCaps_DisableMclkSwitchingForFrameLock);
- disable_mclk_switching = (1 < info.display_count) ||
- disable_mclk_switching_for_frame_lock;
+ disable_mclk_switching = ((1 < info.display_count) ||
+ disable_mclk_switching_for_frame_lock ||
+ smu7_vblank_too_short(hwmgr, mode_info.vblank_time_us) ||
+ (mode_info.refresh_rate > 120));
sclk = smu7_ps->performance_levels[0].engine_clock;
mclk = smu7_ps->performance_levels[0].memory_clock;
enum pp_clock_type type, uint32_t mask)
{
struct vega10_hwmgr *data = (struct vega10_hwmgr *)(hwmgr->backend);
- uint32_t i;
+ int i;
if (hwmgr->dpm_level != AMD_DPM_FORCED_LEVEL_MANUAL)
return -EINVAL;
*/
#include <drm/drmP.h>
+#include <drm/drm_atomic.h>
#include <drm/drm_atomic_helper.h>
#include <drm/drm_crtc.h>
#include <drm/drm_crtc_helper.h>
static int hdlcd_plane_atomic_check(struct drm_plane *plane,
struct drm_plane_state *state)
{
- u32 src_w, src_h;
+ struct drm_rect clip = { 0 };
+ struct drm_crtc_state *crtc_state;
+ u32 src_h = state->src_h >> 16;
- src_w = state->src_w >> 16;
- src_h = state->src_h >> 16;
+ /* only the HDLCD_REG_FB_LINE_COUNT register has a limit */
+ if (src_h >= HDLCD_MAX_YRES) {
+ DRM_DEBUG_KMS("Invalid source width: %d\n", src_h);
+ return -EINVAL;
+ }
+
+ if (!state->fb || !state->crtc)
+ return 0;
- /* we can't do any scaling of the plane source */
- if ((src_w != state->crtc_w) || (src_h != state->crtc_h))
+ crtc_state = drm_atomic_get_existing_crtc_state(state->state,
+ state->crtc);
+ if (!crtc_state) {
+ DRM_DEBUG_KMS("Invalid crtc state\n");
return -EINVAL;
+ }
- return 0;
+ clip.x2 = crtc_state->adjusted_mode.hdisplay;
+ clip.y2 = crtc_state->adjusted_mode.vdisplay;
+
+ return drm_plane_helper_check_state(state, &clip,
+ DRM_PLANE_HELPER_NO_SCALING,
+ DRM_PLANE_HELPER_NO_SCALING,
+ false, true);
}
static void hdlcd_plane_atomic_update(struct drm_plane *plane,
struct drm_framebuffer *fb = plane->state->fb;
struct hdlcd_drm_private *hdlcd;
struct drm_gem_cma_object *gem;
- u32 src_w, src_h, dest_w, dest_h;
+ u32 src_x, src_y, dest_h;
dma_addr_t scanout_start;
if (!fb)
return;
- src_w = plane->state->src_w >> 16;
- src_h = plane->state->src_h >> 16;
- dest_w = plane->state->crtc_w;
- dest_h = plane->state->crtc_h;
+ src_x = plane->state->src.x1 >> 16;
+ src_y = plane->state->src.y1 >> 16;
+ dest_h = drm_rect_height(&plane->state->dst);
gem = drm_fb_cma_get_gem_obj(fb, 0);
+
scanout_start = gem->paddr + fb->offsets[0] +
- plane->state->crtc_y * fb->pitches[0] +
- plane->state->crtc_x *
- fb->format->cpp[0];
+ src_y * fb->pitches[0] +
+ src_x * fb->format->cpp[0];
hdlcd = plane->dev->dev_private;
hdlcd_write(hdlcd, HDLCD_REG_FB_LINE_LENGTH, fb->pitches[0]);
formats, ARRAY_SIZE(formats),
DRM_PLANE_TYPE_PRIMARY, NULL);
if (ret) {
- devm_kfree(drm->dev, plane);
return ERR_PTR(ret);
}
&hdlcd_crtc_funcs, NULL);
if (ret) {
hdlcd_plane_destroy(primary);
- devm_kfree(drm->dev, primary);
return ret;
}
.destroy = drm_encoder_cleanup,
};
-static int atmel_hlcdc_attach_endpoint(struct drm_device *dev,
- const struct device_node *np)
+static int atmel_hlcdc_attach_endpoint(struct drm_device *dev, int endpoint)
{
struct drm_encoder *encoder;
struct drm_panel *panel;
struct drm_bridge *bridge;
int ret;
+ ret = drm_of_find_panel_or_bridge(dev->dev->of_node, 0, endpoint,
+ &panel, &bridge);
+ if (ret)
+ return ret;
+
encoder = devm_kzalloc(dev->dev, sizeof(*encoder), GFP_KERNEL);
if (!encoder)
return -EINVAL;
encoder->possible_crtcs = 0x1;
- ret = drm_of_find_panel_or_bridge(np, 0, 0, &panel, &bridge);
- if (ret)
- return ret;
-
if (panel) {
bridge = drm_panel_bridge_add(panel, DRM_MODE_CONNECTOR_Unknown);
if (IS_ERR(bridge))
int atmel_hlcdc_create_outputs(struct drm_device *dev)
{
- struct device_node *remote;
- int ret = -ENODEV;
- int endpoint = 0;
-
- while (true) {
- /* Loop thru possible multiple connections to the output */
- remote = of_graph_get_remote_node(dev->dev->of_node, 0,
- endpoint++);
- if (!remote)
- break;
-
- ret = atmel_hlcdc_attach_endpoint(dev, remote);
- of_node_put(remote);
- if (ret)
- return ret;
- }
+ int endpoint, ret = 0;
+
+ for (endpoint = 0; !ret; endpoint++)
+ ret = atmel_hlcdc_attach_endpoint(dev, endpoint);
+
+ /* At least one device was successfully attached.*/
+ if (ret == -ENODEV && endpoint)
+ return 0;
return ret;
}
if (drm_core_check_feature(dev, DRIVER_GEM))
drm_gem_open(dev, priv);
+ if (drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ drm_syncobj_open(priv);
+
if (drm_core_check_feature(dev, DRIVER_PRIME))
drm_prime_init_file_private(&priv->prime);
out_prime_destroy:
if (drm_core_check_feature(dev, DRIVER_PRIME))
drm_prime_destroy_file_private(&priv->prime);
+ if (drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ drm_syncobj_release(priv);
if (drm_core_check_feature(dev, DRIVER_GEM))
drm_gem_release(dev, priv);
put_pid(priv->pid);
drm_property_destroy_user_blobs(dev, file_priv);
}
+ if (drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ drm_syncobj_release(file_priv);
+
if (drm_core_check_feature(dev, DRIVER_GEM))
drm_gem_release(dev, file_priv);
{
return 0;
}
+
#endif
+
+/* drm_syncobj.c */
+void drm_syncobj_open(struct drm_file *file_private);
+void drm_syncobj_release(struct drm_file *file_private);
+int drm_syncobj_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
+int drm_syncobj_destroy_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
+int drm_syncobj_handle_to_fd_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
+int drm_syncobj_fd_to_handle_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private);
req->value |= dev->driver->prime_fd_to_handle ? DRM_PRIME_CAP_IMPORT : 0;
req->value |= dev->driver->prime_handle_to_fd ? DRM_PRIME_CAP_EXPORT : 0;
return 0;
+ case DRM_CAP_SYNCOBJ:
+ req->value = drm_core_check_feature(dev, DRIVER_SYNCOBJ);
+ return 0;
}
/* Other caps only work with KMS drivers */
DRM_IOCTL_DEF(DRM_IOCTL_MODE_ATOMIC, drm_mode_atomic_ioctl, DRM_MASTER|DRM_CONTROL_ALLOW|DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_MODE_CREATEPROPBLOB, drm_mode_createblob_ioctl, DRM_CONTROL_ALLOW|DRM_UNLOCKED),
DRM_IOCTL_DEF(DRM_IOCTL_MODE_DESTROYPROPBLOB, drm_mode_destroyblob_ioctl, DRM_CONTROL_ALLOW|DRM_UNLOCKED),
+
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_CREATE, drm_syncobj_create_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_DESTROY, drm_syncobj_destroy_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_HANDLE_TO_FD, drm_syncobj_handle_to_fd_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
+ DRM_IOCTL_DEF(DRM_IOCTL_SYNCOBJ_FD_TO_HANDLE, drm_syncobj_fd_to_handle_ioctl,
+ DRM_UNLOCKED|DRM_RENDER_ALLOW),
};
#define DRM_CORE_IOCTL_COUNT ARRAY_SIZE( drm_ioctls )
}
out:
- if (ret && crtc->funcs->page_flip_target)
- drm_crtc_vblank_put(crtc);
if (fb)
drm_framebuffer_put(fb);
if (crtc->primary->old_fb)
drm_modeset_drop_locks(&ctx);
drm_modeset_acquire_fini(&ctx);
+ if (ret && crtc->funcs->page_flip_target)
+ drm_crtc_vblank_put(crtc);
+
return ret;
}
--- /dev/null
+/*
+ * Copyright 2017 Red Hat
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ * Authors:
+ *
+ */
+
+/**
+ * DOC: Overview
+ *
+ * DRM synchronisation objects (syncobj) are a persistent objects,
+ * that contain an optional fence. The fence can be updated with a new
+ * fence, or be NULL.
+ *
+ * syncobj's can be export to fd's and back, these fd's are opaque and
+ * have no other use case, except passing the syncobj between processes.
+ *
+ * Their primary use-case is to implement Vulkan fences and semaphores.
+ *
+ * syncobj have a kref reference count, but also have an optional file.
+ * The file is only created once the syncobj is exported.
+ * The file takes a reference on the kref.
+ */
+
+#include <drm/drmP.h>
+#include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/anon_inodes.h>
+#include <linux/sync_file.h>
+
+#include "drm_internal.h"
+#include <drm/drm_syncobj.h>
+
+/**
+ * drm_syncobj_find - lookup and reference a sync object.
+ * @file_private: drm file private pointer
+ * @handle: sync object handle to lookup.
+ *
+ * Returns a reference to the syncobj pointed to by handle or NULL.
+ */
+struct drm_syncobj *drm_syncobj_find(struct drm_file *file_private,
+ u32 handle)
+{
+ struct drm_syncobj *syncobj;
+
+ spin_lock(&file_private->syncobj_table_lock);
+
+ /* Check if we currently have a reference on the object */
+ syncobj = idr_find(&file_private->syncobj_idr, handle);
+ if (syncobj)
+ drm_syncobj_get(syncobj);
+
+ spin_unlock(&file_private->syncobj_table_lock);
+
+ return syncobj;
+}
+EXPORT_SYMBOL(drm_syncobj_find);
+
+/**
+ * drm_syncobj_replace_fence - replace fence in a sync object.
+ * @file_private: drm file private pointer.
+ * @syncobj: Sync object to replace fence in
+ * @fence: fence to install in sync file.
+ *
+ * This replaces the fence on a sync object.
+ */
+void drm_syncobj_replace_fence(struct drm_file *file_private,
+ struct drm_syncobj *syncobj,
+ struct dma_fence *fence)
+{
+ struct dma_fence *old_fence = NULL;
+
+ if (fence)
+ dma_fence_get(fence);
+ old_fence = xchg(&syncobj->fence, fence);
+
+ dma_fence_put(old_fence);
+}
+EXPORT_SYMBOL(drm_syncobj_replace_fence);
+
+int drm_syncobj_fence_get(struct drm_file *file_private,
+ u32 handle,
+ struct dma_fence **fence)
+{
+ struct drm_syncobj *syncobj = drm_syncobj_find(file_private, handle);
+ int ret = 0;
+
+ if (!syncobj)
+ return -ENOENT;
+
+ *fence = dma_fence_get(syncobj->fence);
+ if (!*fence) {
+ ret = -EINVAL;
+ }
+ drm_syncobj_put(syncobj);
+ return ret;
+}
+EXPORT_SYMBOL(drm_syncobj_fence_get);
+
+/**
+ * drm_syncobj_free - free a sync object.
+ * @kref: kref to free.
+ *
+ * Only to be called from kref_put in drm_syncobj_put.
+ */
+void drm_syncobj_free(struct kref *kref)
+{
+ struct drm_syncobj *syncobj = container_of(kref,
+ struct drm_syncobj,
+ refcount);
+ dma_fence_put(syncobj->fence);
+ kfree(syncobj);
+}
+EXPORT_SYMBOL(drm_syncobj_free);
+
+static int drm_syncobj_create(struct drm_file *file_private,
+ u32 *handle)
+{
+ int ret;
+ struct drm_syncobj *syncobj;
+
+ syncobj = kzalloc(sizeof(struct drm_syncobj), GFP_KERNEL);
+ if (!syncobj)
+ return -ENOMEM;
+
+ kref_init(&syncobj->refcount);
+
+ idr_preload(GFP_KERNEL);
+ spin_lock(&file_private->syncobj_table_lock);
+ ret = idr_alloc(&file_private->syncobj_idr, syncobj, 1, 0, GFP_NOWAIT);
+ spin_unlock(&file_private->syncobj_table_lock);
+
+ idr_preload_end();
+
+ if (ret < 0) {
+ drm_syncobj_put(syncobj);
+ return ret;
+ }
+
+ *handle = ret;
+ return 0;
+}
+
+static int drm_syncobj_destroy(struct drm_file *file_private,
+ u32 handle)
+{
+ struct drm_syncobj *syncobj;
+
+ spin_lock(&file_private->syncobj_table_lock);
+ syncobj = idr_remove(&file_private->syncobj_idr, handle);
+ spin_unlock(&file_private->syncobj_table_lock);
+
+ if (!syncobj)
+ return -EINVAL;
+
+ drm_syncobj_put(syncobj);
+ return 0;
+}
+
+static int drm_syncobj_file_release(struct inode *inode, struct file *file)
+{
+ struct drm_syncobj *syncobj = file->private_data;
+
+ drm_syncobj_put(syncobj);
+ return 0;
+}
+
+static const struct file_operations drm_syncobj_file_fops = {
+ .release = drm_syncobj_file_release,
+};
+
+static int drm_syncobj_alloc_file(struct drm_syncobj *syncobj)
+{
+ struct file *file = anon_inode_getfile("syncobj_file",
+ &drm_syncobj_file_fops,
+ syncobj, 0);
+ if (IS_ERR(file))
+ return PTR_ERR(file);
+
+ drm_syncobj_get(syncobj);
+ if (cmpxchg(&syncobj->file, NULL, file)) {
+ /* lost the race */
+ fput(file);
+ }
+
+ return 0;
+}
+
+static int drm_syncobj_handle_to_fd(struct drm_file *file_private,
+ u32 handle, int *p_fd)
+{
+ struct drm_syncobj *syncobj = drm_syncobj_find(file_private, handle);
+ int ret;
+ int fd;
+
+ if (!syncobj)
+ return -EINVAL;
+
+ fd = get_unused_fd_flags(O_CLOEXEC);
+ if (fd < 0) {
+ drm_syncobj_put(syncobj);
+ return fd;
+ }
+
+ if (!syncobj->file) {
+ ret = drm_syncobj_alloc_file(syncobj);
+ if (ret)
+ goto out_put_fd;
+ }
+ fd_install(fd, syncobj->file);
+ drm_syncobj_put(syncobj);
+ *p_fd = fd;
+ return 0;
+out_put_fd:
+ put_unused_fd(fd);
+ drm_syncobj_put(syncobj);
+ return ret;
+}
+
+static struct drm_syncobj *drm_syncobj_fdget(int fd)
+{
+ struct file *file = fget(fd);
+
+ if (!file)
+ return NULL;
+ if (file->f_op != &drm_syncobj_file_fops)
+ goto err;
+
+ return file->private_data;
+err:
+ fput(file);
+ return NULL;
+};
+
+static int drm_syncobj_fd_to_handle(struct drm_file *file_private,
+ int fd, u32 *handle)
+{
+ struct drm_syncobj *syncobj = drm_syncobj_fdget(fd);
+ int ret;
+
+ if (!syncobj)
+ return -EINVAL;
+
+ /* take a reference to put in the idr */
+ drm_syncobj_get(syncobj);
+
+ idr_preload(GFP_KERNEL);
+ spin_lock(&file_private->syncobj_table_lock);
+ ret = idr_alloc(&file_private->syncobj_idr, syncobj, 1, 0, GFP_NOWAIT);
+ spin_unlock(&file_private->syncobj_table_lock);
+ idr_preload_end();
+
+ if (ret < 0) {
+ fput(syncobj->file);
+ return ret;
+ }
+ *handle = ret;
+ return 0;
+}
+
+int drm_syncobj_import_sync_file_fence(struct drm_file *file_private,
+ int fd, int handle)
+{
+ struct dma_fence *fence = sync_file_get_fence(fd);
+ struct drm_syncobj *syncobj;
+
+ if (!fence)
+ return -EINVAL;
+
+ syncobj = drm_syncobj_find(file_private, handle);
+ if (!syncobj) {
+ dma_fence_put(fence);
+ return -ENOENT;
+ }
+
+ drm_syncobj_replace_fence(file_private, syncobj, fence);
+ dma_fence_put(fence);
+ drm_syncobj_put(syncobj);
+ return 0;
+}
+
+int drm_syncobj_export_sync_file(struct drm_file *file_private,
+ int handle, int *p_fd)
+{
+ int ret;
+ struct dma_fence *fence;
+ struct sync_file *sync_file;
+ int fd = get_unused_fd_flags(O_CLOEXEC);
+
+ if (fd < 0)
+ return fd;
+
+ ret = drm_syncobj_fence_get(file_private, handle, &fence);
+ if (ret)
+ goto err_put_fd;
+
+ sync_file = sync_file_create(fence);
+
+ dma_fence_put(fence);
+
+ if (!sync_file) {
+ ret = -EINVAL;
+ goto err_put_fd;
+ }
+
+ fd_install(fd, sync_file->file);
+
+ *p_fd = fd;
+ return 0;
+err_put_fd:
+ put_unused_fd(fd);
+ return ret;
+}
+/**
+ * drm_syncobj_open - initalizes syncobj file-private structures at devnode open time
+ * @dev: drm_device which is being opened by userspace
+ * @file_private: drm file-private structure to set up
+ *
+ * Called at device open time, sets up the structure for handling refcounting
+ * of sync objects.
+ */
+void
+drm_syncobj_open(struct drm_file *file_private)
+{
+ idr_init(&file_private->syncobj_idr);
+ spin_lock_init(&file_private->syncobj_table_lock);
+}
+
+static int
+drm_syncobj_release_handle(int id, void *ptr, void *data)
+{
+ struct drm_syncobj *syncobj = ptr;
+
+ drm_syncobj_put(syncobj);
+ return 0;
+}
+
+/**
+ * drm_syncobj_release - release file-private sync object resources
+ * @dev: drm_device which is being closed by userspace
+ * @file_private: drm file-private structure to clean up
+ *
+ * Called at close time when the filp is going away.
+ *
+ * Releases any remaining references on objects by this filp.
+ */
+void
+drm_syncobj_release(struct drm_file *file_private)
+{
+ idr_for_each(&file_private->syncobj_idr,
+ &drm_syncobj_release_handle, file_private);
+ idr_destroy(&file_private->syncobj_idr);
+}
+
+int
+drm_syncobj_create_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_create *args = data;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ return -ENODEV;
+
+ /* no valid flags yet */
+ if (args->flags)
+ return -EINVAL;
+
+ return drm_syncobj_create(file_private,
+ &args->handle);
+}
+
+int
+drm_syncobj_destroy_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_destroy *args = data;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ return -ENODEV;
+
+ /* make sure padding is empty */
+ if (args->pad)
+ return -EINVAL;
+ return drm_syncobj_destroy(file_private, args->handle);
+}
+
+int
+drm_syncobj_handle_to_fd_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_handle *args = data;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ return -ENODEV;
+
+ if (args->pad)
+ return -EINVAL;
+
+ if (args->flags != 0 &&
+ args->flags != DRM_SYNCOBJ_HANDLE_TO_FD_FLAGS_EXPORT_SYNC_FILE)
+ return -EINVAL;
+
+ if (args->flags & DRM_SYNCOBJ_HANDLE_TO_FD_FLAGS_EXPORT_SYNC_FILE)
+ return drm_syncobj_export_sync_file(file_private, args->handle,
+ &args->fd);
+
+ return drm_syncobj_handle_to_fd(file_private, args->handle,
+ &args->fd);
+}
+
+int
+drm_syncobj_fd_to_handle_ioctl(struct drm_device *dev, void *data,
+ struct drm_file *file_private)
+{
+ struct drm_syncobj_handle *args = data;
+
+ if (!drm_core_check_feature(dev, DRIVER_SYNCOBJ))
+ return -ENODEV;
+
+ if (args->pad)
+ return -EINVAL;
+
+ if (args->flags != 0 &&
+ args->flags != DRM_SYNCOBJ_FD_TO_HANDLE_FLAGS_IMPORT_SYNC_FILE)
+ return -EINVAL;
+
+ if (args->flags & DRM_SYNCOBJ_FD_TO_HANDLE_FLAGS_IMPORT_SYNC_FILE)
+ return drm_syncobj_import_sync_file_fence(file_private,
+ args->fd,
+ args->handle);
+
+ return drm_syncobj_fd_to_handle(file_private, args->fd,
+ &args->handle);
+}
/* initially, until copy_from_user() and bo lookup succeeds: */
submit->nr_bos = 0;
+ submit->fence = NULL;
ww_acquire_init(&submit->ticket, &reservation_ww_class);
}
}
ww_acquire_fini(&submit->ticket);
- dma_fence_put(submit->fence);
+ if (submit->fence)
+ dma_fence_put(submit->fence);
kfree(submit);
}
"sclk_decon_eclk",
};
-enum decon_flag_bits {
- BIT_CLKS_ENABLED,
- BIT_IRQS_ENABLED,
- BIT_WIN_UPDATED,
- BIT_SUSPENDED,
- BIT_REQUEST_UPDATE
-};
-
struct decon_context {
struct device *dev;
struct drm_device *drm_dev;
void __iomem *addr;
struct regmap *sysreg;
struct clk *clks[ARRAY_SIZE(decon_clks_name)];
- int pipe;
- unsigned long flags;
+ unsigned int irq;
+ unsigned int te_irq;
unsigned long out_type;
int first_win;
spinlock_t vblank_lock;
struct decon_context *ctx = crtc->ctx;
u32 val;
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return -EPERM;
+ val = VIDINTCON0_INTEN;
+ if (ctx->out_type & IFTYPE_I80)
+ val |= VIDINTCON0_FRAMEDONE;
+ else
+ val |= VIDINTCON0_INTFRMEN | VIDINTCON0_FRAMESEL_FP;
- if (!test_and_set_bit(BIT_IRQS_ENABLED, &ctx->flags)) {
- val = VIDINTCON0_INTEN;
- if (ctx->out_type & IFTYPE_I80)
- val |= VIDINTCON0_FRAMEDONE;
- else
- val |= VIDINTCON0_INTFRMEN | VIDINTCON0_FRAMESEL_FP;
+ writel(val, ctx->addr + DECON_VIDINTCON0);
- writel(val, ctx->addr + DECON_VIDINTCON0);
- }
+ enable_irq(ctx->irq);
+ if (!(ctx->out_type & I80_HW_TRG))
+ enable_irq(ctx->te_irq);
return 0;
}
{
struct decon_context *ctx = crtc->ctx;
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
+ if (!(ctx->out_type & I80_HW_TRG))
+ disable_irq_nosync(ctx->te_irq);
+ disable_irq_nosync(ctx->irq);
- if (test_and_clear_bit(BIT_IRQS_ENABLED, &ctx->flags))
- writel(0, ctx->addr + DECON_VIDINTCON0);
+ writel(0, ctx->addr + DECON_VIDINTCON0);
}
/* return number of starts/ends of frame transmissions since reset */
return frm;
}
+static u32 decon_get_vblank_counter(struct exynos_drm_crtc *crtc)
+{
+ struct decon_context *ctx = crtc->ctx;
+
+ return decon_get_frame_count(ctx, false);
+}
+
static void decon_setup_trigger(struct decon_context *ctx)
{
if (!(ctx->out_type & (IFTYPE_I80 | I80_HW_TRG)))
bool interlaced = false;
u32 val;
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
-
if (ctx->out_type & IFTYPE_HDMI) {
m->crtc_hsync_start = m->crtc_hdisplay + 10;
m->crtc_hsync_end = m->crtc_htotal - 92;
writel(val, ctx->addr + DECON_WINCONx(win));
}
-static void decon_shadow_protect_win(struct decon_context *ctx, int win,
- bool protect)
+static void decon_shadow_protect(struct decon_context *ctx, bool protect)
{
- decon_set_bits(ctx, DECON_SHADOWCON, SHADOWCON_Wx_PROTECT(win),
+ decon_set_bits(ctx, DECON_SHADOWCON, SHADOWCON_PROTECT_MASK,
protect ? ~0 : 0);
}
static void decon_atomic_begin(struct exynos_drm_crtc *crtc)
{
struct decon_context *ctx = crtc->ctx;
- int i;
-
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
- for (i = ctx->first_win; i < WINDOWS_NR; i++)
- decon_shadow_protect_win(ctx, i, true);
+ decon_shadow_protect(ctx, true);
}
#define BIT_VAL(x, e, s) (((x) & ((1 << ((e) - (s) + 1)) - 1)) << (s))
dma_addr_t dma_addr = exynos_drm_fb_dma_addr(fb, 0);
u32 val;
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
-
if (crtc->base.mode.flags & DRM_MODE_FLAG_INTERLACE) {
val = COORDINATE_X(state->crtc.x) |
COORDINATE_Y(state->crtc.y / 2);
/* window enable */
decon_set_bits(ctx, DECON_WINCONx(win), WINCONx_ENWIN_F, ~0);
- set_bit(BIT_REQUEST_UPDATE, &ctx->flags);
}
static void decon_disable_plane(struct exynos_drm_crtc *crtc,
struct decon_context *ctx = crtc->ctx;
unsigned int win = plane->index;
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
-
decon_set_bits(ctx, DECON_WINCONx(win), WINCONx_ENWIN_F, 0);
- set_bit(BIT_REQUEST_UPDATE, &ctx->flags);
}
static void decon_atomic_flush(struct exynos_drm_crtc *crtc)
{
struct decon_context *ctx = crtc->ctx;
unsigned long flags;
- int i;
-
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
spin_lock_irqsave(&ctx->vblank_lock, flags);
- for (i = ctx->first_win; i < WINDOWS_NR; i++)
- decon_shadow_protect_win(ctx, i, false);
-
- if (test_and_clear_bit(BIT_REQUEST_UPDATE, &ctx->flags))
- decon_set_bits(ctx, DECON_UPDATE, STANDALONE_UPDATE_F, ~0);
+ decon_shadow_protect(ctx, false);
- if (ctx->out_type & IFTYPE_I80)
- set_bit(BIT_WIN_UPDATED, &ctx->flags);
+ decon_set_bits(ctx, DECON_UPDATE, STANDALONE_UPDATE_F, ~0);
ctx->frame_id = decon_get_frame_count(ctx, true);
{
struct decon_context *ctx = crtc->ctx;
- if (!test_and_clear_bit(BIT_SUSPENDED, &ctx->flags))
- return;
-
pm_runtime_get_sync(ctx->dev);
exynos_drm_pipe_clk_enable(crtc, true);
- set_bit(BIT_CLKS_ENABLED, &ctx->flags);
-
decon_swreset(ctx);
- /* if vblank was enabled status, enable it again. */
- if (test_and_clear_bit(BIT_IRQS_ENABLED, &ctx->flags))
- decon_enable_vblank(ctx->crtc);
-
decon_commit(ctx->crtc);
}
struct decon_context *ctx = crtc->ctx;
int i;
- if (test_bit(BIT_SUSPENDED, &ctx->flags))
- return;
+ if (!(ctx->out_type & I80_HW_TRG))
+ synchronize_irq(ctx->te_irq);
+ synchronize_irq(ctx->irq);
/*
* We need to make sure that all windows are disabled before we
decon_swreset(ctx);
- clear_bit(BIT_CLKS_ENABLED, &ctx->flags);
-
exynos_drm_pipe_clk_enable(crtc, false);
pm_runtime_put_sync(ctx->dev);
-
- set_bit(BIT_SUSPENDED, &ctx->flags);
}
-static void decon_te_irq_handler(struct exynos_drm_crtc *crtc)
+static irqreturn_t decon_te_irq_handler(int irq, void *dev_id)
{
- struct decon_context *ctx = crtc->ctx;
+ struct decon_context *ctx = dev_id;
- if (!test_bit(BIT_CLKS_ENABLED, &ctx->flags) ||
- (ctx->out_type & I80_HW_TRG))
- return;
+ decon_set_bits(ctx, DECON_TRIGCON, TRIGCON_SWTRIGCMD, ~0);
- if (test_and_clear_bit(BIT_WIN_UPDATED, &ctx->flags))
- decon_set_bits(ctx, DECON_TRIGCON, TRIGCON_SWTRIGCMD, ~0);
+ return IRQ_HANDLED;
}
static void decon_clear_channels(struct exynos_drm_crtc *crtc)
goto err;
}
- for (win = 0; win < WINDOWS_NR; win++) {
- decon_shadow_protect_win(ctx, win, true);
+ decon_shadow_protect(ctx, true);
+ for (win = 0; win < WINDOWS_NR; win++)
decon_set_bits(ctx, DECON_WINCONx(win), WINCONx_ENWIN_F, 0);
- decon_shadow_protect_win(ctx, win, false);
- }
+ decon_shadow_protect(ctx, false);
decon_set_bits(ctx, DECON_UPDATE, STANDALONE_UPDATE_F, ~0);
.disable = decon_disable,
.enable_vblank = decon_enable_vblank,
.disable_vblank = decon_disable_vblank,
+ .get_vblank_counter = decon_get_vblank_counter,
.atomic_begin = decon_atomic_begin,
.update_plane = decon_update_plane,
.disable_plane = decon_disable_plane,
.atomic_flush = decon_atomic_flush,
- .te_handler = decon_te_irq_handler,
};
static int decon_bind(struct device *dev, struct device *master, void *data)
{
struct decon_context *ctx = dev_get_drvdata(dev);
struct drm_device *drm_dev = data;
- struct exynos_drm_private *priv = drm_dev->dev_private;
struct exynos_drm_plane *exynos_plane;
enum exynos_drm_output_type out_type;
unsigned int win;
int ret;
ctx->drm_dev = drm_dev;
- ctx->pipe = priv->pipe++;
+ drm_dev->max_vblank_count = 0xffffffff;
for (win = ctx->first_win; win < WINDOWS_NR; win++) {
int tmp = (win == ctx->first_win) ? 0 : win;
ctx->configs[win].type = decon_win_types[tmp];
ret = exynos_plane_init(drm_dev, &ctx->planes[win], win,
- 1 << ctx->pipe, &ctx->configs[win]);
+ &ctx->configs[win]);
if (ret)
return ret;
}
out_type = (ctx->out_type & IFTYPE_HDMI) ? EXYNOS_DISPLAY_TYPE_HDMI
: EXYNOS_DISPLAY_TYPE_LCD;
ctx->crtc = exynos_drm_crtc_create(drm_dev, &exynos_plane->base,
- ctx->pipe, out_type,
- &decon_crtc_ops, ctx);
- if (IS_ERR(ctx->crtc)) {
- ret = PTR_ERR(ctx->crtc);
- goto err;
- }
+ out_type, &decon_crtc_ops, ctx);
+ if (IS_ERR(ctx->crtc))
+ return PTR_ERR(ctx->crtc);
decon_clear_channels(ctx->crtc);
- ret = drm_iommu_attach_device(drm_dev, dev);
- if (ret)
- goto err;
-
- return ret;
-err:
- priv->pipe--;
- return ret;
+ return drm_iommu_attach_device(drm_dev, dev);
}
static void decon_unbind(struct device *dev, struct device *master, void *data)
struct decon_context *ctx = dev_id;
u32 val;
- if (!test_bit(BIT_CLKS_ENABLED, &ctx->flags))
- goto out;
-
val = readl(ctx->addr + DECON_VIDINTCON1);
val &= VIDINTCON1_INTFRMDONEPEND | VIDINTCON1_INTFRMPEND;
decon_handle_vblank(ctx);
}
-out:
return IRQ_HANDLED;
}
};
MODULE_DEVICE_TABLE(of, exynos5433_decon_driver_dt_match);
+static int decon_conf_irq(struct decon_context *ctx, const char *name,
+ irq_handler_t handler, unsigned long int flags, bool required)
+{
+ struct platform_device *pdev = to_platform_device(ctx->dev);
+ int ret, irq = platform_get_irq_byname(pdev, name);
+
+ if (irq < 0) {
+ if (irq == -EPROBE_DEFER)
+ return irq;
+ if (required)
+ dev_err(ctx->dev, "cannot get %s IRQ\n", name);
+ else
+ irq = 0;
+ return irq;
+ }
+ irq_set_status_flags(irq, IRQ_NOAUTOEN);
+ ret = devm_request_irq(ctx->dev, irq, handler, flags, "drm_decon", ctx);
+ if (ret < 0) {
+ dev_err(ctx->dev, "IRQ %s request failed\n", name);
+ return ret;
+ }
+
+ return irq;
+}
+
static int exynos5433_decon_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
if (!ctx)
return -ENOMEM;
- __set_bit(BIT_SUSPENDED, &ctx->flags);
ctx->dev = dev;
ctx->out_type = (unsigned long)of_device_get_match_data(dev);
spin_lock_init(&ctx->vblank_lock);
ctx->out_type |= IFTYPE_I80;
}
- if (ctx->out_type & I80_HW_TRG) {
- ctx->sysreg = syscon_regmap_lookup_by_phandle(dev->of_node,
- "samsung,disp-sysreg");
- if (IS_ERR(ctx->sysreg)) {
- dev_err(dev, "failed to get system register\n");
- return PTR_ERR(ctx->sysreg);
- }
- }
-
for (i = 0; i < ARRAY_SIZE(decon_clks_name); i++) {
struct clk *clk;
return PTR_ERR(ctx->addr);
}
- res = platform_get_resource_byname(pdev, IORESOURCE_IRQ,
- (ctx->out_type & IFTYPE_I80) ? "lcd_sys" : "vsync");
- if (!res) {
- dev_err(dev, "cannot find IRQ resource\n");
- return -ENXIO;
+ if (ctx->out_type & IFTYPE_I80) {
+ ret = decon_conf_irq(ctx, "lcd_sys", decon_irq_handler, 0, true);
+ if (ret < 0)
+ return ret;
+ ctx->irq = ret;
+
+ ret = decon_conf_irq(ctx, "te", decon_te_irq_handler,
+ IRQF_TRIGGER_RISING, false);
+ if (ret < 0)
+ return ret;
+ if (ret) {
+ ctx->te_irq = ret;
+ ctx->out_type &= ~I80_HW_TRG;
+ }
+ } else {
+ ret = decon_conf_irq(ctx, "vsync", decon_irq_handler, 0, true);
+ if (ret < 0)
+ return ret;
+ ctx->irq = ret;
}
- ret = devm_request_irq(dev, res->start, decon_irq_handler, 0,
- "drm_decon", ctx);
- if (ret < 0) {
- dev_err(dev, "lcd_sys irq request failed\n");
- return ret;
+ if (ctx->out_type & I80_HW_TRG) {
+ ctx->sysreg = syscon_regmap_lookup_by_phandle(dev->of_node,
+ "samsung,disp-sysreg");
+ if (IS_ERR(ctx->sysreg)) {
+ dev_err(dev, "failed to get system register\n");
+ return PTR_ERR(ctx->sysreg);
+ }
}
platform_set_drvdata(pdev, ctx);
unsigned long irq_flags;
bool i80_if;
bool suspended;
- int pipe;
wait_queue_head_t wait_vsync_queue;
atomic_t wait_vsync_event;
static int decon_ctx_initialize(struct decon_context *ctx,
struct drm_device *drm_dev)
{
- struct exynos_drm_private *priv = drm_dev->dev_private;
- int ret;
-
ctx->drm_dev = drm_dev;
- ctx->pipe = priv->pipe++;
decon_clear_channels(ctx->crtc);
- ret = drm_iommu_attach_device(drm_dev, ctx->dev);
- if (ret)
- priv->pipe--;
-
- return ret;
+ return drm_iommu_attach_device(drm_dev, ctx->dev);
}
static void decon_ctx_remove(struct decon_context *ctx)
static const struct exynos_drm_crtc_ops decon_crtc_ops = {
.enable = decon_enable,
.disable = decon_disable,
- .commit = decon_commit,
.enable_vblank = decon_enable_vblank,
.disable_vblank = decon_disable_vblank,
.atomic_begin = decon_atomic_begin,
writel(clear_bit, ctx->regs + VIDINTCON1);
/* check the crtc is detached already from encoder */
- if (ctx->pipe < 0 || !ctx->drm_dev)
+ if (!ctx->drm_dev)
goto out;
if (!ctx->i80_if) {
ctx->configs[i].type = decon_win_types[i];
ret = exynos_plane_init(drm_dev, &ctx->planes[i], i,
- 1 << ctx->pipe, &ctx->configs[i]);
+ &ctx->configs[i]);
if (ret)
return ret;
}
exynos_plane = &ctx->planes[DEFAULT_WIN];
ctx->crtc = exynos_drm_crtc_create(drm_dev, &exynos_plane->base,
- ctx->pipe, EXYNOS_DISPLAY_TYPE_LCD,
- &decon_crtc_ops, ctx);
+ EXYNOS_DISPLAY_TYPE_LCD, &decon_crtc_ops, ctx);
if (IS_ERR(ctx->crtc)) {
decon_ctx_remove(ctx);
return PTR_ERR(ctx->crtc);
}
}
-static void
-exynos_drm_crtc_mode_set_nofb(struct drm_crtc *crtc)
-{
- struct exynos_drm_crtc *exynos_crtc = to_exynos_crtc(crtc);
-
- if (exynos_crtc->ops->commit)
- exynos_crtc->ops->commit(exynos_crtc);
-}
-
static int exynos_crtc_atomic_check(struct drm_crtc *crtc,
struct drm_crtc_state *state)
{
static const struct drm_crtc_helper_funcs exynos_crtc_helper_funcs = {
.enable = exynos_drm_crtc_enable,
.disable = exynos_drm_crtc_disable,
- .mode_set_nofb = exynos_drm_crtc_mode_set_nofb,
.atomic_check = exynos_crtc_atomic_check,
.atomic_begin = exynos_crtc_atomic_begin,
.atomic_flush = exynos_crtc_atomic_flush,
struct drm_pending_vblank_event *event = crtc->state->event;
unsigned long flags;
- if (event) {
- crtc->state->event = NULL;
- spin_lock_irqsave(&crtc->dev->event_lock, flags);
- if (drm_crtc_vblank_get(crtc) == 0)
- drm_crtc_arm_vblank_event(crtc, event);
- else
- drm_crtc_send_vblank_event(crtc, event);
- spin_unlock_irqrestore(&crtc->dev->event_lock, flags);
- }
+ if (!event)
+ return;
+ crtc->state->event = NULL;
+ WARN_ON(drm_crtc_vblank_get(crtc) != 0);
+
+ spin_lock_irqsave(&crtc->dev->event_lock, flags);
+ drm_crtc_arm_vblank_event(crtc, event);
+ spin_unlock_irqrestore(&crtc->dev->event_lock, flags);
}
static void exynos_drm_crtc_destroy(struct drm_crtc *crtc)
exynos_crtc->ops->disable_vblank(exynos_crtc);
}
+static u32 exynos_drm_crtc_get_vblank_counter(struct drm_crtc *crtc)
+{
+ struct exynos_drm_crtc *exynos_crtc = to_exynos_crtc(crtc);
+
+ if (exynos_crtc->ops->get_vblank_counter)
+ return exynos_crtc->ops->get_vblank_counter(exynos_crtc);
+
+ return 0;
+}
+
static const struct drm_crtc_funcs exynos_crtc_funcs = {
.set_config = drm_atomic_helper_set_config,
.page_flip = drm_atomic_helper_page_flip,
.atomic_destroy_state = drm_atomic_helper_crtc_destroy_state,
.enable_vblank = exynos_drm_crtc_enable_vblank,
.disable_vblank = exynos_drm_crtc_disable_vblank,
+ .get_vblank_counter = exynos_drm_crtc_get_vblank_counter,
};
struct exynos_drm_crtc *exynos_drm_crtc_create(struct drm_device *drm_dev,
struct drm_plane *plane,
- int pipe,
enum exynos_drm_output_type type,
const struct exynos_drm_crtc_ops *ops,
void *ctx)
if (!exynos_crtc)
return ERR_PTR(-ENOMEM);
- exynos_crtc->pipe = pipe;
exynos_crtc->type = type;
exynos_crtc->ops = ops;
exynos_crtc->ctx = ctx;
{
struct drm_crtc *crtc;
- list_for_each_entry(crtc, &drm_dev->mode_config.crtc_list, head) {
- struct exynos_drm_crtc *exynos_crtc;
-
- exynos_crtc = to_exynos_crtc(crtc);
- if (exynos_crtc->type == out_type)
- return exynos_crtc->pipe;
- }
+ drm_for_each_crtc(crtc, drm_dev)
+ if (to_exynos_crtc(crtc)->type == out_type)
+ return drm_crtc_index(crtc);
return -EPERM;
}
struct exynos_drm_crtc *exynos_drm_crtc_create(struct drm_device *drm_dev,
struct drm_plane *plane,
- int pipe,
enum exynos_drm_output_type type,
const struct exynos_drm_crtc_ops *ops,
void *context);
*
* @enable: enable the device
* @disable: disable the device
- * @commit: set current hw specific display mode to hw.
* @enable_vblank: specific driver callback for enabling vblank interrupt.
* @disable_vblank: specific driver callback for disabling vblank interrupt.
* @atomic_check: validate state
struct exynos_drm_crtc_ops {
void (*enable)(struct exynos_drm_crtc *crtc);
void (*disable)(struct exynos_drm_crtc *crtc);
- void (*commit)(struct exynos_drm_crtc *crtc);
int (*enable_vblank)(struct exynos_drm_crtc *crtc);
void (*disable_vblank)(struct exynos_drm_crtc *crtc);
+ u32 (*get_vblank_counter)(struct exynos_drm_crtc *crtc);
int (*atomic_check)(struct exynos_drm_crtc *crtc,
struct drm_crtc_state *state);
void (*atomic_begin)(struct exynos_drm_crtc *crtc);
*
* @base: crtc object.
* @type: one of EXYNOS_DISPLAY_TYPE_LCD and HDMI.
- * @pipe: a crtc index created at load() with a new crtc object creation
- * and the crtc object would be set to private->crtc array
- * to get a crtc object corresponding to this pipe from private->crtc
- * array when irq interrupt occurred. the reason of using this pipe is that
- * drm framework doesn't support multiple irq yet.
- * we can refer to the crtc to current hardware interrupt occurred through
- * this pipe value.
- * @enabled: if the crtc is enabled or not
- * @event: vblank event that is currently queued for flip
- * @wait_update: wait all pending planes updates to finish
- * @pending_update: number of pending plane updates in this crtc
* @ops: pointer to callbacks for exynos drm specific functionality
* @ctx: A pointer to the crtc's implementation specific context
+ * @pipe_clk: A pointer to the crtc's pipeline clock.
*/
struct exynos_drm_crtc {
struct drm_crtc base;
enum exynos_drm_output_type type;
- unsigned int pipe;
const struct exynos_drm_crtc_ops *ops;
void *ctx;
struct exynos_drm_clk *pipe_clk;
* otherwise default one.
* @da_space_size: size of device address space.
* if 0 then default value is used for it.
- * @pipe: the pipe number for this crtc/manager.
* @pending: the crtcs that have pending updates to finish
* @lock: protect access to @pending
* @wait: wait an atomic commit to finish
struct device *dma_dev;
void *mapping;
- unsigned int pipe;
-
/* for atomic commit */
u32 pending;
spinlock_t lock;
{
struct device *dev = dsi->dev;
struct device_node *node = dev->of_node;
- struct device_node *ep;
int ret;
ret = exynos_dsi_of_read_u32(node, "samsung,pll-clock-frequency",
if (ret < 0)
return ret;
- ep = of_graph_get_endpoint_by_regs(node, DSI_PORT_OUT, 0);
- if (!ep) {
- dev_err(dev, "no output port with endpoint specified\n");
- return -EINVAL;
- }
-
- ret = exynos_dsi_of_read_u32(ep, "samsung,burst-clock-frequency",
+ ret = exynos_dsi_of_read_u32(node, "samsung,burst-clock-frequency",
&dsi->burst_clk_rate);
if (ret < 0)
- goto end;
+ return ret;
- ret = exynos_dsi_of_read_u32(ep, "samsung,esc-clock-frequency",
+ ret = exynos_dsi_of_read_u32(node, "samsung,esc-clock-frequency",
&dsi->esc_clk_rate);
if (ret < 0)
- goto end;
-
- of_node_put(ep);
+ return ret;
- dsi->bridge_node = of_graph_get_remote_node(node, DSI_PORT_OUT, 0);
+ dsi->bridge_node = of_graph_get_remote_node(node, DSI_PORT_IN, 0);
if (!dsi->bridge_node)
return -EINVAL;
-end:
- of_node_put(ep);
-
- return ret;
+ return 0;
}
static int exynos_dsi_bind(struct device *dev, struct device *master,
static int exynos_dsi_remove(struct platform_device *pdev)
{
+ struct exynos_dsi *dsi = platform_get_drvdata(pdev);
+
+ of_node_put(dsi->bridge_node);
+
pm_runtime_disable(&pdev->dev);
component_del(&pdev->dev, &exynos_dsi_component_ops);
u32 i80ifcon;
bool i80_if;
bool suspended;
- int pipe;
wait_queue_head_t wait_vsync_queue;
atomic_t wait_vsync_event;
atomic_t win_updated;
/* Wait for vsync, as disable channel takes effect at next vsync */
if (ch_enabled) {
- int pipe = ctx->pipe;
-
- /* ensure that vblank interrupt won't be reported to core */
ctx->suspended = false;
- ctx->pipe = -1;
fimd_enable_vblank(ctx->crtc);
fimd_wait_for_vblank(ctx->crtc);
fimd_disable_vblank(ctx->crtc);
ctx->suspended = true;
- ctx->pipe = pipe;
}
clk_disable_unprepare(ctx->lcd_clk);
u32 trg_type = ctx->driver_data->trg_type;
/* Checks the crtc is detached already from encoder */
- if (ctx->pipe < 0 || !ctx->drm_dev)
+ if (!ctx->drm_dev)
return;
if (trg_type == I80_HW_TRG)
static const struct exynos_drm_crtc_ops fimd_crtc_ops = {
.enable = fimd_enable,
.disable = fimd_disable,
- .commit = fimd_commit,
.enable_vblank = fimd_enable_vblank,
.disable_vblank = fimd_disable_vblank,
.atomic_begin = fimd_atomic_begin,
writel(clear_bit, ctx->regs + VIDINTCON1);
/* check the crtc is detached already from encoder */
- if (ctx->pipe < 0 || !ctx->drm_dev)
+ if (!ctx->drm_dev)
goto out;
if (!ctx->i80_if)
{
struct fimd_context *ctx = dev_get_drvdata(dev);
struct drm_device *drm_dev = data;
- struct exynos_drm_private *priv = drm_dev->dev_private;
struct exynos_drm_plane *exynos_plane;
unsigned int i;
int ret;
ctx->drm_dev = drm_dev;
- ctx->pipe = priv->pipe++;
for (i = 0; i < WINDOWS_NR; i++) {
ctx->configs[i].pixel_formats = fimd_formats;
ctx->configs[i].zpos = i;
ctx->configs[i].type = fimd_win_types[i];
ret = exynos_plane_init(drm_dev, &ctx->planes[i], i,
- 1 << ctx->pipe, &ctx->configs[i]);
+ &ctx->configs[i]);
if (ret)
return ret;
}
exynos_plane = &ctx->planes[DEFAULT_WIN];
ctx->crtc = exynos_drm_crtc_create(drm_dev, &exynos_plane->base,
- ctx->pipe, EXYNOS_DISPLAY_TYPE_LCD,
- &fimd_crtc_ops, ctx);
+ EXYNOS_DISPLAY_TYPE_LCD, &fimd_crtc_ops, ctx);
if (IS_ERR(ctx->crtc))
return PTR_ERR(ctx->crtc);
if (is_drm_iommu_supported(drm_dev))
fimd_clear_channels(ctx->crtc);
- ret = drm_iommu_attach_device(drm_dev, dev);
- if (ret)
- priv->pipe--;
-
- return ret;
+ return drm_iommu_attach_device(drm_dev, dev);
}
static void fimd_unbind(struct device *dev, struct device *master,
}
int exynos_plane_init(struct drm_device *dev,
- struct exynos_drm_plane *exynos_plane,
- unsigned int index, unsigned long possible_crtcs,
+ struct exynos_drm_plane *exynos_plane, unsigned int index,
const struct exynos_drm_plane_config *config)
{
int err;
err = drm_universal_plane_init(dev, &exynos_plane->base,
- possible_crtcs,
+ 1 << dev->mode_config.num_crtc,
&exynos_plane_funcs,
config->pixel_formats,
config->num_pixel_formats,
int exynos_plane_init(struct drm_device *dev,
struct exynos_drm_plane *exynos_plane, unsigned int index,
- unsigned long possible_crtcs,
const struct exynos_drm_plane_config *config);
bool suspended;
struct timer_list timer;
struct mutex lock;
- int pipe;
};
static inline struct vidi_context *encoder_to_vidi(struct drm_encoder *e)
mutex_unlock(&ctx->lock);
}
-static int vidi_ctx_initialize(struct vidi_context *ctx,
- struct drm_device *drm_dev)
-{
- struct exynos_drm_private *priv = drm_dev->dev_private;
-
- ctx->drm_dev = drm_dev;
- ctx->pipe = priv->pipe++;
-
- return 0;
-}
-
static const struct exynos_drm_crtc_ops vidi_crtc_ops = {
.enable = vidi_enable,
.disable = vidi_disable,
{
struct vidi_context *ctx = (void *)arg;
- if (ctx->pipe < 0)
- return;
-
if (drm_crtc_handle_vblank(&ctx->crtc->base))
mod_timer(&ctx->timer,
jiffies + msecs_to_jiffies(VIDI_REFRESH_TIME) - 1);
unsigned int i;
int pipe, ret;
- vidi_ctx_initialize(ctx, drm_dev);
+ ctx->drm_dev = drm_dev;
plane_config.pixel_formats = formats;
plane_config.num_pixel_formats = ARRAY_SIZE(formats);
plane_config.type = vidi_win_types[i];
ret = exynos_plane_init(drm_dev, &ctx->planes[i], i,
- 1 << ctx->pipe, &plane_config);
+ &plane_config);
if (ret)
return ret;
}
exynos_plane = &ctx->planes[DEFAULT_WIN];
ctx->crtc = exynos_drm_crtc_create(drm_dev, &exynos_plane->base,
- ctx->pipe, EXYNOS_DISPLAY_TYPE_VIDI,
- &vidi_crtc_ops, ctx);
+ EXYNOS_DISPLAY_TYPE_VIDI, &vidi_crtc_ops, ctx);
if (IS_ERR(ctx->crtc)) {
DRM_ERROR("failed to create crtc.\n");
return PTR_ERR(ctx->crtc);
static void hdmi_disable(struct drm_encoder *encoder)
{
struct hdmi_context *hdata = encoder_to_hdmi(encoder);
- struct drm_crtc *crtc = encoder->crtc;
- const struct drm_crtc_helper_funcs *funcs = NULL;
if (!hdata->powered)
return;
* to disable TV Subsystem should be as following,
* VP -> Mixer -> HDMI
*
- * Below codes will try to disable Mixer and VP(if used)
- * prior to disabling HDMI.
+ * To achieve such sequence HDMI is disabled together with HDMI PHY, via
+ * pipe clock callback.
*/
- if (crtc)
- funcs = crtc->helper_private;
- if (funcs && funcs->disable)
- (*funcs->disable)(crtc);
-
- cec_notifier_set_phys_addr(hdata->notifier, CEC_PHYS_ADDR_INVALID);
cancel_delayed_work(&hdata->hotplug_work);
+ cec_notifier_set_phys_addr(hdata->notifier, CEC_PHYS_ADDR_INVALID);
hdmiphy_disable(hdata);
}
#define MIXER_WIN_NR 3
#define VP_DEFAULT_WIN 2
+/*
+ * Mixer color space conversion coefficient triplet.
+ * Used for CSC from RGB to YCbCr.
+ * Each coefficient is a 10-bit fixed point number with
+ * sign and no integer part, i.e.
+ * [0:8] = fractional part (representing a value y = x / 2^9)
+ * [9] = sign
+ * Negative values are encoded with two's complement.
+ */
+#define MXR_CSC_C(x) ((int)((x) * 512.0) & 0x3ff)
+#define MXR_CSC_CT(a0, a1, a2) \
+ ((MXR_CSC_C(a0) << 20) | (MXR_CSC_C(a1) << 10) | (MXR_CSC_C(a2) << 0))
+
+/* YCbCr value, used for mixer background color configuration. */
+#define MXR_YCBCR_VAL(y, cb, cr) (((y) << 16) | ((cb) << 8) | ((cr) << 0))
+
/* The pixelformats that are natively supported by the mixer. */
#define MXR_FORMAT_RGB565 4
#define MXR_FORMAT_ARGB1555 5
struct drm_device *drm_dev;
struct exynos_drm_crtc *crtc;
struct exynos_drm_plane planes[MIXER_WIN_NR];
- int pipe;
unsigned long flags;
struct mixer_resources mixer_res;
struct mixer_resources *res = &ctx->mixer_res;
u32 val;
- if (height == 480) {
- val = MXR_CFG_RGB601_0_255;
- } else if (height == 576) {
+ switch (height) {
+ case 480:
+ case 576:
val = MXR_CFG_RGB601_0_255;
- } else if (height == 720) {
- val = MXR_CFG_RGB709_16_235;
- mixer_reg_write(res, MXR_CM_COEFF_Y,
- (1 << 30) | (94 << 20) | (314 << 10) |
- (32 << 0));
- mixer_reg_write(res, MXR_CM_COEFF_CB,
- (972 << 20) | (851 << 10) | (225 << 0));
- mixer_reg_write(res, MXR_CM_COEFF_CR,
- (225 << 20) | (820 << 10) | (1004 << 0));
- } else if (height == 1080) {
- val = MXR_CFG_RGB709_16_235;
- mixer_reg_write(res, MXR_CM_COEFF_Y,
- (1 << 30) | (94 << 20) | (314 << 10) |
- (32 << 0));
- mixer_reg_write(res, MXR_CM_COEFF_CB,
- (972 << 20) | (851 << 10) | (225 << 0));
- mixer_reg_write(res, MXR_CM_COEFF_CR,
- (225 << 20) | (820 << 10) | (1004 << 0));
- } else {
+ break;
+ case 720:
+ case 1080:
+ default:
val = MXR_CFG_RGB709_16_235;
+ /* Configure the BT.709 CSC matrix for full range RGB. */
mixer_reg_write(res, MXR_CM_COEFF_Y,
- (1 << 30) | (94 << 20) | (314 << 10) |
- (32 << 0));
+ MXR_CSC_CT( 0.184, 0.614, 0.063) |
+ MXR_CM_COEFF_RGB_FULL);
mixer_reg_write(res, MXR_CM_COEFF_CB,
- (972 << 20) | (851 << 10) | (225 << 0));
+ MXR_CSC_CT(-0.102, -0.338, 0.440));
mixer_reg_write(res, MXR_CM_COEFF_CR,
- (225 << 20) | (820 << 10) | (1004 << 0));
+ MXR_CSC_CT( 0.440, -0.399, -0.040));
+ break;
}
mixer_reg_writemask(res, MXR_CFG, val, MXR_CFG_RGB_FMT_MASK);
/* reset default layer priority */
mixer_reg_write(res, MXR_LAYER_CFG, 0);
- /* setting background color */
- mixer_reg_write(res, MXR_BG_COLOR0, 0x008080);
- mixer_reg_write(res, MXR_BG_COLOR1, 0x008080);
- mixer_reg_write(res, MXR_BG_COLOR2, 0x008080);
+ /* set all background colors to RGB (0,0,0) */
+ mixer_reg_write(res, MXR_BG_COLOR0, MXR_YCBCR_VAL(0, 128, 128));
+ mixer_reg_write(res, MXR_BG_COLOR1, MXR_YCBCR_VAL(0, 128, 128));
+ mixer_reg_write(res, MXR_BG_COLOR2, MXR_YCBCR_VAL(0, 128, 128));
if (test_bit(MXR_BIT_VP_ENABLED, &ctx->flags)) {
/* configuration of Video Processor Registers */
priv = drm_dev->dev_private;
mixer_ctx->drm_dev = drm_dev;
- mixer_ctx->pipe = priv->pipe++;
/* acquire resources: regs, irqs, clocks */
ret = mixer_resources_init(mixer_ctx);
}
}
- ret = drm_iommu_attach_device(drm_dev, mixer_ctx->dev);
- if (ret)
- priv->pipe--;
-
- return ret;
+ return drm_iommu_attach_device(drm_dev, mixer_ctx->dev);
}
static void mixer_ctx_remove(struct mixer_context *mixer_ctx)
continue;
ret = exynos_plane_init(drm_dev, &ctx->planes[i], i,
- 1 << ctx->pipe, &plane_configs[i]);
+ &plane_configs[i]);
if (ret)
return ret;
}
exynos_plane = &ctx->planes[DEFAULT_WIN];
ctx->crtc = exynos_drm_crtc_create(drm_dev, &exynos_plane->base,
- ctx->pipe, EXYNOS_DISPLAY_TYPE_HDMI,
- &mixer_crtc_ops, ctx);
+ EXYNOS_DISPLAY_TYPE_HDMI, &mixer_crtc_ops, ctx);
if (IS_ERR(ctx->crtc)) {
mixer_ctx_remove(ctx);
ret = PTR_ERR(ctx->crtc);
#define MXR_INT_EN_VSYNC (1 << 11)
#define MXR_INT_EN_ALL (0x0f << 8)
-/* bit for MXR_INT_STATUS */
+/* bits for MXR_INT_STATUS */
#define MXR_INT_CLEAR_VSYNC (1 << 11)
#define MXR_INT_STATUS_VSYNC (1 << 0)
-/* bit for MXR_LAYER_CFG */
+/* bits for MXR_LAYER_CFG */
#define MXR_LAYER_CFG_GRP1_VAL(x) MXR_MASK_VAL(x, 11, 8)
#define MXR_LAYER_CFG_GRP1_MASK MXR_LAYER_CFG_GRP1_VAL(~0)
#define MXR_LAYER_CFG_GRP0_VAL(x) MXR_MASK_VAL(x, 7, 4)
#define MXR_LAYER_CFG_VP_VAL(x) MXR_MASK_VAL(x, 3, 0)
#define MXR_LAYER_CFG_VP_MASK MXR_LAYER_CFG_VP_VAL(~0)
+/* bits for MXR_CM_COEFF_Y */
+#define MXR_CM_COEFF_RGB_FULL (1 << 30)
+
#endif /* SAMSUNG_REGS_MIXER_H */
if (scan->type & DRM_MODE_TYPE_PREFERRED) {
mode_dev->panel_fixed_mode =
drm_mode_duplicate(dev, scan);
+ DRM_DEBUG_KMS("Using mode from DDC\n");
goto out; /* FIXME: check for quirks */
}
}
/* Failed to get EDID, what about VBT? do we need this? */
- if (mode_dev->vbt_mode)
+ if (dev_priv->lfp_lvds_vbt_mode) {
mode_dev->panel_fixed_mode =
- drm_mode_duplicate(dev, mode_dev->vbt_mode);
+ drm_mode_duplicate(dev, dev_priv->lfp_lvds_vbt_mode);
- if (!mode_dev->panel_fixed_mode)
- if (dev_priv->lfp_lvds_vbt_mode)
- mode_dev->panel_fixed_mode =
- drm_mode_duplicate(dev,
- dev_priv->lfp_lvds_vbt_mode);
+ if (mode_dev->panel_fixed_mode) {
+ mode_dev->panel_fixed_mode->type |=
+ DRM_MODE_TYPE_PREFERRED;
+ DRM_DEBUG_KMS("Using mode from VBT\n");
+ goto out;
+ }
+ }
/*
* If we didn't get EDID, try checking if the panel is already turned
if (mode_dev->panel_fixed_mode) {
mode_dev->panel_fixed_mode->type |=
DRM_MODE_TYPE_PREFERRED;
+ DRM_DEBUG_KMS("Using pre-programmed mode\n");
goto out; /* FIXME: check for quirks */
}
}
If in doubt, say "N".
+config DRM_I915_SW_FENCE_CHECK_DAG
+ bool "Enable additional driver debugging for detecting dependency cycles"
+ depends on DRM_I915
+ default n
+ help
+ Choose this option to turn on extra driver debugging that may affect
+ performance but will catch some internal issues.
+
+ Recommended for driver developers only.
+
+ If in doubt, say "N".
+
config DRM_I915_SELFTEST
bool "Enable selftests upon driver load"
depends on DRM_I915
i915_params.o \
i915_pci.o \
i915_suspend.o \
+ i915_syncmap.o \
i915_sw_fence.o \
i915_sysfs.o \
intel_csr.o \
# general-purpose microcontroller (GuC) support
i915-y += intel_uc.o \
+ intel_guc_ct.o \
intel_guc_log.o \
intel_guc_loader.o \
intel_huc.o \
(0 << CH7017_PHASE_DETECTOR_SHIFT);
} else {
outputs_enable = CH7017_LVDS_CHANNEL_A | CH7017_CHARGE_PUMP_HIGH;
- lvds_pll_feedback_div = CH7017_LVDS_PLL_FEEDBACK_DEFAULT_RESERVED |
+ lvds_pll_feedback_div =
+ CH7017_LVDS_PLL_FEEDBACK_DEFAULT_RESERVED |
(2 << CH7017_LVDS_PLL_FEED_BACK_DIVIDER_SHIFT) |
(3 << CH7017_LVDS_PLL_FEED_FORWARD_DIVIDER_SHIFT);
- lvds_pll_feedback_div = 35;
lvds_control_2 = (3 << CH7017_LOOP_FILTER_SHIFT) |
(0 << CH7017_PHASE_DETECTOR_SHIFT);
if (1) { /* XXX: dual channel panel detection. Assume yes for now. */
mode = vgpu_vreg(vgpu, offset);
if (GFX_MODE_BIT_SET_IN_MASK(mode, START_DMA)) {
- WARN_ONCE(1, "VM(%d): iGVT-g doesn't supporte GuC\n",
+ WARN_ONCE(1, "VM(%d): iGVT-g doesn't support GuC\n",
vgpu->id);
return 0;
}
} else
v = mmio->value;
+ if (mmio->in_context)
+ continue;
+
I915_WRITE(mmio->reg, v);
POSTING_READ(mmio->reg);
struct vgpu_sched_data *vgpu_data;
ktime_t cur_time;
- /* no target to schedule */
- if (!scheduler->next_vgpu)
+ /* no need to schedule if next_vgpu is the same with current_vgpu,
+ * let scheduler chose next_vgpu again by setting it to NULL.
+ */
+ if (scheduler->next_vgpu == scheduler->current_vgpu) {
+ scheduler->next_vgpu = NULL;
return;
+ }
/*
* after the flag is set, workload dispatch thread will
gvt_dbg_sched("ring id %d workload lrca %x", ring_id,
workload->ctx_desc.lrca);
- context_page_num = intel_lr_context_size(
- gvt->dev_priv->engine[ring_id]);
+ context_page_num = gvt->dev_priv->engine[ring_id]->context_size;
context_page_num = context_page_num >> PAGE_SHIFT;
struct intel_engine_cs *engine = dev_priv->engine[ring_id];
struct drm_i915_gem_request *rq;
struct intel_vgpu *vgpu = workload->vgpu;
+ struct intel_ring *ring;
int ret;
gvt_dbg_sched("ring id %d prepare to dispatch workload %p\n",
* shadow_ctx pages invalid. So gvt need to pin itself. After update
* the guest context, gvt can unpin the shadow_ctx safely.
*/
- ret = engine->context_pin(engine, shadow_ctx);
- if (ret) {
+ ring = engine->context_pin(engine, shadow_ctx);
+ if (IS_ERR(ring)) {
+ ret = PTR_ERR(ring);
gvt_vgpu_err("fail to pin shadow context\n");
workload->status = ret;
mutex_unlock(&dev_priv->drm.struct_mutex);
gvt_dbg_sched("ring id %d workload lrca %x\n", ring_id,
workload->ctx_desc.lrca);
- context_page_num = intel_lr_context_size(
- gvt->dev_priv->engine[ring_id]);
+ context_page_num = gvt->dev_priv->engine[ring_id]->context_size;
context_page_num = context_page_num >> PAGE_SHIFT;
find_reg(engine, is_master, reg_addr);
if (!reg) {
- DRM_DEBUG_DRIVER("CMD: Rejected register 0x%08X in command: 0x%08X (exec_id=%d)\n",
- reg_addr, *cmd, engine->exec_id);
+ DRM_DEBUG_DRIVER("CMD: Rejected register 0x%08X in command: 0x%08X (%s)\n",
+ reg_addr, *cmd, engine->name);
return false;
}
desc->bits[i].mask;
if (dword != desc->bits[i].expected) {
- DRM_DEBUG_DRIVER("CMD: Rejected command 0x%08X for bitmask 0x%08X (exp=0x%08X act=0x%08X) (exec_id=%d)\n",
+ DRM_DEBUG_DRIVER("CMD: Rejected command 0x%08X for bitmask 0x%08X (exp=0x%08X act=0x%08X) (%s)\n",
*cmd,
desc->bits[i].mask,
desc->bits[i].expected,
- dword, engine->exec_id);
+ dword, engine->name);
return false;
}
}
if (*cmd == MI_BATCH_BUFFER_END) {
if (needs_clflush_after) {
- void *ptr = ptr_mask_bits(shadow_batch_obj->mm.mapping);
+ void *ptr = page_mask_bits(shadow_batch_obj->mm.mapping);
drm_clflush_virt_range(ptr,
(void *)(cmd + 1) - ptr);
}
client->wq_size, client->wq_offset, client->wq_tail);
seq_printf(m, "\tWork queue full: %u\n", client->no_wq_space);
- seq_printf(m, "\tFailed doorbell: %u\n", client->b_fail);
- seq_printf(m, "\tLast submission result: %d\n", client->retcode);
for_each_engine(engine, dev_priv, id) {
u64 submissions = client->submissions[id];
seq_printf(m, "\tTotal: %llu\n", tot);
}
-static int i915_guc_info(struct seq_file *m, void *data)
+static bool check_guc_submission(struct seq_file *m)
{
struct drm_i915_private *dev_priv = node_to_i915(m->private);
const struct intel_guc *guc = &dev_priv->guc;
- struct intel_engine_cs *engine;
- enum intel_engine_id id;
- u64 total;
if (!guc->execbuf_client) {
seq_printf(m, "GuC submission %s\n",
HAS_GUC_SCHED(dev_priv) ?
"disabled" :
"not supported");
- return 0;
+ return false;
}
+ return true;
+}
+
+static int i915_guc_info(struct seq_file *m, void *data)
+{
+ struct drm_i915_private *dev_priv = node_to_i915(m->private);
+ const struct intel_guc *guc = &dev_priv->guc;
+
+ if (!check_guc_submission(m))
+ return 0;
+
seq_printf(m, "Doorbell map:\n");
seq_printf(m, "\t%*pb\n", GUC_NUM_DOORBELLS, guc->doorbell_bitmap);
seq_printf(m, "Doorbell next cacheline: 0x%x\n\n", guc->db_cacheline);
- seq_printf(m, "GuC total action count: %llu\n", guc->action_count);
- seq_printf(m, "GuC action failure count: %u\n", guc->action_fail);
- seq_printf(m, "GuC last action command: 0x%x\n", guc->action_cmd);
- seq_printf(m, "GuC last action status: 0x%x\n", guc->action_status);
- seq_printf(m, "GuC last action error code: %d\n", guc->action_err);
-
- total = 0;
- seq_printf(m, "\nGuC submissions:\n");
- for_each_engine(engine, dev_priv, id) {
- u64 submissions = guc->submissions[id];
- total += submissions;
- seq_printf(m, "\t%-24s: %10llu, last seqno 0x%08x\n",
- engine->name, submissions, guc->last_seqno[id]);
- }
- seq_printf(m, "\t%s: %llu\n", "Total", total);
-
seq_printf(m, "\nGuC execbuf client @ %p:\n", guc->execbuf_client);
i915_guc_client_info(m, dev_priv, guc->execbuf_client);
return 0;
}
-static int i915_guc_log_dump(struct seq_file *m, void *data)
+static int i915_guc_stage_pool(struct seq_file *m, void *data)
{
struct drm_i915_private *dev_priv = node_to_i915(m->private);
- struct drm_i915_gem_object *obj;
- int i = 0, pg;
+ const struct intel_guc *guc = &dev_priv->guc;
+ struct guc_stage_desc *desc = guc->stage_desc_pool_vaddr;
+ struct i915_guc_client *client = guc->execbuf_client;
+ unsigned int tmp;
+ int index;
- if (!dev_priv->guc.log.vma)
+ if (!check_guc_submission(m))
return 0;
- obj = dev_priv->guc.log.vma->obj;
- for (pg = 0; pg < obj->base.size / PAGE_SIZE; pg++) {
- u32 *log = kmap_atomic(i915_gem_object_get_page(obj, pg));
+ for (index = 0; index < GUC_MAX_STAGE_DESCRIPTORS; index++, desc++) {
+ struct intel_engine_cs *engine;
+
+ if (!(desc->attribute & GUC_STAGE_DESC_ATTR_ACTIVE))
+ continue;
+
+ seq_printf(m, "GuC stage descriptor %u:\n", index);
+ seq_printf(m, "\tIndex: %u\n", desc->stage_id);
+ seq_printf(m, "\tAttribute: 0x%x\n", desc->attribute);
+ seq_printf(m, "\tPriority: %d\n", desc->priority);
+ seq_printf(m, "\tDoorbell id: %d\n", desc->db_id);
+ seq_printf(m, "\tEngines used: 0x%x\n",
+ desc->engines_used);
+ seq_printf(m, "\tDoorbell trigger phy: 0x%llx, cpu: 0x%llx, uK: 0x%x\n",
+ desc->db_trigger_phy,
+ desc->db_trigger_cpu,
+ desc->db_trigger_uk);
+ seq_printf(m, "\tProcess descriptor: 0x%x\n",
+ desc->process_desc);
+ seq_printf(m, "\tWorkqueue address: 0x%x, size: 0x%x\n",
+ desc->wq_addr, desc->wq_size);
+ seq_putc(m, '\n');
+
+ for_each_engine_masked(engine, dev_priv, client->engines, tmp) {
+ u32 guc_engine_id = engine->guc_id;
+ struct guc_execlist_context *lrc =
+ &desc->lrc[guc_engine_id];
+
+ seq_printf(m, "\t%s LRC:\n", engine->name);
+ seq_printf(m, "\t\tContext desc: 0x%x\n",
+ lrc->context_desc);
+ seq_printf(m, "\t\tContext id: 0x%x\n", lrc->context_id);
+ seq_printf(m, "\t\tLRCA: 0x%x\n", lrc->ring_lrca);
+ seq_printf(m, "\t\tRing begin: 0x%x\n", lrc->ring_begin);
+ seq_printf(m, "\t\tRing end: 0x%x\n", lrc->ring_end);
+ seq_putc(m, '\n');
+ }
+ }
+
+ return 0;
+}
+
+static int i915_guc_log_dump(struct seq_file *m, void *data)
+{
+ struct drm_info_node *node = m->private;
+ struct drm_i915_private *dev_priv = node_to_i915(node);
+ bool dump_load_err = !!node->info_ent->data;
+ struct drm_i915_gem_object *obj = NULL;
+ u32 *log;
+ int i = 0;
+
+ if (dump_load_err)
+ obj = dev_priv->guc.load_err_log;
+ else if (dev_priv->guc.log.vma)
+ obj = dev_priv->guc.log.vma->obj;
- for (i = 0; i < PAGE_SIZE / sizeof(u32); i += 4)
- seq_printf(m, "0x%08x 0x%08x 0x%08x 0x%08x\n",
- *(log + i), *(log + i + 1),
- *(log + i + 2), *(log + i + 3));
+ if (!obj)
+ return 0;
- kunmap_atomic(log);
+ log = i915_gem_object_pin_map(obj, I915_MAP_WC);
+ if (IS_ERR(log)) {
+ DRM_DEBUG("Failed to pin object\n");
+ seq_puts(m, "(log data unaccessible)\n");
+ return PTR_ERR(log);
}
+ for (i = 0; i < obj->base.size / sizeof(u32); i += 4)
+ seq_printf(m, "0x%08x 0x%08x 0x%08x 0x%08x\n",
+ *(log + i), *(log + i + 1),
+ *(log + i + 2), *(log + i + 3));
+
seq_putc(m, '\n');
+ i915_gem_object_unpin_map(obj);
+
return 0;
}
static int i915_guc_log_control_get(void *data, u64 *val)
{
- struct drm_device *dev = data;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct drm_i915_private *dev_priv = data;
if (!dev_priv->guc.log.vma)
return -EINVAL;
static int i915_guc_log_control_set(void *data, u64 val)
{
- struct drm_device *dev = data;
- struct drm_i915_private *dev_priv = to_i915(dev);
+ struct drm_i915_private *dev_priv = data;
int ret;
if (!dev_priv->guc.log.vma)
return -EINVAL;
- ret = mutex_lock_interruptible(&dev->struct_mutex);
+ ret = mutex_lock_interruptible(&dev_priv->drm.struct_mutex);
if (ret)
return ret;
ret = i915_guc_log_control(dev_priv, val);
intel_runtime_pm_put(dev_priv);
- mutex_unlock(&dev->struct_mutex);
+ mutex_unlock(&dev_priv->drm.struct_mutex);
return ret;
}
seq_printf(m, "version: %d.%d\n", CSR_VERSION_MAJOR(csr->version),
CSR_VERSION_MINOR(csr->version));
- if (IS_SKYLAKE(dev_priv) && csr->version >= CSR_VERSION(1, 6)) {
+ if (IS_KABYLAKE(dev_priv) ||
+ (IS_SKYLAKE(dev_priv) && csr->version >= CSR_VERSION(1, 6))) {
seq_printf(m, "DC3 -> DC5 count: %d\n",
I915_READ(SKL_CSR_DC3_DC5_COUNT));
seq_printf(m, "DC5 -> DC6 count: %d\n",
intel_seq_print_mode(m, 2, mode);
}
-static bool cursor_active(struct drm_i915_private *dev_priv, int pipe)
-{
- u32 state;
-
- if (IS_I845G(dev_priv) || IS_I865G(dev_priv))
- state = I915_READ(CURCNTR(PIPE_A)) & CURSOR_ENABLE;
- else
- state = I915_READ(CURCNTR(pipe)) & CURSOR_MODE;
-
- return state;
-}
-
-static bool cursor_position(struct drm_i915_private *dev_priv,
- int pipe, int *x, int *y)
-{
- u32 pos;
-
- pos = I915_READ(CURPOS(pipe));
-
- *x = (pos >> CURSOR_X_SHIFT) & CURSOR_POS_MASK;
- if (pos & (CURSOR_POS_SIGN << CURSOR_X_SHIFT))
- *x = -*x;
-
- *y = (pos >> CURSOR_Y_SHIFT) & CURSOR_POS_MASK;
- if (pos & (CURSOR_POS_SIGN << CURSOR_Y_SHIFT))
- *y = -*y;
-
- return cursor_active(dev_priv, pipe);
-}
-
static const char *plane_type(enum drm_plane_type type)
{
switch (type) {
seq_printf(m, "CRTC info\n");
seq_printf(m, "---------\n");
for_each_intel_crtc(dev, crtc) {
- bool active;
struct intel_crtc_state *pipe_config;
- int x, y;
drm_modeset_lock(&crtc->base.mutex, NULL);
pipe_config = to_intel_crtc_state(crtc->base.state);
yesno(pipe_config->dither), pipe_config->pipe_bpp);
if (pipe_config->base.active) {
+ struct intel_plane *cursor =
+ to_intel_plane(crtc->base.cursor);
+
intel_crtc_info(m, crtc);
- active = cursor_position(dev_priv, crtc->pipe, &x, &y);
- seq_printf(m, "\tcursor visible? %s, position (%d, %d), size %dx%d, addr 0x%08x, active? %s\n",
- yesno(crtc->cursor_base),
- x, y, crtc->base.cursor->state->crtc_w,
- crtc->base.cursor->state->crtc_h,
- crtc->cursor_addr, yesno(active));
+ seq_printf(m, "\tcursor visible? %s, position (%d, %d), size %dx%d, addr 0x%08x\n",
+ yesno(cursor->base.state->visible),
+ cursor->base.state->crtc_x,
+ cursor->base.state->crtc_y,
+ cursor->base.state->crtc_w,
+ cursor->base.state->crtc_h,
+ cursor->cursor.base);
intel_scaler_info(m, crtc);
intel_plane_info(m, crtc);
}
if (i915.enable_execlists) {
u32 ptr, read, write;
- struct rb_node *rb;
+ unsigned int idx;
seq_printf(m, "\tExeclist status: 0x%08x %08x\n",
I915_READ(RING_EXECLIST_STATUS_LO(engine)),
if (read > write)
write += GEN8_CSB_ENTRIES;
while (read < write) {
- unsigned int idx = ++read % GEN8_CSB_ENTRIES;
-
+ idx = ++read % GEN8_CSB_ENTRIES;
seq_printf(m, "\tExeclist CSB[%d]: 0x%08x, context: %d\n",
idx,
I915_READ(RING_CONTEXT_STATUS_BUF_LO(engine, idx)),
}
rcu_read_lock();
- rq = READ_ONCE(engine->execlist_port[0].request);
- if (rq) {
- seq_printf(m, "\t\tELSP[0] count=%d, ",
- engine->execlist_port[0].count);
- print_request(m, rq, "rq: ");
- } else {
- seq_printf(m, "\t\tELSP[0] idle\n");
- }
- rq = READ_ONCE(engine->execlist_port[1].request);
- if (rq) {
- seq_printf(m, "\t\tELSP[1] count=%d, ",
- engine->execlist_port[1].count);
- print_request(m, rq, "rq: ");
- } else {
- seq_printf(m, "\t\tELSP[1] idle\n");
+ for (idx = 0; idx < ARRAY_SIZE(engine->execlist_port); idx++) {
+ unsigned int count;
+
+ rq = port_unpack(&engine->execlist_port[idx],
+ &count);
+ if (rq) {
+ seq_printf(m, "\t\tELSP[%d] count=%d, ",
+ idx, count);
+ print_request(m, rq, "rq: ");
+ } else {
+ seq_printf(m, "\t\tELSP[%d] idle\n",
+ idx);
+ }
}
rcu_read_unlock();
spin_lock_irq(&engine->timeline->lock);
- for (rb = engine->execlist_first; rb; rb = rb_next(rb)) {
- rq = rb_entry(rb, typeof(*rq), priotree.node);
- print_request(m, rq, "\t\tQ ");
+ for (rb = engine->execlist_first; rb; rb = rb_next(rb)){
+ struct i915_priolist *p =
+ rb_entry(rb, typeof(*p), node);
+
+ list_for_each_entry(rq, &p->requests,
+ priotree.link)
+ print_request(m, rq, "\t\tQ ");
}
spin_unlock_irq(&engine->timeline->lock);
} else if (INTEL_GEN(dev_priv) > 6) {
if (len == 0)
return 0;
- input_buffer = kmalloc(len + 1, GFP_KERNEL);
- if (!input_buffer)
- return -ENOMEM;
+ input_buffer = memdup_user_nul(ubuf, len);
+ if (IS_ERR(input_buffer))
+ return PTR_ERR(input_buffer);
- if (copy_from_user(input_buffer, ubuf, len)) {
- status = -EFAULT;
- goto out;
- }
-
- input_buffer[len] = '\0';
DRM_DEBUG_DRIVER("Copied %d bytes from user\n", (unsigned int)len);
drm_connector_list_iter_begin(dev, &conn_iter);
}
}
drm_connector_list_iter_end(&conn_iter);
-out:
kfree(input_buffer);
if (status < 0)
return status;
num_levels = 3;
else if (IS_VALLEYVIEW(dev_priv))
num_levels = 1;
+ else if (IS_G4X(dev_priv))
+ num_levels = 3;
else
num_levels = ilk_wm_max_level(dev_priv) + 1;
* - WM1+ latency values in 0.5us units
* - latencies are in us on gen9/vlv/chv
*/
- if (INTEL_GEN(dev_priv) >= 9 || IS_VALLEYVIEW(dev_priv) ||
- IS_CHERRYVIEW(dev_priv))
+ if (INTEL_GEN(dev_priv) >= 9 ||
+ IS_VALLEYVIEW(dev_priv) ||
+ IS_CHERRYVIEW(dev_priv) ||
+ IS_G4X(dev_priv))
latency *= 10;
else if (level > 0)
latency *= 5;
{
struct drm_i915_private *dev_priv = inode->i_private;
- if (INTEL_GEN(dev_priv) < 5)
+ if (INTEL_GEN(dev_priv) < 5 && !IS_G4X(dev_priv))
return -ENODEV;
return single_open(file, pri_wm_latency_show, dev_priv);
num_levels = 3;
else if (IS_VALLEYVIEW(dev_priv))
num_levels = 1;
+ else if (IS_G4X(dev_priv))
+ num_levels = 3;
else
num_levels = ilk_wm_max_level(dev_priv) + 1;
{"i915_guc_info", i915_guc_info, 0},
{"i915_guc_load_status", i915_guc_load_status_info, 0},
{"i915_guc_log_dump", i915_guc_log_dump, 0},
+ {"i915_guc_load_err_log_dump", i915_guc_log_dump, 0, (void *)1},
+ {"i915_guc_stage_pool", i915_guc_stage_pool, 0},
{"i915_huc_load_status", i915_huc_load_status_info, 0},
{"i915_frequency_info", i915_frequency_info, 0},
{"i915_hangcheck_info", i915_hangcheck_info, 0},
case I915_PARAM_HAS_EXEC_SOFTPIN:
case I915_PARAM_HAS_EXEC_ASYNC:
case I915_PARAM_HAS_EXEC_FENCE:
+ case I915_PARAM_HAS_EXEC_CAPTURE:
/* For the time being all of these are always true;
* if some supported hardware does not have one of these
* features this value needs to be provided from
intel_uc_init_early(dev_priv);
i915_memcpy_init_early(dev_priv);
- ret = intel_engines_init_early(dev_priv);
- if (ret)
- return ret;
-
ret = i915_workqueues_init(dev_priv);
if (ret < 0)
goto err_engines;
intel_init_audio_hooks(dev_priv);
ret = i915_gem_load_init(dev_priv);
if (ret < 0)
- goto err_workqueues;
+ goto err_irq;
intel_display_crc_init(dev_priv);
return 0;
-err_workqueues:
+err_irq:
+ intel_irq_fini(dev_priv);
i915_workqueues_cleanup(dev_priv);
err_engines:
i915_engines_cleanup(dev_priv);
{
i915_perf_fini(dev_priv);
i915_gem_load_cleanup(dev_priv);
+ intel_irq_fini(dev_priv);
i915_workqueues_cleanup(dev_priv);
i915_engines_cleanup(dev_priv);
}
ret = i915_mmio_setup(dev_priv);
if (ret < 0)
- goto put_bridge;
+ goto err_bridge;
intel_uncore_init(dev_priv);
+
+ ret = intel_engines_init_mmio(dev_priv);
+ if (ret)
+ goto err_uncore;
+
i915_gem_init_mmio(dev_priv);
return 0;
-put_bridge:
+err_uncore:
+ intel_uncore_fini(dev_priv);
+err_bridge:
pci_dev_put(dev_priv->bridge_dev);
return ret;
struct drm_i915_private *dev_priv;
int ret;
- /* Enable nuclear pageflip on ILK+, except vlv/chv */
- if (!i915.nuclear_pageflip &&
- (match_info->gen < 5 || match_info->has_gmch_display))
+ /* Enable nuclear pageflip on ILK+ */
+ if (!i915.nuclear_pageflip && match_info->gen < 5)
driver.driver_features &= ~DRIVER_ATOMIC;
ret = -ENOMEM;
dev_priv->ipc_enabled = false;
- /* Everything is in place, we can now relax! */
- DRM_INFO("Initialized %s %d.%d.%d %s for %s on minor %d\n",
- driver.name, driver.major, driver.minor, driver.patchlevel,
- driver.date, pci_name(pdev), dev_priv->drm.primary->index);
if (IS_ENABLED(CONFIG_DRM_I915_DEBUG))
DRM_INFO("DRM_I915_DEBUG enabled\n");
if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
#include "i915_reg.h"
#include "i915_utils.h"
+#include "intel_uncore.h"
#include "intel_bios.h"
#include "intel_dpll_mgr.h"
#include "intel_uc.h"
#define DRIVER_NAME "i915"
#define DRIVER_DESC "Intel Graphics"
-#define DRIVER_DATE "20170403"
-#define DRIVER_TIMESTAMP 1491198738
+#define DRIVER_DATE "20170529"
+#define DRIVER_TIMESTAMP 1496041258
/* Use I915_STATE_WARN(x) and I915_STATE_WARN_ON() (rather than WARN() and
* WARN_ON()) for hw state sanity checks to check for unexpected conditions
fp; \
})
+static inline bool is_fixed16_zero(uint_fixed_16_16_t val)
+{
+ if (val.val == 0)
+ return true;
+ return false;
+}
+
static inline uint_fixed_16_16_t u32_to_fixed_16_16(uint32_t val)
{
uint_fixed_16_16_t fp;
return max;
}
-static inline uint_fixed_16_16_t fixed_16_16_div_round_up(uint32_t val,
- uint32_t d)
+static inline uint32_t div_round_up_fixed16(uint_fixed_16_16_t val,
+ uint_fixed_16_16_t d)
+{
+ return DIV_ROUND_UP(val.val, d.val);
+}
+
+static inline uint32_t mul_round_up_u32_fixed16(uint32_t val,
+ uint_fixed_16_16_t mul)
+{
+ uint64_t intermediate_val;
+ uint32_t result;
+
+ intermediate_val = (uint64_t) val * mul.val;
+ intermediate_val = DIV_ROUND_UP_ULL(intermediate_val, 1 << 16);
+ WARN_ON(intermediate_val >> 32);
+ result = clamp_t(uint32_t, intermediate_val, 0, ~0);
+ return result;
+}
+
+static inline uint_fixed_16_16_t mul_fixed16(uint_fixed_16_16_t val,
+ uint_fixed_16_16_t mul)
+{
+ uint64_t intermediate_val;
+ uint_fixed_16_16_t fp;
+
+ intermediate_val = (uint64_t) val.val * mul.val;
+ intermediate_val = intermediate_val >> 16;
+ WARN_ON(intermediate_val >> 32);
+ fp.val = clamp_t(uint32_t, intermediate_val, 0, ~0);
+ return fp;
+}
+
+static inline uint_fixed_16_16_t fixed_16_16_div(uint32_t val, uint32_t d)
{
uint_fixed_16_16_t fp, res;
return res;
}
-static inline uint_fixed_16_16_t fixed_16_16_div_round_up_u64(uint32_t val,
- uint32_t d)
+static inline uint_fixed_16_16_t fixed_16_16_div_u64(uint32_t val, uint32_t d)
{
uint_fixed_16_16_t res;
uint64_t interm_val;
return res;
}
+static inline uint32_t div_round_up_u32_fixed16(uint32_t val,
+ uint_fixed_16_16_t d)
+{
+ uint64_t interm_val;
+
+ interm_val = (uint64_t)val << 16;
+ interm_val = DIV_ROUND_UP_ULL(interm_val, d.val);
+ WARN_ON(interm_val >> 32);
+ return clamp_t(uint32_t, interm_val, 0, ~0);
+}
+
static inline uint_fixed_16_16_t mul_u32_fixed_16_16(uint32_t val,
uint_fixed_16_16_t mul)
{
void (*load_luts)(struct drm_crtc_state *crtc_state);
};
-enum forcewake_domain_id {
- FW_DOMAIN_ID_RENDER = 0,
- FW_DOMAIN_ID_BLITTER,
- FW_DOMAIN_ID_MEDIA,
-
- FW_DOMAIN_ID_COUNT
-};
-
-enum forcewake_domains {
- FORCEWAKE_RENDER = BIT(FW_DOMAIN_ID_RENDER),
- FORCEWAKE_BLITTER = BIT(FW_DOMAIN_ID_BLITTER),
- FORCEWAKE_MEDIA = BIT(FW_DOMAIN_ID_MEDIA),
- FORCEWAKE_ALL = (FORCEWAKE_RENDER |
- FORCEWAKE_BLITTER |
- FORCEWAKE_MEDIA)
-};
-
-#define FW_REG_READ (1)
-#define FW_REG_WRITE (2)
-
-enum decoupled_power_domain {
- GEN9_DECOUPLED_PD_BLITTER = 0,
- GEN9_DECOUPLED_PD_RENDER,
- GEN9_DECOUPLED_PD_MEDIA,
- GEN9_DECOUPLED_PD_ALL
-};
-
-enum decoupled_ops {
- GEN9_DECOUPLED_OP_WRITE = 0,
- GEN9_DECOUPLED_OP_READ
-};
-
-enum forcewake_domains
-intel_uncore_forcewake_for_reg(struct drm_i915_private *dev_priv,
- i915_reg_t reg, unsigned int op);
-
-struct intel_uncore_funcs {
- void (*force_wake_get)(struct drm_i915_private *dev_priv,
- enum forcewake_domains domains);
- void (*force_wake_put)(struct drm_i915_private *dev_priv,
- enum forcewake_domains domains);
-
- uint8_t (*mmio_readb)(struct drm_i915_private *dev_priv,
- i915_reg_t r, bool trace);
- uint16_t (*mmio_readw)(struct drm_i915_private *dev_priv,
- i915_reg_t r, bool trace);
- uint32_t (*mmio_readl)(struct drm_i915_private *dev_priv,
- i915_reg_t r, bool trace);
- uint64_t (*mmio_readq)(struct drm_i915_private *dev_priv,
- i915_reg_t r, bool trace);
-
- void (*mmio_writeb)(struct drm_i915_private *dev_priv,
- i915_reg_t r, uint8_t val, bool trace);
- void (*mmio_writew)(struct drm_i915_private *dev_priv,
- i915_reg_t r, uint16_t val, bool trace);
- void (*mmio_writel)(struct drm_i915_private *dev_priv,
- i915_reg_t r, uint32_t val, bool trace);
-};
-
-struct intel_forcewake_range {
- u32 start;
- u32 end;
-
- enum forcewake_domains domains;
-};
-
-struct intel_uncore {
- spinlock_t lock; /** lock is also taken in irq contexts. */
-
- const struct intel_forcewake_range *fw_domains_table;
- unsigned int fw_domains_table_entries;
-
- struct notifier_block pmic_bus_access_nb;
- struct intel_uncore_funcs funcs;
-
- unsigned fifo_count;
-
- enum forcewake_domains fw_domains;
- enum forcewake_domains fw_domains_active;
-
- u32 fw_set;
- u32 fw_clear;
- u32 fw_reset;
-
- struct intel_uncore_forcewake_domain {
- enum forcewake_domain_id id;
- enum forcewake_domains mask;
- unsigned wake_count;
- struct hrtimer timer;
- i915_reg_t reg_set;
- i915_reg_t reg_ack;
- } fw_domain[FW_DOMAIN_ID_COUNT];
-
- int unclaimed_mmio_check;
-};
-
-#define __mask_next_bit(mask) ({ \
- int __idx = ffs(mask) - 1; \
- mask &= ~BIT(__idx); \
- __idx; \
-})
-
-/* Iterate over initialised fw domains */
-#define for_each_fw_domain_masked(domain__, mask__, dev_priv__, tmp__) \
- for (tmp__ = (mask__); \
- tmp__ ? (domain__ = &(dev_priv__)->uncore.fw_domain[__mask_next_bit(tmp__)]), 1 : 0;)
-
-#define for_each_fw_domain(domain__, dev_priv__, tmp__) \
- for_each_fw_domain_masked(domain__, (dev_priv__)->uncore.fw_domains, dev_priv__, tmp__)
-
#define CSR_VERSION(major, minor) ((major) << 16 | (minor))
#define CSR_VERSION_MAJOR(version) ((version) >> 16)
#define CSR_VERSION_MINOR(version) ((version) & 0xffff)
func(has_gmbus_irq); \
func(has_gmch_display); \
func(has_guc); \
+ func(has_guc_ct); \
func(has_hotplug); \
- func(has_hw_contexts); \
func(has_l3_dpf); \
func(has_llc); \
func(has_logical_ring_contexts); \
u32 *pages[0];
} *ringbuffer, *batchbuffer, *wa_batchbuffer, *ctx, *hws_page;
+ struct drm_i915_error_object **user_bo;
+ long user_bo_count;
+
struct drm_i915_error_object *wa_ctx;
struct drm_i915_error_request {
/** LRU list of objects with fence regs on them. */
struct list_head fence_list;
- /**
- * Are we in a non-interruptible section of code like
- * modesetting?
- */
- bool interruptible;
+ u64 unordered_timeline;
/* the indicator for dispatch video commands on two BSD rings */
atomic_t bsd_engine_dispatch_index;
*
* This is a counter which gets incremented when reset is triggered,
*
- * Before the reset commences, the I915_RESET_IN_PROGRESS bit is set
+ * Before the reset commences, the I915_RESET_BACKOFF bit is set
* meaning that any waiters holding onto the struct_mutex should
* relinquish the lock immediately in order for the reset to start.
*
enum intel_ddb_partitioning partitioning;
};
-struct vlv_pipe_wm {
+struct g4x_pipe_wm {
uint16_t plane[I915_MAX_PLANES];
+ uint16_t fbc;
};
-struct vlv_sr_wm {
+struct g4x_sr_wm {
uint16_t plane;
uint16_t cursor;
+ uint16_t fbc;
};
struct vlv_wm_ddl_values {
};
struct vlv_wm_values {
- struct vlv_pipe_wm pipe[3];
- struct vlv_sr_wm sr;
+ struct g4x_pipe_wm pipe[3];
+ struct g4x_sr_wm sr;
struct vlv_wm_ddl_values ddl[3];
uint8_t level;
bool cxsr;
};
+struct g4x_wm_values {
+ struct g4x_pipe_wm pipe[2];
+ struct g4x_sr_wm sr;
+ struct g4x_sr_wm hpll;
+ bool cxsr;
+ bool hpll_en;
+ bool fbc_en;
+};
+
struct skl_ddb_entry {
uint16_t start, end; /* in number of blocks, 'end' is exclusive */
};
size_t *offset);
/**
- * @oa_buffer_is_empty: Check if OA buffer empty (false positives OK)
+ * @oa_buffer_check: Check for OA buffer data + update tail
*
* This is either called via fops or the poll check hrtimer (atomic
* ctx) without any locks taken.
* here, which will be handled gracefully - likely resulting in an
* %EAGAIN error for userspace.
*/
- bool (*oa_buffer_is_empty)(struct drm_i915_private *dev_priv);
+ bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
};
struct intel_cdclk_state {
struct kmem_cache *vmas;
struct kmem_cache *requests;
struct kmem_cache *dependencies;
+ struct kmem_cache *priorities;
const struct intel_device_info info;
*/
struct mutex av_mutex;
- uint32_t hw_context_size;
struct list_head context_list;
u32 fdi_rx_config;
struct ilk_wm_values hw;
struct skl_wm_values skl_hw;
struct vlv_wm_values vlv;
+ struct g4x_wm_values g4x;
};
uint8_t max_level;
wait_queue_head_t poll_wq;
bool pollin;
+ /**
+ * For rate limiting any notifications of spurious
+ * invalid OA reports
+ */
+ struct ratelimit_state spurious_report_rs;
+
bool periodic;
int period_exponent;
- int timestamp_frequency;
-
- int tail_margin;
int metrics_set;
u8 *vaddr;
int format;
int format_size;
+
+ /**
+ * Locks reads and writes to all head/tail state
+ *
+ * Consider: the head and tail pointer state
+ * needs to be read consistently from a hrtimer
+ * callback (atomic context) and read() fop
+ * (user context) with tail pointer updates
+ * happening in atomic context and head updates
+ * in user context and the (unlikely)
+ * possibility of read() errors needing to
+ * reset all head/tail state.
+ *
+ * Note: Contention or performance aren't
+ * currently a significant concern here
+ * considering the relatively low frequency of
+ * hrtimer callbacks (5ms period) and that
+ * reads typically only happen in response to a
+ * hrtimer event and likely complete before the
+ * next callback.
+ *
+ * Note: This lock is not held *while* reading
+ * and copying data to userspace so the value
+ * of head observed in htrimer callbacks won't
+ * represent any partial consumption of data.
+ */
+ spinlock_t ptr_lock;
+
+ /**
+ * One 'aging' tail pointer and one 'aged'
+ * tail pointer ready to used for reading.
+ *
+ * Initial values of 0xffffffff are invalid
+ * and imply that an update is required
+ * (and should be ignored by an attempted
+ * read)
+ */
+ struct {
+ u32 offset;
+ } tails[2];
+
+ /**
+ * Index for the aged tail ready to read()
+ * data up to.
+ */
+ unsigned int aged_tail_idx;
+
+ /**
+ * A monotonic timestamp for when the current
+ * aging tail pointer was read; used to
+ * determine when it is old enough to trust.
+ */
+ u64 aging_timestamp;
+
+ /**
+ * Although we can always read back the head
+ * pointer register, we prefer to avoid
+ * trusting the HW state, just to avoid any
+ * risk that some hardware condition could
+ * somehow bump the head pointer unpredictably
+ * and cause us to forward the wrong OA buffer
+ * data to userspace.
+ */
+ u32 head;
} oa_buffer;
u32 gen7_latched_oastatus1;
#define HWS_NEEDS_PHYSICAL(dev_priv) ((dev_priv)->info.hws_needs_physical)
-#define HAS_HW_CONTEXTS(dev_priv) ((dev_priv)->info.has_hw_contexts)
#define HAS_LOGICAL_RING_CONTEXTS(dev_priv) \
((dev_priv)->info.has_logical_ring_contexts)
#define USES_PPGTT(dev_priv) (i915.enable_ppgtt)
#define HAS_FW_BLC(dev_priv) (INTEL_GEN(dev_priv) > 2)
#define HAS_PIPE_CXSR(dev_priv) ((dev_priv)->info.has_pipe_cxsr)
#define HAS_FBC(dev_priv) ((dev_priv)->info.has_fbc)
+#define HAS_CUR_FBC(dev_priv) (!HAS_GMCH_DISPLAY(dev_priv) && INTEL_INFO(dev_priv)->gen >= 7)
#define HAS_IPS(dev_priv) (IS_HSW_ULT(dev_priv) || IS_BROADWELL(dev_priv))
* properties, so we have separate macros to test them.
*/
#define HAS_GUC(dev_priv) ((dev_priv)->info.has_guc)
+#define HAS_GUC_CT(dev_priv) ((dev_priv)->info.has_guc_ct)
#define HAS_GUC_UCODE(dev_priv) (HAS_GUC(dev_priv))
#define HAS_GUC_SCHED(dev_priv) (HAS_GUC(dev_priv))
#define HAS_HUC_UCODE(dev_priv) (HAS_GUC(dev_priv))
#include "i915_trace.h"
-static inline bool intel_scanout_needs_vtd_wa(struct drm_i915_private *dev_priv)
+static inline bool intel_vtd_active(void)
{
#ifdef CONFIG_INTEL_IOMMU
- if (INTEL_GEN(dev_priv) >= 6 && intel_iommu_gfx_mapped)
+ if (intel_iommu_gfx_mapped)
return true;
#endif
return false;
}
+static inline bool intel_scanout_needs_vtd_wa(struct drm_i915_private *dev_priv)
+{
+ return INTEL_GEN(dev_priv) >= 6 && intel_vtd_active();
+}
+
+static inline bool
+intel_ggtt_update_needs_vtd_wa(struct drm_i915_private *dev_priv)
+{
+ return IS_BROXTON(dev_priv) && intel_vtd_active();
+}
+
int intel_sanitize_enable_ppgtt(struct drm_i915_private *dev_priv,
int enable_ppgtt);
extern void i915_update_gfx_val(struct drm_i915_private *dev_priv);
int vlv_force_gfx_clock(struct drm_i915_private *dev_priv, bool on);
-int intel_engines_init_early(struct drm_i915_private *dev_priv);
+int intel_engines_init_mmio(struct drm_i915_private *dev_priv);
int intel_engines_init(struct drm_i915_private *dev_priv);
/* intel_hotplug.c */
const char *fmt, ...);
extern void intel_irq_init(struct drm_i915_private *dev_priv);
+extern void intel_irq_fini(struct drm_i915_private *dev_priv);
int intel_irq_install(struct drm_i915_private *dev_priv);
void intel_irq_uninstall(struct drm_i915_private *dev_priv);
-extern void intel_uncore_sanitize(struct drm_i915_private *dev_priv);
-extern void intel_uncore_init(struct drm_i915_private *dev_priv);
-extern bool intel_uncore_unclaimed_mmio(struct drm_i915_private *dev_priv);
-extern bool intel_uncore_arm_unclaimed_mmio_detection(struct drm_i915_private *dev_priv);
-extern void intel_uncore_fini(struct drm_i915_private *dev_priv);
-extern void intel_uncore_suspend(struct drm_i915_private *dev_priv);
-extern void intel_uncore_resume_early(struct drm_i915_private *dev_priv);
-const char *intel_uncore_forcewake_domain_to_str(const enum forcewake_domain_id id);
-void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
- enum forcewake_domains domains);
-void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
- enum forcewake_domains domains);
-/* Like above but the caller must manage the uncore.lock itself.
- * Must be used with I915_READ_FW and friends.
- */
-void intel_uncore_forcewake_get__locked(struct drm_i915_private *dev_priv,
- enum forcewake_domains domains);
-void intel_uncore_forcewake_put__locked(struct drm_i915_private *dev_priv,
- enum forcewake_domains domains);
-u64 intel_uncore_edram_size(struct drm_i915_private *dev_priv);
-
-void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
-
-int intel_wait_for_register(struct drm_i915_private *dev_priv,
- i915_reg_t reg,
- const u32 mask,
- const u32 value,
- const unsigned long timeout_ms);
-int intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
- i915_reg_t reg,
- const u32 mask,
- const u32 value,
- const unsigned long timeout_ms);
-
static inline bool intel_gvt_active(struct drm_i915_private *dev_priv)
{
return dev_priv->gvt;
#define I915_PRIORITY_DISPLAY I915_PRIORITY_MAX
int __must_check
-i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj,
- bool write);
+i915_gem_object_set_to_wc_domain(struct drm_i915_gem_object *obj, bool write);
+int __must_check
+i915_gem_object_set_to_gtt_domain(struct drm_i915_gem_object *obj, bool write);
int __must_check
i915_gem_object_set_to_cpu_domain(struct drm_i915_gem_object *obj, bool write);
struct i915_vma * __must_check
void intel_lpe_audio_teardown(struct drm_i915_private *dev_priv);
void intel_lpe_audio_irq_handler(struct drm_i915_private *dev_priv);
void intel_lpe_audio_notify(struct drm_i915_private *dev_priv,
- void *eld, int port, int pipe, int tmds_clk_speed,
- bool dp_output, int link_rate);
+ enum pipe pipe, enum port port,
+ const void *eld, int ls_clock, bool dp_output);
/* intel_i2c.c */
extern int intel_setup_gmbus(struct drm_i915_private *dev_priv);
#include <linux/dma-buf.h>
static void i915_gem_flush_free_objects(struct drm_i915_private *i915);
-static void i915_gem_object_flush_gtt_write_domain(struct drm_i915_gem_object *obj);
-static void i915_gem_object_flush_cpu_write_domain(struct drm_i915_gem_object *obj);
static bool cpu_write_needs_clflush(struct drm_i915_gem_object *obj)
{
args->size, &args->handle);
}
+static inline enum fb_op_origin
+fb_write_origin(struct drm_i915_gem_object *obj, unsigned int domain)
+{
+ return (domain == I915_GEM_DOMAIN_GTT ?
+ obj->frontbuffer_ggtt_origin : ORIGIN_CPU);
+}
+
+static void
+flush_write_domain(struct drm_i915_gem_object *obj, unsigned int flush_domains)
+{
+ struct drm_i915_private *dev_priv = to_i915(obj->base.dev);
+
+ if (!(obj->base.write_domain & flush_domains))
+ return;
+
+ /* No actual flushing is required for the GTT write domain. Writes
+ * to it "immediately" go to main memory as far as we know, so there's
+ * no chipset flush. It also doesn't land in render cache.
+ *
+ * However, we do have to enforce the order so that all writes through
+ * the GTT land before any writes to the device, such as updates to
+ * the GATT itself.
+ *
+ * We also have to wait a bit for the writes to land from the GTT.
+ * An uncached read (i.e. mmio) seems to be ideal for the round-trip
+ * timing. This issue has only been observed when switching quickly
+ * between GTT writes and CPU reads from inside the kernel on recent hw,
+ * and it appears to only affect discrete GTT blocks (i.e. on LLC
+ * system agents we cannot reproduce this behaviour).
+ */
+ wmb();
+
+ switch (obj->base.write_domain) {
+ case I915_GEM_DOMAIN_GTT:
+ if (INTEL_GEN(dev_priv) >= 6 && !HAS_LLC(dev_priv)) {
+ if (intel_runtime_pm_get_if_in_use(dev_priv)) {
+ spin_lock_irq(&dev_priv->uncore.lock);
+ POSTING_READ_FW(RING_ACTHD(dev_priv->engine[RCS]->mmio_base));
+ spin_unlock_irq(&dev_priv->uncore.lock);
+ intel_runtime_pm_put(dev_priv);
+ }
+ }
+
+ intel_fb_obj_flush(obj,
+ fb_write_origin(obj, I915_GEM_DOMAIN_GTT));
+ break;
+
+ case I915_GEM_DOMAIN_CPU:
+ i915_gem_clflush_object(obj, I915_CLFLUSH_SYNC);
+ break;
+ }
+
+ obj->base.write_domain = 0;
+}
+
static inline int
__copy_to_user_swizzled(char __user *cpu_vaddr,
const char *gpu_vaddr, int gpu_offset,
goto out;
}
- i915_gem_object_flush_gtt_write_domain(obj);
+ flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);
/* If we're not in the cpu read domain, set ourself into the gtt
* read domain and manually flush cachelines (if required). This
goto out;
}
- i915_gem_object_flush_gtt_write_domain(obj);
+ flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);
/* If we're not in the cpu write domain, set ourself into the
* gtt write domain and manually flush cachelines (as required).
return ret;
}
-static inline enum fb_op_origin
-write_origin(struct drm_i915_gem_object *obj, unsigned domain)
-{
- return (domain == I915_GEM_DOMAIN_GTT ?
- obj->frontbuffer_ggtt_origin : ORIGIN_CPU);
-}
-
static void i915_gem_object_bump_inactive_ggtt(struct drm_i915_gem_object *obj)
{
struct drm_i915_private *i915;
if (err)
goto out_unpin;
- if (read_domains & I915_GEM_DOMAIN_GTT)
- err = i915_gem_object_set_to_gtt_domain(obj, write_domain != 0);
+ if (read_domains & I915_GEM_DOMAIN_WC)
+ err = i915_gem_object_set_to_wc_domain(obj, write_domain);
+ else if (read_domains & I915_GEM_DOMAIN_GTT)
+ err = i915_gem_object_set_to_gtt_domain(obj, write_domain);
else
- err = i915_gem_object_set_to_cpu_domain(obj, write_domain != 0);
+ err = i915_gem_object_set_to_cpu_domain(obj, write_domain);
/* And bump the LRU for this access */
i915_gem_object_bump_inactive_ggtt(obj);
mutex_unlock(&dev->struct_mutex);
if (write_domain != 0)
- intel_fb_obj_invalidate(obj, write_origin(obj, write_domain));
+ intel_fb_obj_invalidate(obj,
+ fb_write_origin(obj, write_domain));
out_unpin:
i915_gem_object_unpin_pages(obj);
* into userspace. (This view is aligned and sized appropriately for
* fenced access.)
*
+ * 2 - Recognise WC as a separate cache domain so that we can flush the
+ * delayed writes via GTT before performing direct access via WC.
+ *
* Restrictions:
*
* * snoopable objects cannot be accessed via the GTT. It can cause machine
*/
int i915_gem_mmap_gtt_version(void)
{
- return 1;
+ return 2;
}
static inline struct i915_ggtt_view
if (obj->mm.mapping) {
void *ptr;
- ptr = ptr_mask_bits(obj->mm.mapping);
+ ptr = page_mask_bits(obj->mm.mapping);
if (is_vmalloc_addr(ptr))
vunmap(ptr);
else
}
GEM_BUG_ON(!obj->mm.pages);
- ptr = ptr_unpack_bits(obj->mm.mapping, has_type);
+ ptr = page_unpack_bits(obj->mm.mapping, &has_type);
if (ptr && has_type != type) {
if (pinned) {
ret = -EBUSY;
goto err_unpin;
}
- obj->mm.mapping = ptr_pack_bits(ptr, type);
+ obj->mm.mapping = page_pack_bits(ptr, type);
}
out_unlock:
*/
if (i915.enable_execlists) {
+ struct execlist_port *port = engine->execlist_port;
unsigned long flags;
+ unsigned int n;
spin_lock_irqsave(&engine->timeline->lock, flags);
- i915_gem_request_put(engine->execlist_port[0].request);
- i915_gem_request_put(engine->execlist_port[1].request);
+ for (n = 0; n < ARRAY_SIZE(engine->execlist_port); n++)
+ i915_gem_request_put(port_request(&port[n]));
memset(engine->execlist_port, 0, sizeof(engine->execlist_port));
engine->execlist_queue = RB_ROOT;
engine->execlist_first = NULL;
struct drm_i915_private *dev_priv =
container_of(work, typeof(*dev_priv), gt.idle_work.work);
struct drm_device *dev = &dev_priv->drm;
- struct intel_engine_cs *engine;
- enum intel_engine_id id;
bool rearm_hangcheck;
if (!READ_ONCE(dev_priv->gt.awake))
if (wait_for(intel_engines_are_idle(dev_priv), 10))
DRM_ERROR("Timeout waiting for engines to idle\n");
- for_each_engine(engine, dev_priv, id) {
- intel_engine_disarm_breadcrumbs(engine);
- i915_gem_batch_pool_fini(&engine->batch_pool);
- }
+ intel_engines_mark_idle(dev_priv);
+ i915_gem_timelines_mark_idle(dev_priv);
GEM_BUG_ON(!dev_priv->gt.awake);
dev_priv->gt.awake = false;
return ret;
}
-/** Flushes the GTT write domain for the object if it's dirty. */
-static void
-i915_gem_object_flush_gtt_write_domain(struct drm_i915_gem_object *obj)
-{
- struct drm_i915_private *dev_priv = to_i915(obj->base.dev);
-
- if (obj->base.write_domain != I915_GEM_DOMAIN_GTT)
- return;
-
- /* No actual flushing is required for the GTT write domain. Writes
- * to it "immediately" go to main memory as far as we know, so there's
- * no chipset flush. It also doesn't land in render cache.
- *
- * However, we do have to enforce the order so that all writes through
- * the GTT land before any writes to the device, such as updates to
- * the GATT itself.
- *
- * We also have to wait a bit for the writes to land from the GTT.
- * An uncached read (i.e. mmio) seems to be ideal for the round-trip
- * timing. This issue has only been observed when switching quickly
- * between GTT writes and CPU reads from inside the kernel on recent hw,
- * and it appears to only affect discrete GTT blocks (i.e. on LLC
- * system agents we cannot reproduce this behaviour).
- */
- wmb();
- if (INTEL_GEN(dev_priv) >= 6 && !HAS_LLC(dev_priv)) {
- if (intel_runtime_pm_get_if_in_use(dev_priv)) {
- spin_lock_irq(&dev_priv->uncore.lock);
- POSTING_READ_FW(RING_ACTHD(dev_priv->engine[RCS]->mmio_base));
- spin_unlock_irq(&dev_priv->uncore.lock);
- intel_runtime_pm_put(dev_priv);
- }
- }
-
- intel_fb_obj_flush(obj, write_origin(obj, I915_GEM_DOMAIN_GTT));
-
- obj->base.write_domain = 0;
-}
-
-/** Flushes the CPU write domain for the object if it's dirty. */
-static void
-i915_gem_object_flush_cpu_write_domain(struct drm_i915_gem_object *obj)
-{
- if (obj->base.write_domain != I915_GEM_DOMAIN_CPU)
- return;
-
- i915_gem_clflush_object(obj, I915_CLFLUSH_SYNC);
- obj->base.write_domain = 0;
-}
-
static void __i915_gem_object_flush_for_display(struct drm_i915_gem_object *obj)
{
if (obj->base.write_domain != I915_GEM_DOMAIN_CPU && !obj->cache_dirty)
mutex_unlock(&obj->base.dev->struct_mutex);
}
+/**
+ * Moves a single object to the WC read, and possibly write domain.
+ * @obj: object to act on
+ * @write: ask for write access or read only
+ *
+ * This function returns when the move is complete, including waiting on
+ * flushes to occur.
+ */
+int
+i915_gem_object_set_to_wc_domain(struct drm_i915_gem_object *obj, bool write)
+{
+ int ret;
+
+ lockdep_assert_held(&obj->base.dev->struct_mutex);
+
+ ret = i915_gem_object_wait(obj,
+ I915_WAIT_INTERRUPTIBLE |
+ I915_WAIT_LOCKED |
+ (write ? I915_WAIT_ALL : 0),
+ MAX_SCHEDULE_TIMEOUT,
+ NULL);
+ if (ret)
+ return ret;
+
+ if (obj->base.write_domain == I915_GEM_DOMAIN_WC)
+ return 0;
+
+ /* Flush and acquire obj->pages so that we are coherent through
+ * direct access in memory with previous cached writes through
+ * shmemfs and that our cache domain tracking remains valid.
+ * For example, if the obj->filp was moved to swap without us
+ * being notified and releasing the pages, we would mistakenly
+ * continue to assume that the obj remained out of the CPU cached
+ * domain.
+ */
+ ret = i915_gem_object_pin_pages(obj);
+ if (ret)
+ return ret;
+
+ flush_write_domain(obj, ~I915_GEM_DOMAIN_WC);
+
+ /* Serialise direct access to this object with the barriers for
+ * coherent writes from the GPU, by effectively invalidating the
+ * WC domain upon first access.
+ */
+ if ((obj->base.read_domains & I915_GEM_DOMAIN_WC) == 0)
+ mb();
+
+ /* It should now be out of any other write domains, and we can update
+ * the domain values for our changes.
+ */
+ GEM_BUG_ON((obj->base.write_domain & ~I915_GEM_DOMAIN_WC) != 0);
+ obj->base.read_domains |= I915_GEM_DOMAIN_WC;
+ if (write) {
+ obj->base.read_domains = I915_GEM_DOMAIN_WC;
+ obj->base.write_domain = I915_GEM_DOMAIN_WC;
+ obj->mm.dirty = true;
+ }
+
+ i915_gem_object_unpin_pages(obj);
+ return 0;
+}
+
/**
* Moves a single object to the GTT read, and possibly write domain.
* @obj: object to act on
if (ret)
return ret;
- i915_gem_object_flush_cpu_write_domain(obj);
+ flush_write_domain(obj, ~I915_GEM_DOMAIN_GTT);
/* Serialise direct access to this object with the barriers for
* coherent writes from the GPU, by effectively invalidating the
if (obj->base.write_domain == I915_GEM_DOMAIN_CPU)
return 0;
- i915_gem_object_flush_gtt_write_domain(obj);
+ flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);
/* Flush the CPU cache if it's still invalid. */
if ((obj->base.read_domains & I915_GEM_DOMAIN_CPU) == 0) {
if (i915_gem_request_completed(rq))
return 0;
- return flag(rq->engine->exec_id);
+ return flag(rq->engine->uabi_id);
}
static __always_inline unsigned int
* catch if we ever need to fix it. In the meantime, if you do spot
* such a local variable, please consider fixing!
*/
- if (WARN_ON(size >> PAGE_SHIFT > INT_MAX))
+ if (size >> PAGE_SHIFT > INT_MAX)
return ERR_PTR(-E2BIG);
if (overflows_type(size, obj->base.size))
intel_runtime_pm_put(i915);
mutex_unlock(&i915->drm.struct_mutex);
+ cond_resched();
+
llist_for_each_entry_safe(obj, on, freed, freed) {
GEM_BUG_ON(obj->bind_count);
GEM_BUG_ON(atomic_read(&obj->frontbuffer_bits));
* unbound now.
*/
- while ((freed = llist_del_all(&i915->mm.free_list)))
+ while ((freed = llist_del_all(&i915->mm.free_list))) {
__i915_gem_free_objects(i915, freed);
+ if (need_resched())
+ break;
+ }
}
static void __i915_gem_free_object_rcu(struct rcu_head *head)
* try to take over. The only way to remove the earlier state
* is by resetting. However, resetting on earlier gen is tricky as
* it may impact the display and we are uncertain about the stability
- * of the reset, so we only reset recent machines with logical
- * context support (that must be reset to remove any stray contexts).
+ * of the reset, so this could be applied to even earlier gen.
*/
- if (HAS_HW_CONTEXTS(i915)) {
+ if (INTEL_GEN(i915) >= 5) {
int reset = intel_gpu_reset(i915, ALL_ENGINES);
WARN_ON(reset && reset != -ENODEV);
}
if (value >= 0)
return value;
-#ifdef CONFIG_INTEL_IOMMU
/* Enable semaphores on SNB when IO remapping is off */
- if (INTEL_INFO(dev_priv)->gen == 6 && intel_iommu_gfx_mapped)
+ if (IS_GEN6(dev_priv) && intel_vtd_active())
return false;
-#endif
return true;
}
mutex_lock(&dev_priv->drm.struct_mutex);
- i915_gem_clflush_init(dev_priv);
+ dev_priv->mm.unordered_timeline = dma_fence_context_alloc(1);
if (!i915.enable_execlists) {
dev_priv->gt.resume = intel_legacy_submission_resume;
if (!dev_priv->dependencies)
goto err_requests;
+ dev_priv->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN);
+ if (!dev_priv->priorities)
+ goto err_dependencies;
+
mutex_lock(&dev_priv->drm.struct_mutex);
INIT_LIST_HEAD(&dev_priv->gt.timelines);
err = i915_gem_timeline_init__global(dev_priv);
mutex_unlock(&dev_priv->drm.struct_mutex);
if (err)
- goto err_dependencies;
+ goto err_priorities;
INIT_LIST_HEAD(&dev_priv->context_list);
INIT_WORK(&dev_priv->mm.free_work, __i915_gem_free_work);
init_waitqueue_head(&dev_priv->pending_flip_queue);
- dev_priv->mm.interruptible = true;
-
atomic_set(&dev_priv->mm.bsd_engine_dispatch_index, 0);
spin_lock_init(&dev_priv->fb_tracking.lock);
return 0;
+err_priorities:
+ kmem_cache_destroy(dev_priv->priorities);
err_dependencies:
kmem_cache_destroy(dev_priv->dependencies);
err_requests:
WARN_ON(!list_empty(&dev_priv->gt.timelines));
mutex_unlock(&dev_priv->drm.struct_mutex);
+ kmem_cache_destroy(dev_priv->priorities);
kmem_cache_destroy(dev_priv->dependencies);
kmem_cache_destroy(dev_priv->requests);
kmem_cache_destroy(dev_priv->vmas);
int i915_gem_freeze(struct drm_i915_private *dev_priv)
{
- mutex_lock(&dev_priv->drm.struct_mutex);
+ /* Discard all purgeable objects, let userspace recover those as
+ * required after resuming.
+ */
i915_gem_shrink_all(dev_priv);
- mutex_unlock(&dev_priv->drm.struct_mutex);
return 0;
}
* we update that state just before writing out the image.
*
* To try and reduce the hibernation image, we manually shrink
- * the objects as well.
+ * the objects as well, see i915_gem_freeze()
*/
- mutex_lock(&dev_priv->drm.struct_mutex);
i915_gem_shrink(dev_priv, -1UL, I915_SHRINK_UNBOUND);
+ i915_gem_drain_freed_objects(dev_priv);
+ mutex_lock(&dev_priv->drm.struct_mutex);
for (p = phases; *p; p++) {
list_for_each_entry(obj, *p, global_link) {
obj->base.read_domains = I915_GEM_DOMAIN_CPU;
#ifndef __I915_GEM_H__
#define __I915_GEM_H__
+#include <linux/bug.h>
+
#ifdef CONFIG_DRM_I915_DEBUG_GEM
#define GEM_BUG_ON(expr) BUG_ON(expr)
#define GEM_WARN_ON(expr) WARN_ON(expr)
#include "i915_gem_clflush.h"
static DEFINE_SPINLOCK(clflush_lock);
-static u64 clflush_context;
struct clflush {
struct dma_fence dma; /* Must be first for dma_fence_free() */
dma_fence_init(&clflush->dma,
&i915_clflush_ops,
&clflush_lock,
- clflush_context,
+ to_i915(obj->base.dev)->mm.unordered_timeline,
0);
i915_sw_fence_init(&clflush->wait, i915_clflush_notify);
GEM_BUG_ON(obj->base.write_domain != I915_GEM_DOMAIN_CPU);
}
}
-
-void i915_gem_clflush_init(struct drm_i915_private *i915)
-{
- clflush_context = dma_fence_context_alloc(1);
-}
struct drm_i915_private;
struct drm_i915_gem_object;
-void i915_gem_clflush_init(struct drm_i915_private *i915);
void i915_gem_clflush_object(struct drm_i915_gem_object *obj,
unsigned int flags);
#define I915_CLFLUSH_FORCE BIT(0)
#define ALL_L3_SLICES(dev) (1 << NUM_L3_SLICES(dev)) - 1
-static int get_context_size(struct drm_i915_private *dev_priv)
-{
- int ret;
- u32 reg;
-
- switch (INTEL_GEN(dev_priv)) {
- case 6:
- reg = I915_READ(CXT_SIZE);
- ret = GEN6_CXT_TOTAL_SIZE(reg) * 64;
- break;
- case 7:
- reg = I915_READ(GEN7_CXT_SIZE);
- if (IS_HASWELL(dev_priv))
- ret = HSW_CXT_TOTAL_SIZE;
- else
- ret = GEN7_CXT_TOTAL_SIZE(reg) * 64;
- break;
- case 8:
- ret = GEN8_CXT_TOTAL_SIZE;
- break;
- default:
- BUG();
- }
-
- return ret;
-}
-
void i915_gem_context_free(struct kref *ctx_ref)
{
struct i915_gem_context *ctx = container_of(ctx_ref, typeof(*ctx), ref);
kfree(ctx);
}
-static struct drm_i915_gem_object *
-alloc_context_obj(struct drm_i915_private *dev_priv, u64 size)
-{
- struct drm_i915_gem_object *obj;
- int ret;
-
- lockdep_assert_held(&dev_priv->drm.struct_mutex);
-
- obj = i915_gem_object_create(dev_priv, size);
- if (IS_ERR(obj))
- return obj;
-
- /*
- * Try to make the context utilize L3 as well as LLC.
- *
- * On VLV we don't have L3 controls in the PTEs so we
- * shouldn't touch the cache level, especially as that
- * would make the object snooped which might have a
- * negative performance impact.
- *
- * Snooping is required on non-llc platforms in execlist
- * mode, but since all GGTT accesses use PAT entry 0 we
- * get snooping anyway regardless of cache_level.
- *
- * This is only applicable for Ivy Bridge devices since
- * later platforms don't have L3 control bits in the PTE.
- */
- if (IS_IVYBRIDGE(dev_priv)) {
- ret = i915_gem_object_set_cache_level(obj, I915_CACHE_L3_LLC);
- /* Failure shouldn't ever happen this early */
- if (WARN_ON(ret)) {
- i915_gem_object_put(obj);
- return ERR_PTR(ret);
- }
- }
-
- return obj;
-}
-
static void context_close(struct i915_gem_context *ctx)
{
i915_gem_context_set_closed(ctx);
kref_init(&ctx->ref);
list_add_tail(&ctx->link, &dev_priv->context_list);
ctx->i915 = dev_priv;
-
- if (dev_priv->hw_context_size) {
- struct drm_i915_gem_object *obj;
- struct i915_vma *vma;
-
- obj = alloc_context_obj(dev_priv, dev_priv->hw_context_size);
- if (IS_ERR(obj)) {
- ret = PTR_ERR(obj);
- goto err_out;
- }
-
- vma = i915_vma_instance(obj, &dev_priv->ggtt.base, NULL);
- if (IS_ERR(vma)) {
- i915_gem_object_put(obj);
- ret = PTR_ERR(vma);
- goto err_out;
- }
-
- ctx->engine[RCS].state = vma;
- }
+ ctx->priority = I915_PRIORITY_NORMAL;
/* Default context will never have a file_priv */
ret = DEFAULT_CONTEXT_HANDLE;
BUILD_BUG_ON(MAX_CONTEXT_HW_ID > INT_MAX);
ida_init(&dev_priv->context_hw_ida);
- if (i915.enable_execlists) {
- /* NB: intentionally left blank. We will allocate our own
- * backing objects as we need them, thank you very much */
- dev_priv->hw_context_size = 0;
- } else if (HAS_HW_CONTEXTS(dev_priv)) {
- dev_priv->hw_context_size =
- round_up(get_context_size(dev_priv),
- I915_GTT_PAGE_SIZE);
- if (dev_priv->hw_context_size > (1<<20)) {
- DRM_DEBUG_DRIVER("Disabling HW Contexts; invalid size %d\n",
- dev_priv->hw_context_size);
- dev_priv->hw_context_size = 0;
- }
- }
-
ctx = i915_gem_create_context(dev_priv, NULL);
if (IS_ERR(ctx)) {
DRM_ERROR("Failed to create default global context (error %ld)\n",
GEM_BUG_ON(!i915_gem_context_is_kernel(ctx));
DRM_DEBUG_DRIVER("%s context support initialized\n",
- i915.enable_execlists ? "LR" :
- dev_priv->hw_context_size ? "HW" : "fake");
+ dev_priv->engine[RCS]->context_size ? "logical" :
+ "fake");
return 0;
}
return 0;
}
-static bool contexts_enabled(struct drm_device *dev)
-{
- return i915.enable_execlists || to_i915(dev)->hw_context_size;
-}
-
static bool client_is_banned(struct drm_i915_file_private *file_priv)
{
return file_priv->context_bans > I915_MAX_CLIENT_CONTEXT_BANS;
int i915_gem_context_create_ioctl(struct drm_device *dev, void *data,
struct drm_file *file)
{
+ struct drm_i915_private *dev_priv = to_i915(dev);
struct drm_i915_gem_context_create *args = data;
struct drm_i915_file_private *file_priv = file->driver_priv;
struct i915_gem_context *ctx;
int ret;
- if (!contexts_enabled(dev))
+ if (!dev_priv->engine[RCS]->context_size)
return -ENODEV;
if (args->pad != 0)
if (ret)
return ret;
- ctx = i915_gem_create_context(to_i915(dev), file_priv);
+ ctx = i915_gem_create_context(dev_priv, file_priv);
mutex_unlock(&dev->struct_mutex);
if (IS_ERR(ctx))
return PTR_ERR(ctx);
}
static void *i915_gem_dmabuf_kmap(struct dma_buf *dma_buf, unsigned long page_num)
{
+ struct drm_i915_gem_object *obj = dma_buf_to_obj(dma_buf);
+ struct page *page;
+
+ if (page_num >= obj->base.size >> PAGE_SHIFT)
+ return NULL;
+
+ if (!i915_gem_object_has_struct_page(obj))
+ return NULL;
+
+ if (i915_gem_object_pin_pages(obj))
+ return NULL;
+
+ /* Synchronisation is left to the caller (via .begin_cpu_access()) */
+ page = i915_gem_object_get_page(obj, page_num);
+ if (IS_ERR(page))
+ goto err_unpin;
+
+ return kmap(page);
+
+err_unpin:
+ i915_gem_object_unpin_pages(obj);
return NULL;
}
static void i915_gem_dmabuf_kunmap(struct dma_buf *dma_buf, unsigned long page_num, void *addr)
{
+ struct drm_i915_gem_object *obj = dma_buf_to_obj(dma_buf);
+ kunmap(virt_to_page(addr));
+ i915_gem_object_unpin_pages(obj);
}
static int i915_gem_dmabuf_mmap(struct dma_buf *dma_buf, struct vm_area_struct *vma)
list_for_each_entry(vma, vmas, exec_list) {
struct drm_i915_gem_object *obj = vma->obj;
+ if (vma->exec_entry->flags & EXEC_OBJECT_CAPTURE) {
+ struct i915_gem_capture_list *capture;
+
+ capture = kmalloc(sizeof(*capture), GFP_KERNEL);
+ if (unlikely(!capture))
+ return -ENOMEM;
+
+ capture->next = req->capture_list;
+ capture->vma = vma;
+ req->capture_list = capture;
+ }
+
if (vma->exec_entry->flags & EXEC_OBJECT_ASYNC)
continue;
if (enable_ppgtt == 3 && has_full_48bit_ppgtt)
return 3;
-#ifdef CONFIG_INTEL_IOMMU
/* Disable ppgtt on SNB if VT-d is on. */
- if (IS_GEN6(dev_priv) && intel_iommu_gfx_mapped) {
+ if (IS_GEN6(dev_priv) && intel_vtd_active()) {
DRM_INFO("Disabling PPGTT because VT-d is on\n");
return 0;
}
-#endif
/* Early VLV doesn't have this */
if (IS_VALLEYVIEW(dev_priv) && dev_priv->drm.pdev->revision < 0xb) {
u32 pte_flags;
int ret;
- ret = vma->vm->allocate_va_range(vma->vm, vma->node.start, vma->size);
- if (ret)
- return ret;
+ if (!(vma->flags & I915_VMA_LOCAL_BIND)) {
+ ret = vma->vm->allocate_va_range(vma->vm, vma->node.start,
+ vma->size);
+ if (ret)
+ return ret;
+ }
vma->pages = vma->obj->mm.pages;
*/
static bool needs_idle_maps(struct drm_i915_private *dev_priv)
{
-#ifdef CONFIG_INTEL_IOMMU
/* Query intel_iommu to see if we need the workaround. Presumably that
* was loaded first.
*/
- if (IS_GEN5(dev_priv) && IS_MOBILE(dev_priv) && intel_iommu_gfx_mapped)
- return true;
-#endif
- return false;
+ return IS_GEN5(dev_priv) && IS_MOBILE(dev_priv) && intel_vtd_active();
}
void i915_check_and_clear_faults(struct drm_i915_private *dev_priv)
gen8_set_pte(>t_base[i], scratch_pte);
}
+static void bxt_vtd_ggtt_wa(struct i915_address_space *vm)
+{
+ struct drm_i915_private *dev_priv = vm->i915;
+
+ /*
+ * Make sure the internal GAM fifo has been cleared of all GTT
+ * writes before exiting stop_machine(). This guarantees that
+ * any aperture accesses waiting to start in another process
+ * cannot back up behind the GTT writes causing a hang.
+ * The register can be any arbitrary GAM register.
+ */
+ POSTING_READ(GFX_FLSH_CNTL_GEN6);
+}
+
+struct insert_page {
+ struct i915_address_space *vm;
+ dma_addr_t addr;
+ u64 offset;
+ enum i915_cache_level level;
+};
+
+static int bxt_vtd_ggtt_insert_page__cb(void *_arg)
+{
+ struct insert_page *arg = _arg;
+
+ gen8_ggtt_insert_page(arg->vm, arg->addr, arg->offset, arg->level, 0);
+ bxt_vtd_ggtt_wa(arg->vm);
+
+ return 0;
+}
+
+static void bxt_vtd_ggtt_insert_page__BKL(struct i915_address_space *vm,
+ dma_addr_t addr,
+ u64 offset,
+ enum i915_cache_level level,
+ u32 unused)
+{
+ struct insert_page arg = { vm, addr, offset, level };
+
+ stop_machine(bxt_vtd_ggtt_insert_page__cb, &arg, NULL);
+}
+
+struct insert_entries {
+ struct i915_address_space *vm;
+ struct sg_table *st;
+ u64 start;
+ enum i915_cache_level level;
+};
+
+static int bxt_vtd_ggtt_insert_entries__cb(void *_arg)
+{
+ struct insert_entries *arg = _arg;
+
+ gen8_ggtt_insert_entries(arg->vm, arg->st, arg->start, arg->level, 0);
+ bxt_vtd_ggtt_wa(arg->vm);
+
+ return 0;
+}
+
+static void bxt_vtd_ggtt_insert_entries__BKL(struct i915_address_space *vm,
+ struct sg_table *st,
+ u64 start,
+ enum i915_cache_level level,
+ u32 unused)
+{
+ struct insert_entries arg = { vm, st, start, level };
+
+ stop_machine(bxt_vtd_ggtt_insert_entries__cb, &arg, NULL);
+}
+
+struct clear_range {
+ struct i915_address_space *vm;
+ u64 start;
+ u64 length;
+};
+
+static int bxt_vtd_ggtt_clear_range__cb(void *_arg)
+{
+ struct clear_range *arg = _arg;
+
+ gen8_ggtt_clear_range(arg->vm, arg->start, arg->length);
+ bxt_vtd_ggtt_wa(arg->vm);
+
+ return 0;
+}
+
+static void bxt_vtd_ggtt_clear_range__BKL(struct i915_address_space *vm,
+ u64 start,
+ u64 length)
+{
+ struct clear_range arg = { vm, start, length };
+
+ stop_machine(bxt_vtd_ggtt_clear_range__cb, &arg, NULL);
+}
+
static void gen6_ggtt_clear_range(struct i915_address_space *vm,
u64 start, u64 length)
{
if (flags & I915_VMA_LOCAL_BIND) {
struct i915_hw_ppgtt *appgtt = i915->mm.aliasing_ppgtt;
- if (appgtt->base.allocate_va_range) {
+ if (!(vma->flags & I915_VMA_LOCAL_BIND) &&
+ appgtt->base.allocate_va_range) {
ret = appgtt->base.allocate_va_range(&appgtt->base,
vma->node.start,
- vma->node.size);
+ vma->size);
if (ret)
goto err_pages;
}
{
snb_gmch_ctl >>= SNB_GMCH_GMS_SHIFT;
snb_gmch_ctl &= SNB_GMCH_GMS_MASK;
- return snb_gmch_ctl << 25; /* 32 MB units */
+ return (size_t)snb_gmch_ctl << 25; /* 32 MB units */
}
static size_t gen8_get_stolen_size(u16 bdw_gmch_ctl)
{
bdw_gmch_ctl >>= BDW_GMCH_GMS_SHIFT;
bdw_gmch_ctl &= BDW_GMCH_GMS_MASK;
- return bdw_gmch_ctl << 25; /* 32 MB units */
+ return (size_t)bdw_gmch_ctl << 25; /* 32 MB units */
}
static size_t chv_get_stolen_size(u16 gmch_ctrl)
* 0x17 to 0x1d: 4MB increments start at 36MB
*/
if (gmch_ctrl < 0x11)
- return gmch_ctrl << 25;
+ return (size_t)gmch_ctrl << 25;
else if (gmch_ctrl < 0x17)
- return (gmch_ctrl - 0x11 + 2) << 22;
+ return (size_t)(gmch_ctrl - 0x11 + 2) << 22;
else
- return (gmch_ctrl - 0x17 + 9) << 22;
+ return (size_t)(gmch_ctrl - 0x17 + 9) << 22;
}
static size_t gen9_get_stolen_size(u16 gen9_gmch_ctl)
gen9_gmch_ctl &= BDW_GMCH_GMS_MASK;
if (gen9_gmch_ctl < 0xf0)
- return gen9_gmch_ctl << 25; /* 32 MB units */
+ return (size_t)gen9_gmch_ctl << 25; /* 32 MB units */
else
/* 4MB increments starting at 0xf0 for 4MB */
- return (gen9_gmch_ctl - 0xf0 + 1) << 22;
+ return (size_t)(gen9_gmch_ctl - 0xf0 + 1) << 22;
}
static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size)
struct pci_dev *pdev = dev_priv->drm.pdev;
unsigned int size;
u16 snb_gmch_ctl;
+ int err;
/* TODO: We're not aware of mappable constraints on gen8 yet */
ggtt->mappable_base = pci_resource_start(pdev, 2);
ggtt->mappable_end = pci_resource_len(pdev, 2);
- if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(39)))
- pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(39));
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(39));
+ if (!err)
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(39));
+ if (err)
+ DRM_ERROR("Can't set DMA mask/consistent mask (%d)\n", err);
pci_read_config_word(pdev, SNB_GMCH_CTRL, &snb_gmch_ctl);
ggtt->base.insert_entries = gen8_ggtt_insert_entries;
+ /* Serialize GTT updates with aperture access on BXT if VT-d is on. */
+ if (intel_ggtt_update_needs_vtd_wa(dev_priv)) {
+ ggtt->base.insert_entries = bxt_vtd_ggtt_insert_entries__BKL;
+ ggtt->base.insert_page = bxt_vtd_ggtt_insert_page__BKL;
+ if (ggtt->base.clear_range != nop_clear_range)
+ ggtt->base.clear_range = bxt_vtd_ggtt_clear_range__BKL;
+ }
+
ggtt->invalidate = gen6_ggtt_invalidate;
return ggtt_probe_common(ggtt, size);
struct pci_dev *pdev = dev_priv->drm.pdev;
unsigned int size;
u16 snb_gmch_ctl;
+ int err;
ggtt->mappable_base = pci_resource_start(pdev, 2);
ggtt->mappable_end = pci_resource_len(pdev, 2);
return -ENXIO;
}
- if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(40)))
- pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(40));
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(40));
+ if (!err)
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(40));
+ if (err)
+ DRM_ERROR("Can't set DMA mask/consistent mask (%d)\n", err);
pci_read_config_word(pdev, SNB_GMCH_CTRL, &snb_gmch_ctl);
ggtt->stolen_size = gen6_get_stolen_size(snb_gmch_ctl);
ggtt->base.total >> 20);
DRM_DEBUG_DRIVER("GMADR size = %lldM\n", ggtt->mappable_end >> 20);
DRM_DEBUG_DRIVER("GTT stolen size = %uM\n", ggtt->stolen_size >> 20);
-#ifdef CONFIG_INTEL_IOMMU
- if (intel_iommu_gfx_mapped)
+ if (intel_vtd_active())
DRM_INFO("VT-d active for gfx access\n");
-#endif
return 0;
}
struct drm_i915_gem_object_ops {
unsigned int flags;
-#define I915_GEM_OBJECT_HAS_STRUCT_PAGE 0x1
-#define I915_GEM_OBJECT_IS_SHRINKABLE 0x2
+#define I915_GEM_OBJECT_HAS_STRUCT_PAGE BIT(0)
+#define I915_GEM_OBJECT_IS_SHRINKABLE BIT(1)
/* Interface between the GEM object and its backing storage.
* get_pages() is called once prior to the use of the associated set
if (i915_fence_signaled(fence))
return false;
- intel_engine_enable_signaling(to_request(fence));
+ intel_engine_enable_signaling(to_request(fence), true);
return true;
}
{
struct i915_dependency *dep, *next;
- GEM_BUG_ON(!RB_EMPTY_NODE(&pt->node));
+ GEM_BUG_ON(!list_empty(&pt->link));
/* Everyone we depended upon (the fences we wait to be signaled)
* should retire before us and remove themselves from our list.
{
INIT_LIST_HEAD(&pt->signalers_list);
INIT_LIST_HEAD(&pt->waiters_list);
- RB_CLEAR_NODE(&pt->node);
+ INIT_LIST_HEAD(&pt->link);
pt->priority = INT_MIN;
}
}
/* Finally reset hw state */
- tl->seqno = seqno;
intel_engine_init_global_seqno(engine, seqno);
+ tl->seqno = seqno;
list_for_each_entry(timeline, &i915->gt.timelines, link)
- memset(timeline->engine[id].sync_seqno, 0,
- sizeof(timeline->engine[id].sync_seqno));
+ memset(timeline->engine[id].global_sync, 0,
+ sizeof(timeline->engine[id].global_sync));
}
return 0;
/* Space left intentionally blank */
}
+static void advance_ring(struct drm_i915_gem_request *request)
+{
+ unsigned int tail;
+
+ /* We know the GPU must have read the request to have
+ * sent us the seqno + interrupt, so use the position
+ * of tail of the request to update the last known position
+ * of the GPU head.
+ *
+ * Note this requires that we are always called in request
+ * completion order.
+ */
+ if (list_is_last(&request->ring_link, &request->ring->request_list)) {
+ /* We may race here with execlists resubmitting this request
+ * as we retire it. The resubmission will move the ring->tail
+ * forwards (to request->wa_tail). We either read the
+ * current value that was written to hw, or the value that
+ * is just about to be. Either works, if we miss the last two
+ * noops - they are safe to be replayed on a reset.
+ */
+ tail = READ_ONCE(request->ring->tail);
+ } else {
+ tail = request->postfix;
+ }
+ list_del(&request->ring_link);
+
+ request->ring->head = tail;
+}
+
+static void free_capture_list(struct drm_i915_gem_request *request)
+{
+ struct i915_gem_capture_list *capture;
+
+ capture = request->capture_list;
+ while (capture) {
+ struct i915_gem_capture_list *next = capture->next;
+
+ kfree(capture);
+ capture = next;
+ }
+}
+
static void i915_gem_request_retire(struct drm_i915_gem_request *request)
{
struct intel_engine_cs *engine = request->engine;
list_del_init(&request->link);
spin_unlock_irq(&engine->timeline->lock);
- /* We know the GPU must have read the request to have
- * sent us the seqno + interrupt, so use the position
- * of tail of the request to update the last known position
- * of the GPU head.
- *
- * Note this requires that we are always called in request
- * completion order.
- */
- list_del(&request->ring_link);
- request->ring->head = request->postfix;
if (!--request->i915->gt.active_requests) {
GEM_BUG_ON(!request->i915->gt.awake);
mod_delayed_work(request->i915->wq,
msecs_to_jiffies(100));
}
unreserve_seqno(request->engine);
+ advance_ring(request);
+
+ free_capture_list(request);
/* Walk through the active list, calling retire on each. This allows
* objects to track their GPU activity and mark themselves as idle
spin_lock_nested(&request->lock, SINGLE_DEPTH_NESTING);
request->global_seqno = seqno;
if (test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &request->fence.flags))
- intel_engine_enable_signaling(request);
+ intel_engine_enable_signaling(request, false);
spin_unlock(&request->lock);
engine->emit_breadcrumb(request,
*
* @engine: engine that we wish to issue the request on.
* @ctx: context that the request will be associated with.
- * This can be NULL if the request is not directly related to
- * any specific user context, in which case this function will
- * choose an appropriate context to use.
*
* Returns a pointer to the allocated request if successful,
* or an error code if not.
{
struct drm_i915_private *dev_priv = engine->i915;
struct drm_i915_gem_request *req;
+ struct intel_ring *ring;
int ret;
lockdep_assert_held(&dev_priv->drm.struct_mutex);
* GGTT space, so do this first before we reserve a seqno for
* ourselves.
*/
- ret = engine->context_pin(engine, ctx);
- if (ret)
- return ERR_PTR(ret);
+ ring = engine->context_pin(engine, ctx);
+ if (IS_ERR(ring))
+ return ERR_CAST(ring);
+ GEM_BUG_ON(!ring);
ret = reserve_seqno(engine);
if (ret)
req->i915 = dev_priv;
req->engine = engine;
req->ctx = ctx;
+ req->ring = ring;
/* No zalloc, must clear what we need by hand */
req->global_seqno = 0;
req->file_priv = NULL;
req->batch = NULL;
+ req->capture_list = NULL;
/*
* Reserve space in the ring buffer for all the commands required to
* GPU processing the request, we never over-estimate the
* position of the head.
*/
- req->head = req->ring->tail;
+ req->head = req->ring->emit;
/* Check that we didn't interrupt ourselves with a new request */
GEM_BUG_ON(req->timeline->seqno != req->fence.seqno);
int ret;
GEM_BUG_ON(to == from);
+ GEM_BUG_ON(to->timeline == from->timeline);
if (i915_gem_request_completed(from))
return 0;
return ret;
}
- if (to->timeline == from->timeline)
- return 0;
-
if (to->engine == from->engine) {
ret = i915_sw_fence_await_sw_fence_gfp(&to->submit,
&from->submit,
}
seqno = i915_gem_request_global_seqno(from);
- if (!seqno) {
- ret = i915_sw_fence_await_dma_fence(&to->submit,
- &from->fence, 0,
- GFP_KERNEL);
- return ret < 0 ? ret : 0;
- }
+ if (!seqno)
+ goto await_dma_fence;
- if (seqno <= to->timeline->sync_seqno[from->engine->id])
- return 0;
+ if (!to->engine->semaphore.sync_to) {
+ if (!__i915_gem_request_started(from, seqno))
+ goto await_dma_fence;
- trace_i915_gem_ring_sync_to(to, from);
- if (!i915.semaphores) {
- if (!i915_spin_request(from, TASK_INTERRUPTIBLE, 2)) {
- ret = i915_sw_fence_await_dma_fence(&to->submit,
- &from->fence, 0,
- GFP_KERNEL);
- if (ret < 0)
- return ret;
- }
+ if (!__i915_spin_request(from, seqno, TASK_INTERRUPTIBLE, 2))
+ goto await_dma_fence;
} else {
+ GEM_BUG_ON(!from->engine->semaphore.signal);
+
+ if (seqno <= to->timeline->global_sync[from->engine->id])
+ return 0;
+
+ trace_i915_gem_ring_sync_to(to, from);
ret = to->engine->semaphore.sync_to(to, from);
if (ret)
return ret;
+
+ to->timeline->global_sync[from->engine->id] = seqno;
}
- to->timeline->sync_seqno[from->engine->id] = seqno;
return 0;
+
+await_dma_fence:
+ ret = i915_sw_fence_await_dma_fence(&to->submit,
+ &from->fence, 0,
+ GFP_KERNEL);
+ return ret < 0 ? ret : 0;
}
int
i915_gem_request_await_dma_fence(struct drm_i915_gem_request *req,
struct dma_fence *fence)
{
- struct dma_fence_array *array;
+ struct dma_fence **child = &fence;
+ unsigned int nchild = 1;
int ret;
- int i;
-
- if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
- return 0;
-
- if (dma_fence_is_i915(fence))
- return i915_gem_request_await_request(req, to_request(fence));
-
- if (!dma_fence_is_array(fence)) {
- ret = i915_sw_fence_await_dma_fence(&req->submit,
- fence, I915_FENCE_TIMEOUT,
- GFP_KERNEL);
- return ret < 0 ? ret : 0;
- }
/* Note that if the fence-array was created in signal-on-any mode,
* we should *not* decompose it into its individual fences. However,
* amdgpu and we should not see any incoming fence-array from
* sync-file being in signal-on-any mode.
*/
+ if (dma_fence_is_array(fence)) {
+ struct dma_fence_array *array = to_dma_fence_array(fence);
+
+ child = array->fences;
+ nchild = array->num_fences;
+ GEM_BUG_ON(!nchild);
+ }
+
+ do {
+ fence = *child++;
+ if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
+ continue;
+
+ /*
+ * Requests on the same timeline are explicitly ordered, along
+ * with their dependencies, by i915_add_request() which ensures
+ * that requests are submitted in-order through each ring.
+ */
+ if (fence->context == req->fence.context)
+ continue;
- array = to_dma_fence_array(fence);
- for (i = 0; i < array->num_fences; i++) {
- struct dma_fence *child = array->fences[i];
+ /* Squash repeated waits to the same timelines */
+ if (fence->context != req->i915->mm.unordered_timeline &&
+ intel_timeline_sync_is_later(req->timeline, fence))
+ continue;
- if (dma_fence_is_i915(child))
+ if (dma_fence_is_i915(fence))
ret = i915_gem_request_await_request(req,
- to_request(child));
+ to_request(fence));
else
- ret = i915_sw_fence_await_dma_fence(&req->submit,
- child, I915_FENCE_TIMEOUT,
+ ret = i915_sw_fence_await_dma_fence(&req->submit, fence,
+ I915_FENCE_TIMEOUT,
GFP_KERNEL);
if (ret < 0)
return ret;
- }
+
+ /* Record the latest fence used against each timeline */
+ if (fence->context != req->i915->mm.unordered_timeline)
+ intel_timeline_sync_set(req->timeline, fence);
+ } while (--nchild);
return 0;
}
struct i915_priotree {
struct list_head signalers_list; /* those before us, we depend upon */
struct list_head waiters_list; /* those after us, they depend upon us */
- struct rb_node node;
+ struct list_head link;
int priority;
#define I915_PRIORITY_MAX 1024
+#define I915_PRIORITY_NORMAL 0
#define I915_PRIORITY_MIN (-I915_PRIORITY_MAX)
};
+struct i915_gem_capture_list {
+ struct i915_gem_capture_list *next;
+ struct i915_vma *vma;
+};
+
/**
* Request queue structure.
*
* error state dump only).
*/
struct i915_vma *batch;
+ /** Additional buffers requested by userspace to be captured upon
+ * a GPU hang. The vma/obj on this list are protected by their
+ * active reference - all objects on this list must also be
+ * on the active_list (of their final request).
+ */
+ struct i915_gem_capture_list *capture_list;
struct list_head active_list;
/** Time at which this request was emitted, in jiffies. */
#include "i915_drv.h"
#include "i915_trace.h"
-static bool i915_gem_shrinker_lock(struct drm_device *dev, bool *unlock)
+static bool shrinker_lock(struct drm_i915_private *dev_priv, bool *unlock)
{
- switch (mutex_trylock_recursive(&dev->struct_mutex)) {
+ switch (mutex_trylock_recursive(&dev_priv->drm.struct_mutex)) {
case MUTEX_TRYLOCK_FAILED:
return false;
BUG();
}
-static void i915_gem_shrinker_unlock(struct drm_device *dev, bool unlock)
+static void shrinker_unlock(struct drm_i915_private *dev_priv, bool unlock)
{
if (!unlock)
return;
- mutex_unlock(&dev->struct_mutex);
-
- /* expedite the RCU grace period to free some request slabs */
- synchronize_rcu_expedited();
+ mutex_unlock(&dev_priv->drm.struct_mutex);
}
static bool any_vma_pinned(struct drm_i915_gem_object *obj)
{
struct i915_vma *vma;
- list_for_each_entry(vma, &obj->vma_list, obj_link)
+ list_for_each_entry(vma, &obj->vma_list, obj_link) {
+ /* Only GGTT vma may be permanently pinned, and are always
+ * at the start of the list. We can stop hunting as soon
+ * as we see a ppGTT vma.
+ */
+ if (!i915_vma_is_ggtt(vma))
+ break;
+
if (i915_vma_is_pinned(vma))
return true;
+ }
return false;
}
unsigned long count = 0;
bool unlock;
- if (!i915_gem_shrinker_lock(&dev_priv->drm, &unlock))
+ if (!shrinker_lock(dev_priv, &unlock))
return 0;
trace_i915_gem_shrink(dev_priv, target, flags);
i915_gem_retire_requests(dev_priv);
- i915_gem_shrinker_unlock(&dev_priv->drm, unlock);
+ shrinker_unlock(dev_priv, unlock);
return count;
}
I915_SHRINK_ACTIVE);
intel_runtime_pm_put(dev_priv);
- synchronize_rcu(); /* wait for our earlier RCU delayed slab frees */
-
return freed;
}
{
struct drm_i915_private *dev_priv =
container_of(shrinker, struct drm_i915_private, mm.shrinker);
- struct drm_device *dev = &dev_priv->drm;
struct drm_i915_gem_object *obj;
unsigned long count;
bool unlock;
- if (!i915_gem_shrinker_lock(dev, &unlock))
+ if (!shrinker_lock(dev_priv, &unlock))
return 0;
i915_gem_retire_requests(dev_priv);
count += obj->base.size >> PAGE_SHIFT;
}
- i915_gem_shrinker_unlock(dev, unlock);
+ shrinker_unlock(dev_priv, unlock);
return count;
}
{
struct drm_i915_private *dev_priv =
container_of(shrinker, struct drm_i915_private, mm.shrinker);
- struct drm_device *dev = &dev_priv->drm;
unsigned long freed;
bool unlock;
- if (!i915_gem_shrinker_lock(dev, &unlock))
+ if (!shrinker_lock(dev_priv, &unlock))
return SHRINK_STOP;
freed = i915_gem_shrink(dev_priv,
I915_SHRINK_BOUND |
I915_SHRINK_UNBOUND);
- i915_gem_shrinker_unlock(dev, unlock);
+ shrinker_unlock(dev_priv, unlock);
return freed;
}
-struct shrinker_lock_uninterruptible {
- bool was_interruptible;
- bool unlock;
-};
-
static bool
-i915_gem_shrinker_lock_uninterruptible(struct drm_i915_private *dev_priv,
- struct shrinker_lock_uninterruptible *slu,
- int timeout_ms)
+shrinker_lock_uninterruptible(struct drm_i915_private *dev_priv, bool *unlock,
+ int timeout_ms)
{
unsigned long timeout = jiffies + msecs_to_jiffies_timeout(timeout_ms);
do {
if (i915_gem_wait_for_idle(dev_priv, 0) == 0 &&
- i915_gem_shrinker_lock(&dev_priv->drm, &slu->unlock))
+ shrinker_lock(dev_priv, unlock))
break;
schedule_timeout_killable(1);
}
} while (1);
- slu->was_interruptible = dev_priv->mm.interruptible;
- dev_priv->mm.interruptible = false;
return true;
}
-static void
-i915_gem_shrinker_unlock_uninterruptible(struct drm_i915_private *dev_priv,
- struct shrinker_lock_uninterruptible *slu)
-{
- dev_priv->mm.interruptible = slu->was_interruptible;
- i915_gem_shrinker_unlock(&dev_priv->drm, slu->unlock);
-}
-
static int
i915_gem_shrinker_oom(struct notifier_block *nb, unsigned long event, void *ptr)
{
struct drm_i915_private *dev_priv =
container_of(nb, struct drm_i915_private, mm.oom_notifier);
- struct shrinker_lock_uninterruptible slu;
struct drm_i915_gem_object *obj;
unsigned long unevictable, bound, unbound, freed_pages;
+ bool unlock;
- if (!i915_gem_shrinker_lock_uninterruptible(dev_priv, &slu, 5000))
+ if (!shrinker_lock_uninterruptible(dev_priv, &unlock, 5000))
return NOTIFY_DONE;
freed_pages = i915_gem_shrink_all(dev_priv);
bound += obj->base.size >> PAGE_SHIFT;
}
- i915_gem_shrinker_unlock_uninterruptible(dev_priv, &slu);
+ shrinker_unlock(dev_priv, unlock);
if (freed_pages || unbound || bound)
pr_info("Purging GPU memory, %lu pages freed, "
{
struct drm_i915_private *dev_priv =
container_of(nb, struct drm_i915_private, mm.vmap_notifier);
- struct shrinker_lock_uninterruptible slu;
struct i915_vma *vma, *next;
unsigned long freed_pages = 0;
+ bool unlock;
int ret;
- if (!i915_gem_shrinker_lock_uninterruptible(dev_priv, &slu, 5000))
+ if (!shrinker_lock_uninterruptible(dev_priv, &unlock, 5000))
return NOTIFY_DONE;
/* Force everything onto the inactive lists */
}
out:
- i915_gem_shrinker_unlock_uninterruptible(dev_priv, &slu);
+ shrinker_unlock(dev_priv, unlock);
*(unsigned long *)ptr += freed_pages;
return NOTIFY_DONE;
return 0;
}
-#ifdef CONFIG_INTEL_IOMMU
- if (intel_iommu_gfx_mapped && INTEL_GEN(dev_priv) < 8) {
+ if (intel_vtd_active() && INTEL_GEN(dev_priv) < 8) {
DRM_INFO("DMAR active, disabling use of stolen memory\n");
return 0;
}
-#endif
if (ggtt->stolen_size == 0)
return 0;
*/
#include "i915_drv.h"
+#include "i915_syncmap.h"
+
+static void __intel_timeline_init(struct intel_timeline *tl,
+ struct i915_gem_timeline *parent,
+ u64 context,
+ struct lock_class_key *lockclass,
+ const char *lockname)
+{
+ tl->fence_context = context;
+ tl->common = parent;
+#ifdef CONFIG_DEBUG_SPINLOCK
+ __raw_spin_lock_init(&tl->lock.rlock, lockname, lockclass);
+#else
+ spin_lock_init(&tl->lock);
+#endif
+ init_request_active(&tl->last_request, NULL);
+ INIT_LIST_HEAD(&tl->requests);
+ i915_syncmap_init(&tl->sync);
+}
+
+static void __intel_timeline_fini(struct intel_timeline *tl)
+{
+ GEM_BUG_ON(!list_empty(&tl->requests));
+
+ i915_syncmap_free(&tl->sync);
+}
static int __i915_gem_timeline_init(struct drm_i915_private *i915,
struct i915_gem_timeline *timeline,
lockdep_assert_held(&i915->drm.struct_mutex);
+ /*
+ * Ideally we want a set of engines on a single leaf as we expect
+ * to mostly be tracking synchronisation between engines. It is not
+ * a huge issue if this is not the case, but we may want to mitigate
+ * any page crossing penalties if they become an issue.
+ */
+ BUILD_BUG_ON(KSYNCMAP < I915_NUM_ENGINES);
+
timeline->i915 = i915;
timeline->name = kstrdup(name ?: "[kernel]", GFP_KERNEL);
if (!timeline->name)
/* Called during early_init before we know how many engines there are */
fences = dma_fence_context_alloc(ARRAY_SIZE(timeline->engine));
- for (i = 0; i < ARRAY_SIZE(timeline->engine); i++) {
- struct intel_timeline *tl = &timeline->engine[i];
-
- tl->fence_context = fences++;
- tl->common = timeline;
-#ifdef CONFIG_DEBUG_SPINLOCK
- __raw_spin_lock_init(&tl->lock.rlock, lockname, lockclass);
-#else
- spin_lock_init(&tl->lock);
-#endif
- init_request_active(&tl->last_request, NULL);
- INIT_LIST_HEAD(&tl->requests);
- }
+ for (i = 0; i < ARRAY_SIZE(timeline->engine); i++)
+ __intel_timeline_init(&timeline->engine[i],
+ timeline, fences++,
+ lockclass, lockname);
return 0;
}
&class, "&global_timeline->lock");
}
+/**
+ * i915_gem_timelines_mark_idle -- called when the driver idles
+ * @i915 - the drm_i915_private device
+ *
+ * When the driver is completely idle, we know that all of our sync points
+ * have been signaled and our tracking is then entirely redundant. Any request
+ * to wait upon an older sync point will be completed instantly as we know
+ * the fence is signaled and therefore we will not even look them up in the
+ * sync point map.
+ */
+void i915_gem_timelines_mark_idle(struct drm_i915_private *i915)
+{
+ struct i915_gem_timeline *timeline;
+ int i;
+
+ lockdep_assert_held(&i915->drm.struct_mutex);
+
+ list_for_each_entry(timeline, &i915->gt.timelines, link) {
+ for (i = 0; i < ARRAY_SIZE(timeline->engine); i++) {
+ struct intel_timeline *tl = &timeline->engine[i];
+
+ /*
+ * All known fences are completed so we can scrap
+ * the current sync point tracking and start afresh,
+ * any attempt to wait upon a previous sync point
+ * will be skipped as the fence was signaled.
+ */
+ i915_syncmap_free(&tl->sync);
+ }
+ }
+}
+
void i915_gem_timeline_fini(struct i915_gem_timeline *timeline)
{
int i;
lockdep_assert_held(&timeline->i915->drm.struct_mutex);
- for (i = 0; i < ARRAY_SIZE(timeline->engine); i++) {
- struct intel_timeline *tl = &timeline->engine[i];
-
- GEM_BUG_ON(!list_empty(&tl->requests));
- }
+ for (i = 0; i < ARRAY_SIZE(timeline->engine); i++)
+ __intel_timeline_fini(&timeline->engine[i]);
list_del(&timeline->link);
kfree(timeline->name);
}
+
+#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
+#include "selftests/mock_timeline.c"
+#include "selftests/i915_gem_timeline.c"
+#endif
#include <linux/list.h>
+#include "i915_utils.h"
#include "i915_gem_request.h"
+#include "i915_syncmap.h"
struct i915_gem_timeline;
* struct_mutex.
*/
struct i915_gem_active last_request;
- u32 sync_seqno[I915_NUM_ENGINES];
+
+ /**
+ * We track the most recent seqno that we wait on in every context so
+ * that we only have to emit a new await and dependency on a more
+ * recent sync point. As the contexts may be executed out-of-order, we
+ * have to track each individually and can not rely on an absolute
+ * global_seqno. When we know that all tracked fences are completed
+ * (i.e. when the driver is idle), we know that the syncmap is
+ * redundant and we can discard it without loss of generality.
+ */
+ struct i915_syncmap *sync;
+ /**
+ * Separately to the inter-context seqno map above, we track the last
+ * barrier (e.g. semaphore wait) to the global engine timelines. Note
+ * that this tracks global_seqno rather than the context.seqno, and
+ * so it is subject to the limitations of hw wraparound and that we
+ * may need to revoke global_seqno (on pre-emption).
+ */
+ u32 global_sync[I915_NUM_ENGINES];
struct i915_gem_timeline *common;
};
struct i915_gem_timeline *tl,
const char *name);
int i915_gem_timeline_init__global(struct drm_i915_private *i915);
+void i915_gem_timelines_mark_idle(struct drm_i915_private *i915);
void i915_gem_timeline_fini(struct i915_gem_timeline *tl);
+static inline int __intel_timeline_sync_set(struct intel_timeline *tl,
+ u64 context, u32 seqno)
+{
+ return i915_syncmap_set(&tl->sync, context, seqno);
+}
+
+static inline int intel_timeline_sync_set(struct intel_timeline *tl,
+ const struct dma_fence *fence)
+{
+ return __intel_timeline_sync_set(tl, fence->context, fence->seqno);
+}
+
+static inline bool __intel_timeline_sync_is_later(struct intel_timeline *tl,
+ u64 context, u32 seqno)
+{
+ return i915_syncmap_is_later(&tl->sync, context, seqno);
+}
+
+static inline bool intel_timeline_sync_is_later(struct intel_timeline *tl,
+ const struct dma_fence *fence)
+{
+ return __intel_timeline_sync_is_later(tl, fence->context, fence->seqno);
+}
+
#endif
print_error_obj(m, dev_priv->engine[i], NULL, obj);
}
+ for (j = 0; j < ee->user_bo_count; j++)
+ print_error_obj(m, dev_priv->engine[i],
+ "user", ee->user_bo[j]);
+
if (ee->num_requests) {
err_printf(m, "%s --- %d requests\n",
dev_priv->engine[i]->name,
{
struct i915_gpu_state *error =
container_of(error_ref, typeof(*error), ref);
- int i;
+ long i, j;
for (i = 0; i < ARRAY_SIZE(error->engine); i++) {
struct drm_i915_error_engine *ee = &error->engine[i];
+ for (j = 0; j < ee->user_bo_count; j++)
+ i915_error_object_free(ee->user_bo[j]);
+ kfree(ee->user_bo);
+
i915_error_object_free(ee->batchbuffer);
i915_error_object_free(ee->wa_batchbuffer);
i915_error_object_free(ee->ringbuffer);
static void error_record_engine_execlists(struct intel_engine_cs *engine,
struct drm_i915_error_engine *ee)
{
+ const struct execlist_port *port = engine->execlist_port;
unsigned int n;
- for (n = 0; n < ARRAY_SIZE(engine->execlist_port); n++)
- if (engine->execlist_port[n].request)
- record_request(engine->execlist_port[n].request,
- &ee->execlist[n]);
+ for (n = 0; n < ARRAY_SIZE(engine->execlist_port); n++) {
+ struct drm_i915_gem_request *rq = port_request(&port[n]);
+
+ if (!rq)
+ break;
+
+ record_request(rq, &ee->execlist[n]);
+ }
}
static void record_context(struct drm_i915_error_context *e,
e->active = ctx->active_count;
}
+static void request_record_user_bo(struct drm_i915_gem_request *request,
+ struct drm_i915_error_engine *ee)
+{
+ struct i915_gem_capture_list *c;
+ struct drm_i915_error_object **bo;
+ long count;
+
+ count = 0;
+ for (c = request->capture_list; c; c = c->next)
+ count++;
+
+ bo = NULL;
+ if (count)
+ bo = kcalloc(count, sizeof(*bo), GFP_ATOMIC);
+ if (!bo)
+ return;
+
+ count = 0;
+ for (c = request->capture_list; c; c = c->next) {
+ bo[count] = i915_error_object_create(request->i915, c->vma);
+ if (!bo[count])
+ break;
+ count++;
+ }
+
+ ee->user_bo = bo;
+ ee->user_bo_count = count;
+}
+
static void i915_gem_record_rings(struct drm_i915_private *dev_priv,
struct i915_gpu_state *error)
{
ee->wa_batchbuffer =
i915_error_object_create(dev_priv,
engine->scratch);
+ request_record_user_bo(request, ee);
ee->ctx =
i915_error_object_create(dev_priv,
error->done_reg = I915_READ(DONE_REG);
}
+ if (INTEL_GEN(dev_priv) >= 5)
+ error->ccid = I915_READ(CCID);
+
/* 3: Feature specific registers */
if (IS_GEN6(dev_priv) || IS_GEN7(dev_priv)) {
error->gam_ecochk = I915_READ(GAM_ECOCHK);
}
/* 4: Everything else */
- if (HAS_HW_CONTEXTS(dev_priv))
- error->ccid = I915_READ(CCID);
-
if (INTEL_GEN(dev_priv) >= 8) {
error->ier = I915_READ(GEN8_DE_MISC_IER);
for (i = 0; i < 4; i++)
GEM_BUG_ON(freespace < wqi_size);
/* The GuC firmware wants the tail index in QWords, not bytes */
- tail = rq->tail;
- assert_ring_tail_valid(rq->ring, rq->tail);
- tail >>= 3;
+ tail = intel_ring_set_tail(rq->ring, rq->tail) >> 3;
GEM_BUG_ON(tail > WQ_RING_TAIL_MAX);
/* For now workqueue item is 4 DWs; workqueue buffer is 2 pages. So we
b_ret = guc_ring_doorbell(client);
client->submissions[engine_id] += 1;
- client->retcode = b_ret;
- if (b_ret)
- client->b_fail += 1;
-
- guc->submissions[engine_id] += 1;
- guc->last_seqno[engine_id] = rq->global_seqno;
spin_unlock_irqrestore(&client->wq_lock, flags);
}
trace_dma_fence_enable_signal(&rq->fence);
spin_lock_nested(&rq->lock, SINGLE_DEPTH_NESTING);
- intel_engine_enable_signaling(rq);
+ intel_engine_enable_signaling(rq, true);
spin_unlock(&rq->lock);
}
+static void port_assign(struct execlist_port *port,
+ struct drm_i915_gem_request *rq)
+{
+ GEM_BUG_ON(rq == port_request(port));
+
+ if (port_isset(port))
+ i915_gem_request_put(port_request(port));
+
+ port_set(port, i915_gem_request_get(rq));
+ nested_enable_signaling(rq);
+}
+
static bool i915_guc_dequeue(struct intel_engine_cs *engine)
{
struct execlist_port *port = engine->execlist_port;
- struct drm_i915_gem_request *last = port[0].request;
+ struct drm_i915_gem_request *last = port_request(port);
struct rb_node *rb;
bool submit = false;
spin_lock_irq(&engine->timeline->lock);
rb = engine->execlist_first;
+ GEM_BUG_ON(rb_first(&engine->execlist_queue) != rb);
while (rb) {
- struct drm_i915_gem_request *rq =
- rb_entry(rb, typeof(*rq), priotree.node);
-
- if (last && rq->ctx != last->ctx) {
- if (port != engine->execlist_port)
- break;
-
- i915_gem_request_assign(&port->request, last);
- nested_enable_signaling(last);
- port++;
+ struct i915_priolist *p = rb_entry(rb, typeof(*p), node);
+ struct drm_i915_gem_request *rq, *rn;
+
+ list_for_each_entry_safe(rq, rn, &p->requests, priotree.link) {
+ if (last && rq->ctx != last->ctx) {
+ if (port != engine->execlist_port) {
+ __list_del_many(&p->requests,
+ &rq->priotree.link);
+ goto done;
+ }
+
+ if (submit)
+ port_assign(port, last);
+ port++;
+ }
+
+ INIT_LIST_HEAD(&rq->priotree.link);
+ rq->priotree.priority = INT_MAX;
+
+ i915_guc_submit(rq);
+ trace_i915_gem_request_in(rq, port_index(port, engine));
+ last = rq;
+ submit = true;
}
rb = rb_next(rb);
- rb_erase(&rq->priotree.node, &engine->execlist_queue);
- RB_CLEAR_NODE(&rq->priotree.node);
- rq->priotree.priority = INT_MAX;
-
- i915_guc_submit(rq);
- trace_i915_gem_request_in(rq, port - engine->execlist_port);
- last = rq;
- submit = true;
- }
- if (submit) {
- i915_gem_request_assign(&port->request, last);
- nested_enable_signaling(last);
- engine->execlist_first = rb;
+ rb_erase(&p->node, &engine->execlist_queue);
+ INIT_LIST_HEAD(&p->requests);
+ if (p->priority != I915_PRIORITY_NORMAL)
+ kmem_cache_free(engine->i915->priorities, p);
}
+done:
+ engine->execlist_first = rb;
+ if (submit)
+ port_assign(port, last);
spin_unlock_irq(&engine->timeline->lock);
return submit;
bool submit;
do {
- rq = port[0].request;
+ rq = port_request(&port[0]);
while (rq && i915_gem_request_completed(rq)) {
trace_i915_gem_request_out(rq);
i915_gem_request_put(rq);
- port[0].request = port[1].request;
- port[1].request = NULL;
- rq = port[0].request;
+
+ port[0] = port[1];
+ memset(&port[1], 0, sizeof(port[1]));
+
+ rq = port_request(&port[0]);
}
submit = false;
- if (!port[1].request)
+ if (!port_count(&port[1]))
submit = i915_guc_dequeue(engine);
} while (submit);
}
dev_priv->engine[RCS]->status_page.ggtt_offset;
for_each_engine(engine, dev_priv, id)
- blob->ads.eng_state_size[engine->guc_id] =
- intel_lr_context_size(engine);
+ blob->ads.eng_state_size[engine->guc_id] = engine->context_size;
base = guc_ggtt_offset(vma);
blob->ads.scheduler_policies = base + ptr_offset(blob, policies);
static void ivybridge_parity_work(struct work_struct *work)
{
struct drm_i915_private *dev_priv =
- container_of(work, struct drm_i915_private, l3_parity.error_work);
+ container_of(work, typeof(*dev_priv), l3_parity.error_work);
u32 error_status, row, bank, subbank;
char *parity_event[6];
uint32_t misccpctl;
ivybridge_parity_error_irq_handler(dev_priv, gt_iir);
}
-static __always_inline void
+static void
gen8_cs_irq_handler(struct intel_engine_cs *engine, u32 iir, int test_shift)
{
bool tasklet = false;
if (iir & (GT_CONTEXT_SWITCH_INTERRUPT << test_shift)) {
- set_bit(ENGINE_IRQ_EXECLIST, &engine->irq_posted);
- tasklet = true;
+ if (port_count(&engine->execlist_port[0])) {
+ __set_bit(ENGINE_IRQ_EXECLIST, &engine->irq_posted);
+ tasklet = true;
+ }
}
if (iir & (GT_RENDER_USER_INTERRUPT << test_shift)) {
u32 pipestat_mask;
u32 enable_mask;
enum pipe pipe;
- u32 val;
pipestat_mask = PLANE_FLIP_DONE_INT_STATUS_VLV |
PIPE_CRC_DONE_INTERRUPT_STATUS;
enable_mask = I915_DISPLAY_PORT_INTERRUPT |
I915_DISPLAY_PIPE_A_EVENT_INTERRUPT |
- I915_DISPLAY_PIPE_B_EVENT_INTERRUPT;
+ I915_DISPLAY_PIPE_B_EVENT_INTERRUPT |
+ I915_LPE_PIPE_A_INTERRUPT |
+ I915_LPE_PIPE_B_INTERRUPT;
+
if (IS_CHERRYVIEW(dev_priv))
- enable_mask |= I915_DISPLAY_PIPE_C_EVENT_INTERRUPT;
+ enable_mask |= I915_DISPLAY_PIPE_C_EVENT_INTERRUPT |
+ I915_LPE_PIPE_C_INTERRUPT;
WARN_ON(dev_priv->irq_mask != ~0);
- val = (I915_LPE_PIPE_A_INTERRUPT |
- I915_LPE_PIPE_B_INTERRUPT |
- I915_LPE_PIPE_C_INTERRUPT);
-
- enable_mask |= val;
-
dev_priv->irq_mask = ~enable_mask;
GEN5_IRQ_INIT(VLV_, dev_priv->irq_mask, enable_mask);
void intel_irq_init(struct drm_i915_private *dev_priv)
{
struct drm_device *dev = &dev_priv->drm;
+ int i;
intel_hpd_init_work(dev_priv);
INIT_WORK(&dev_priv->rps.work, gen6_pm_rps_work);
+
INIT_WORK(&dev_priv->l3_parity.error_work, ivybridge_parity_work);
+ for (i = 0; i < MAX_L3_SLICES; ++i)
+ dev_priv->l3_parity.remap_info[i] = NULL;
if (HAS_GUC_SCHED(dev_priv))
dev_priv->pm_guc_events = GEN9_GUC_TO_HOST_INT_EVENT;
}
}
+/**
+ * intel_irq_fini - deinitializes IRQ support
+ * @i915: i915 device instance
+ *
+ * This function deinitializes all the IRQ support.
+ */
+void intel_irq_fini(struct drm_i915_private *i915)
+{
+ int i;
+
+ for (i = 0; i < MAX_L3_SLICES; ++i)
+ kfree(i915->l3_parity.remap_info[i]);
+}
+
/**
* intel_irq_install - enables the hardware interrupt
* @dev_priv: i915 device instance
.has_rc6 = 1, \
.has_rc6p = 1, \
.has_gmbus_irq = 1, \
- .has_hw_contexts = 1, \
.has_aliasing_ppgtt = 1, \
GEN_DEFAULT_PIPEOFFSETS, \
CURSOR_OFFSETS
.has_rc6 = 1, \
.has_rc6p = 1, \
.has_gmbus_irq = 1, \
- .has_hw_contexts = 1, \
.has_aliasing_ppgtt = 1, \
.has_full_ppgtt = 1, \
GEN_DEFAULT_PIPEOFFSETS, \
.has_runtime_pm = 1,
.has_rc6 = 1,
.has_gmbus_irq = 1,
- .has_hw_contexts = 1,
.has_gmch_display = 1,
.has_hotplug = 1,
.has_aliasing_ppgtt = 1,
.has_resource_streamer = 1,
.has_rc6 = 1,
.has_gmbus_irq = 1,
- .has_hw_contexts = 1,
.has_logical_ring_contexts = 1,
.has_gmch_display = 1,
.has_aliasing_ppgtt = 1,
.has_rc6 = 1, \
.has_dp_mst = 1, \
.has_gmbus_irq = 1, \
- .has_hw_contexts = 1, \
.has_logical_ring_contexts = 1, \
.has_guc = 1, \
.has_decoupled_mmio = 1, \
#define OA_TAKEN(tail, head) ((tail - head) & (OA_BUFFER_SIZE - 1))
-/* There's a HW race condition between OA unit tail pointer register updates and
+/**
+ * DOC: OA Tail Pointer Race
+ *
+ * There's a HW race condition between OA unit tail pointer register updates and
* writes to memory whereby the tail pointer can sometimes get ahead of what's
- * been written out to the OA buffer so far.
+ * been written out to the OA buffer so far (in terms of what's visible to the
+ * CPU).
+ *
+ * Although this can be observed explicitly while copying reports to userspace
+ * by checking for a zeroed report-id field in tail reports, we want to account
+ * for this earlier, as part of the _oa_buffer_check to avoid lots of redundant
+ * read() attempts.
+ *
+ * In effect we define a tail pointer for reading that lags the real tail
+ * pointer by at least %OA_TAIL_MARGIN_NSEC nanoseconds, which gives enough
+ * time for the corresponding reports to become visible to the CPU.
+ *
+ * To manage this we actually track two tail pointers:
+ * 1) An 'aging' tail with an associated timestamp that is tracked until we
+ * can trust the corresponding data is visible to the CPU; at which point
+ * it is considered 'aged'.
+ * 2) An 'aged' tail that can be used for read()ing.
+ *
+ * The two separate pointers let us decouple read()s from tail pointer aging.
+ *
+ * The tail pointers are checked and updated at a limited rate within a hrtimer
+ * callback (the same callback that is used for delivering POLLIN events)
*
- * Although this can be observed explicitly by checking for a zeroed report-id
- * field in tail reports, it seems preferable to account for this earlier e.g.
- * as part of the _oa_buffer_is_empty checks to minimize -EAGAIN polling cycles
- * in this situation.
+ * Initially the tails are marked invalid with %INVALID_TAIL_PTR which
+ * indicates that an updated tail pointer is needed.
*
- * To give time for the most recent reports to land before they may be copied to
- * userspace, the driver operates as if the tail pointer effectively lags behind
- * the HW tail pointer by 'tail_margin' bytes. The margin in bytes is calculated
- * based on this constant in nanoseconds, the current OA sampling exponent
- * and current report size.
+ * Most of the implementation details for this workaround are in
+ * gen7_oa_buffer_check_unlocked() and gen7_appand_oa_reports()
*
- * There is also a fallback check while reading to simply skip over reports with
- * a zeroed report-id.
+ * Note for posterity: previously the driver used to define an effective tail
+ * pointer that lagged the real pointer by a 'tail margin' measured in bytes
+ * derived from %OA_TAIL_MARGIN_NSEC and the configured sampling frequency.
+ * This was flawed considering that the OA unit may also automatically generate
+ * non-periodic reports (such as on context switch) or the OA unit may be
+ * enabled without any periodic sampling.
*/
#define OA_TAIL_MARGIN_NSEC 100000ULL
+#define INVALID_TAIL_PTR 0xffffffff
/* frequency for checking whether the OA unit has written new reports to the
* circular OA buffer...
int oa_period_exponent;
};
-/* NB: This is either called via fops or the poll check hrtimer (atomic ctx)
+/**
+ * gen7_oa_buffer_check_unlocked - check for data and update tail ptr state
+ * @dev_priv: i915 device instance
+ *
+ * This is either called via fops (for blocking reads in user ctx) or the poll
+ * check hrtimer (atomic ctx) to check the OA buffer tail pointer and check
+ * if there is data available for userspace to read.
*
- * It's safe to read OA config state here unlocked, assuming that this is only
- * called while the stream is enabled, while the global OA configuration can't
- * be modified.
+ * This function is central to providing a workaround for the OA unit tail
+ * pointer having a race with respect to what data is visible to the CPU.
+ * It is responsible for reading tail pointers from the hardware and giving
+ * the pointers time to 'age' before they are made available for reading.
+ * (See description of OA_TAIL_MARGIN_NSEC above for further details.)
*
- * Note: we don't lock around the head/tail reads even though there's the slim
- * possibility of read() fop errors forcing a re-init of the OA buffer
- * pointers. A race here could result in a false positive !empty status which
- * is acceptable.
+ * Besides returning true when there is data available to read() this function
+ * also has the side effect of updating the oa_buffer.tails[], .aging_timestamp
+ * and .aged_tail_idx state used for reading.
+ *
+ * Note: It's safe to read OA config state here unlocked, assuming that this is
+ * only called while the stream is enabled, while the global OA configuration
+ * can't be modified.
+ *
+ * Returns: %true if the OA buffer contains data, else %false
*/
-static bool gen7_oa_buffer_is_empty_fop_unlocked(struct drm_i915_private *dev_priv)
+static bool gen7_oa_buffer_check_unlocked(struct drm_i915_private *dev_priv)
{
int report_size = dev_priv->perf.oa.oa_buffer.format_size;
- u32 oastatus2 = I915_READ(GEN7_OASTATUS2);
- u32 oastatus1 = I915_READ(GEN7_OASTATUS1);
- u32 head = oastatus2 & GEN7_OASTATUS2_HEAD_MASK;
- u32 tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+ unsigned long flags;
+ unsigned int aged_idx;
+ u32 oastatus1;
+ u32 head, hw_tail, aged_tail, aging_tail;
+ u64 now;
+
+ /* We have to consider the (unlikely) possibility that read() errors
+ * could result in an OA buffer reset which might reset the head,
+ * tails[] and aged_tail state.
+ */
+ spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+ /* NB: The head we observe here might effectively be a little out of
+ * date (between head and tails[aged_idx].offset if there is currently
+ * a read() in progress.
+ */
+ head = dev_priv->perf.oa.oa_buffer.head;
+
+ aged_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+ aged_tail = dev_priv->perf.oa.oa_buffer.tails[aged_idx].offset;
+ aging_tail = dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset;
- return OA_TAKEN(tail, head) <
- dev_priv->perf.oa.tail_margin + report_size;
+ oastatus1 = I915_READ(GEN7_OASTATUS1);
+ hw_tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
+
+ /* The tail pointer increases in 64 byte increments,
+ * not in report_size steps...
+ */
+ hw_tail &= ~(report_size - 1);
+
+ now = ktime_get_mono_fast_ns();
+
+ /* Update the aged tail
+ *
+ * Flip the tail pointer available for read()s once the aging tail is
+ * old enough to trust that the corresponding data will be visible to
+ * the CPU...
+ *
+ * Do this before updating the aging pointer in case we may be able to
+ * immediately start aging a new pointer too (if new data has become
+ * available) without needing to wait for a later hrtimer callback.
+ */
+ if (aging_tail != INVALID_TAIL_PTR &&
+ ((now - dev_priv->perf.oa.oa_buffer.aging_timestamp) >
+ OA_TAIL_MARGIN_NSEC)) {
+ aged_idx ^= 1;
+ dev_priv->perf.oa.oa_buffer.aged_tail_idx = aged_idx;
+
+ aged_tail = aging_tail;
+
+ /* Mark that we need a new pointer to start aging... */
+ dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset = INVALID_TAIL_PTR;
+ aging_tail = INVALID_TAIL_PTR;
+ }
+
+ /* Update the aging tail
+ *
+ * We throttle aging tail updates until we have a new tail that
+ * represents >= one report more data than is already available for
+ * reading. This ensures there will be enough data for a successful
+ * read once this new pointer has aged and ensures we will give the new
+ * pointer time to age.
+ */
+ if (aging_tail == INVALID_TAIL_PTR &&
+ (aged_tail == INVALID_TAIL_PTR ||
+ OA_TAKEN(hw_tail, aged_tail) >= report_size)) {
+ struct i915_vma *vma = dev_priv->perf.oa.oa_buffer.vma;
+ u32 gtt_offset = i915_ggtt_offset(vma);
+
+ /* Be paranoid and do a bounds check on the pointer read back
+ * from hardware, just in case some spurious hardware condition
+ * could put the tail out of bounds...
+ */
+ if (hw_tail >= gtt_offset &&
+ hw_tail < (gtt_offset + OA_BUFFER_SIZE)) {
+ dev_priv->perf.oa.oa_buffer.tails[!aged_idx].offset =
+ aging_tail = hw_tail;
+ dev_priv->perf.oa.oa_buffer.aging_timestamp = now;
+ } else {
+ DRM_ERROR("Ignoring spurious out of range OA buffer tail pointer = %u\n",
+ hw_tail);
+ }
+ }
+
+ spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+ return aged_tail == INVALID_TAIL_PTR ?
+ false : OA_TAKEN(aged_tail, head) >= report_size;
}
/**
* @buf: destination buffer given by userspace
* @count: the number of bytes userspace wants to read
* @offset: (inout): the current position for writing into @buf
- * @head_ptr: (inout): the current oa buffer cpu read position
- * @tail: the current oa buffer gpu write position
*
* Notably any error condition resulting in a short read (-%ENOSPC or
* -%EFAULT) will be returned even though one or more records may
* userspace.
*
* Note: reports are consumed from the head, and appended to the
- * tail, so the head chases the tail?... If you think that's mad
+ * tail, so the tail chases the head?... If you think that's mad
* and back-to-front you're not alone, but this follows the
* Gen PRM naming convention.
*
static int gen7_append_oa_reports(struct i915_perf_stream *stream,
char __user *buf,
size_t count,
- size_t *offset,
- u32 *head_ptr,
- u32 tail)
+ size_t *offset)
{
struct drm_i915_private *dev_priv = stream->dev_priv;
int report_size = dev_priv->perf.oa.oa_buffer.format_size;
u8 *oa_buf_base = dev_priv->perf.oa.oa_buffer.vaddr;
- int tail_margin = dev_priv->perf.oa.tail_margin;
u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
u32 mask = (OA_BUFFER_SIZE - 1);
- u32 head;
+ size_t start_offset = *offset;
+ unsigned long flags;
+ unsigned int aged_tail_idx;
+ u32 head, tail;
u32 taken;
int ret = 0;
if (WARN_ON(!stream->enabled))
return -EIO;
- head = *head_ptr - gtt_offset;
- tail -= gtt_offset;
+ spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
- /* The OA unit is expected to wrap the tail pointer according to the OA
- * buffer size and since we should never write a misaligned head
- * pointer we don't expect to read one back either...
- */
- if (tail > OA_BUFFER_SIZE || head > OA_BUFFER_SIZE ||
- head % report_size) {
- DRM_ERROR("Inconsistent OA buffer pointer (head = %u, tail = %u): force restart\n",
- head, tail);
- dev_priv->perf.oa.ops.oa_disable(dev_priv);
- dev_priv->perf.oa.ops.oa_enable(dev_priv);
- *head_ptr = I915_READ(GEN7_OASTATUS2) &
- GEN7_OASTATUS2_HEAD_MASK;
- return -EIO;
- }
+ head = dev_priv->perf.oa.oa_buffer.head;
+ aged_tail_idx = dev_priv->perf.oa.oa_buffer.aged_tail_idx;
+ tail = dev_priv->perf.oa.oa_buffer.tails[aged_tail_idx].offset;
+ spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
- /* The tail pointer increases in 64 byte increments, not in report_size
- * steps...
+ /* An invalid tail pointer here means we're still waiting for the poll
+ * hrtimer callback to give us a pointer
*/
- tail &= ~(report_size - 1);
+ if (tail == INVALID_TAIL_PTR)
+ return -EAGAIN;
- /* Move the tail pointer back by the current tail_margin to account for
- * the possibility that the latest reports may not have really landed
- * in memory yet...
+ /* NB: oa_buffer.head/tail include the gtt_offset which we don't want
+ * while indexing relative to oa_buf_base.
*/
+ head -= gtt_offset;
+ tail -= gtt_offset;
- if (OA_TAKEN(tail, head) < report_size + tail_margin)
- return -EAGAIN;
+ /* An out of bounds or misaligned head or tail pointer implies a driver
+ * bug since we validate + align the tail pointers we read from the
+ * hardware and we are in full control of the head pointer which should
+ * only be incremented by multiples of the report size (notably also
+ * all a power of two).
+ */
+ if (WARN_ONCE(head > OA_BUFFER_SIZE || head % report_size ||
+ tail > OA_BUFFER_SIZE || tail % report_size,
+ "Inconsistent OA buffer pointers: head = %u, tail = %u\n",
+ head, tail))
+ return -EIO;
- tail -= tail_margin;
- tail &= mask;
for (/* none */;
(taken = OA_TAKEN(tail, head));
* copying it to userspace...
*/
if (report32[0] == 0) {
- DRM_NOTE("Skipping spurious, invalid OA report\n");
+ if (__ratelimit(&dev_priv->perf.oa.spurious_report_rs))
+ DRM_NOTE("Skipping spurious, invalid OA report\n");
continue;
}
report32[0] = 0;
}
- *head_ptr = gtt_offset + head;
+ if (start_offset != *offset) {
+ spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
+ /* We removed the gtt_offset for the copy loop above, indexing
+ * relative to oa_buf_base so put back here...
+ */
+ head += gtt_offset;
+
+ I915_WRITE(GEN7_OASTATUS2,
+ ((head & GEN7_OASTATUS2_HEAD_MASK) |
+ OA_MEM_SELECT_GGTT));
+ dev_priv->perf.oa.oa_buffer.head = head;
+
+ spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+ }
return ret;
}
size_t *offset)
{
struct drm_i915_private *dev_priv = stream->dev_priv;
- int report_size = dev_priv->perf.oa.oa_buffer.format_size;
- u32 oastatus2;
u32 oastatus1;
- u32 head;
- u32 tail;
int ret;
if (WARN_ON(!dev_priv->perf.oa.oa_buffer.vaddr))
return -EIO;
- oastatus2 = I915_READ(GEN7_OASTATUS2);
oastatus1 = I915_READ(GEN7_OASTATUS1);
- head = oastatus2 & GEN7_OASTATUS2_HEAD_MASK;
- tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
-
/* XXX: On Haswell we don't have a safe way to clear oastatus1
* bits while the OA unit is enabled (while the tail pointer
* may be updated asynchronously) so we ignore status bits
dev_priv->perf.oa.ops.oa_disable(dev_priv);
dev_priv->perf.oa.ops.oa_enable(dev_priv);
- oastatus2 = I915_READ(GEN7_OASTATUS2);
oastatus1 = I915_READ(GEN7_OASTATUS1);
-
- head = oastatus2 & GEN7_OASTATUS2_HEAD_MASK;
- tail = oastatus1 & GEN7_OASTATUS1_TAIL_MASK;
}
if (unlikely(oastatus1 & GEN7_OASTATUS1_REPORT_LOST)) {
GEN7_OASTATUS1_REPORT_LOST;
}
- ret = gen7_append_oa_reports(stream, buf, count, offset,
- &head, tail);
-
- /* All the report sizes are a power of two and the
- * head should always be incremented by some multiple
- * of the report size.
- *
- * A warning here, but notably if we later read back a
- * misaligned pointer we will treat that as a bug since
- * it could lead to a buffer overrun.
- */
- WARN_ONCE(head & (report_size - 1),
- "i915: Writing misaligned OA head pointer");
-
- /* Note: we update the head pointer here even if an error
- * was returned since the error may represent a short read
- * where some some reports were successfully copied.
- */
- I915_WRITE(GEN7_OASTATUS2,
- ((head & GEN7_OASTATUS2_HEAD_MASK) |
- OA_MEM_SELECT_GGTT));
-
- return ret;
+ return gen7_append_oa_reports(stream, buf, count, offset);
}
/**
if (!dev_priv->perf.oa.periodic)
return -EIO;
- /* Note: the oa_buffer_is_empty() condition is ok to run unlocked as it
- * just performs mmio reads of the OA buffer head + tail pointers and
- * it's assumed we're handling some operation that implies the stream
- * can't be destroyed until completion (such as a read()) that ensures
- * the device + OA buffer can't disappear
- */
return wait_event_interruptible(dev_priv->perf.oa.poll_wq,
- !dev_priv->perf.oa.ops.oa_buffer_is_empty(dev_priv));
+ dev_priv->perf.oa.ops.oa_buffer_check(dev_priv));
}
/**
{
struct drm_i915_private *dev_priv = stream->dev_priv;
struct intel_engine_cs *engine = dev_priv->engine[RCS];
+ struct intel_ring *ring;
int ret;
ret = i915_mutex_lock_interruptible(&dev_priv->drm);
*
* NB: implied RCS engine...
*/
- ret = engine->context_pin(engine, stream->ctx);
- if (ret)
- goto unlock;
+ ring = engine->context_pin(engine, stream->ctx);
+ mutex_unlock(&dev_priv->drm.struct_mutex);
+ if (IS_ERR(ring))
+ return PTR_ERR(ring);
/* Explicitly track the ID (instead of calling i915_ggtt_offset()
* on the fly) considering the difference with gen8+ and
dev_priv->perf.oa.specific_ctx_id =
i915_ggtt_offset(stream->ctx->engine[engine->id].state);
-unlock:
- mutex_unlock(&dev_priv->drm.struct_mutex);
-
- return ret;
+ return 0;
}
/**
oa_put_render_ctx_id(stream);
dev_priv->perf.oa.exclusive_stream = NULL;
+
+ if (dev_priv->perf.oa.spurious_report_rs.missed) {
+ DRM_NOTE("%d spurious OA report notices suppressed due to ratelimiting\n",
+ dev_priv->perf.oa.spurious_report_rs.missed);
+ }
}
static void gen7_init_oa_buffer(struct drm_i915_private *dev_priv)
{
u32 gtt_offset = i915_ggtt_offset(dev_priv->perf.oa.oa_buffer.vma);
+ unsigned long flags;
+
+ spin_lock_irqsave(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
/* Pre-DevBDW: OABUFFER must be set with counters off,
* before OASTATUS1, but after OASTATUS2
*/
I915_WRITE(GEN7_OASTATUS2, gtt_offset | OA_MEM_SELECT_GGTT); /* head */
+ dev_priv->perf.oa.oa_buffer.head = gtt_offset;
+
I915_WRITE(GEN7_OABUFFER, gtt_offset);
+
I915_WRITE(GEN7_OASTATUS1, gtt_offset | OABUFFER_SIZE_16M); /* tail */
+ /* Mark that we need updated tail pointers to read from... */
+ dev_priv->perf.oa.oa_buffer.tails[0].offset = INVALID_TAIL_PTR;
+ dev_priv->perf.oa.oa_buffer.tails[1].offset = INVALID_TAIL_PTR;
+
+ spin_unlock_irqrestore(&dev_priv->perf.oa.oa_buffer.ptr_lock, flags);
+
/* On Haswell we have to track which OASTATUS1 flags we've
* already seen since they can't be cleared while periodic
* sampling is enabled.
hrtimer_cancel(&dev_priv->perf.oa.poll_check_timer);
}
-static u64 oa_exponent_to_ns(struct drm_i915_private *dev_priv, int exponent)
-{
- return div_u64(1000000000ULL * (2ULL << exponent),
- dev_priv->perf.oa.timestamp_frequency);
-}
-
static const struct i915_perf_stream_ops i915_oa_stream_ops = {
.destroy = i915_oa_stream_destroy,
.enable = i915_oa_stream_enable,
return -EINVAL;
}
+ /* We set up some ratelimit state to potentially throttle any _NOTES
+ * about spurious, invalid OA reports which we don't forward to
+ * userspace.
+ *
+ * The initialization is associated with opening the stream (not driver
+ * init) considering we print a _NOTE about any throttling when closing
+ * the stream instead of waiting until driver _fini which no one would
+ * ever see.
+ *
+ * Using the same limiting factors as printk_ratelimit()
+ */
+ ratelimit_state_init(&dev_priv->perf.oa.spurious_report_rs,
+ 5 * HZ, 10);
+ /* Since we use a DRM_NOTE for spurious reports it would be
+ * inconsistent to let __ratelimit() automatically print a warning for
+ * throttling.
+ */
+ ratelimit_set_flags(&dev_priv->perf.oa.spurious_report_rs,
+ RATELIMIT_MSG_ON_RELEASE);
+
stream->sample_size = sizeof(struct drm_i915_perf_record_header);
format_size = dev_priv->perf.oa.oa_formats[props->oa_format].size;
dev_priv->perf.oa.metrics_set = props->metrics_set;
dev_priv->perf.oa.periodic = props->oa_periodic;
- if (dev_priv->perf.oa.periodic) {
- u32 tail;
-
+ if (dev_priv->perf.oa.periodic)
dev_priv->perf.oa.period_exponent = props->oa_period_exponent;
- /* See comment for OA_TAIL_MARGIN_NSEC for details
- * about this tail_margin...
- */
- tail = div64_u64(OA_TAIL_MARGIN_NSEC,
- oa_exponent_to_ns(dev_priv,
- props->oa_period_exponent));
- dev_priv->perf.oa.tail_margin = (tail + 1) * format_size;
- }
-
if (stream->ctx) {
ret = oa_get_render_ctx_id(stream);
if (ret)
mutex_unlock(&dev_priv->perf.lock);
}
- if (ret >= 0) {
+ /* We allow the poll checking to sometimes report false positive POLLIN
+ * events where we might actually report EAGAIN on read() if there's
+ * not really any data available. In this situation though we don't
+ * want to enter a busy loop between poll() reporting a POLLIN event
+ * and read() returning -EAGAIN. Clearing the oa.pollin state here
+ * effectively ensures we back off until the next hrtimer callback
+ * before reporting another POLLIN event.
+ */
+ if (ret >= 0 || ret == -EAGAIN) {
/* Maybe make ->pollin per-stream state if we support multiple
* concurrent streams in the future.
*/
container_of(hrtimer, typeof(*dev_priv),
perf.oa.poll_check_timer);
- if (!dev_priv->perf.oa.ops.oa_buffer_is_empty(dev_priv)) {
+ if (dev_priv->perf.oa.ops.oa_buffer_check(dev_priv)) {
dev_priv->perf.oa.pollin = true;
wake_up(&dev_priv->perf.oa.poll_wq);
}
break;
case DRM_I915_PERF_PROP_OA_FORMAT:
if (value == 0 || value >= I915_OA_FORMAT_MAX) {
- DRM_DEBUG("Invalid OA report format\n");
+ DRM_DEBUG("Out-of-range OA report format %llu\n",
+ value);
return -EINVAL;
}
if (!dev_priv->perf.oa.oa_formats[value].size) {
- DRM_DEBUG("Invalid OA report format\n");
+ DRM_DEBUG("Unsupported OA report format %llu\n",
+ value);
return -EINVAL;
}
props->oa_format = value;
INIT_LIST_HEAD(&dev_priv->perf.streams);
mutex_init(&dev_priv->perf.lock);
spin_lock_init(&dev_priv->perf.hook_lock);
+ spin_lock_init(&dev_priv->perf.oa.oa_buffer.ptr_lock);
dev_priv->perf.oa.ops.init_oa_buffer = gen7_init_oa_buffer;
dev_priv->perf.oa.ops.enable_metric_set = hsw_enable_metric_set;
dev_priv->perf.oa.ops.oa_enable = gen7_oa_enable;
dev_priv->perf.oa.ops.oa_disable = gen7_oa_disable;
dev_priv->perf.oa.ops.read = gen7_oa_read;
- dev_priv->perf.oa.ops.oa_buffer_is_empty =
- gen7_oa_buffer_is_empty_fop_unlocked;
-
- dev_priv->perf.oa.timestamp_frequency = 12500000;
+ dev_priv->perf.oa.ops.oa_buffer_check =
+ gen7_oa_buffer_check_unlocked;
dev_priv->perf.oa.oa_formats = hsw_oa_formats;
#define VECS_HW 3
#define VCS2_HW 4
+/* Engine class */
+
+#define RENDER_CLASS 0
+#define VIDEO_DECODE_CLASS 1
+#define VIDEO_ENHANCEMENT_CLASS 2
+#define COPY_ENGINE_CLASS 3
+#define OTHER_CLASS 4
+
/* PCI config space */
#define MCHBAR_I915 0x44
#define CLKCFG_FSB_667 (3 << 0) /* hrawclk 166 */
#define CLKCFG_FSB_800 (2 << 0) /* hrawclk 200 */
#define CLKCFG_FSB_1067 (6 << 0) /* hrawclk 266 */
+#define CLKCFG_FSB_1067_ALT (0 << 0) /* hrawclk 266 */
#define CLKCFG_FSB_1333 (7 << 0) /* hrawclk 333 */
-/* Note, below two are guess */
-#define CLKCFG_FSB_1600 (4 << 0) /* hrawclk 400 */
-#define CLKCFG_FSB_1600_ALT (0 << 0) /* hrawclk 400 */
+/*
+ * Note that on at least on ELK the below value is reported for both
+ * 333 and 400 MHz BIOS FSB setting, but given that the gmch datasheet
+ * lists only 200/266/333 MHz FSB as supported let's decode it as 333 MHz.
+ */
+#define CLKCFG_FSB_1333_ALT (4 << 0) /* hrawclk 333 */
#define CLKCFG_FSB_MASK (7 << 0)
#define CLKCFG_MEM_533 (1 << 4)
#define CLKCFG_MEM_667 (2 << 4)
#define GEN7_CXT_VFSTATE_SIZE(ctx_reg) (((ctx_reg) >> 0) & 0x3f)
#define GEN7_CXT_TOTAL_SIZE(ctx_reg) (GEN7_CXT_EXTENDED_SIZE(ctx_reg) + \
GEN7_CXT_VFSTATE_SIZE(ctx_reg))
-/* Haswell does have the CXT_SIZE register however it does not appear to be
- * valid. Now, docs explain in dwords what is in the context object. The full
- * size is 70720 bytes, however, the power context and execlist context will
- * never be saved (power context is stored elsewhere, and execlists don't work
- * on HSW) - so the final size, including the extra state required for the
- * Resource Streamer, is 66944 bytes, which rounds to 17 pages.
- */
-#define HSW_CXT_TOTAL_SIZE (17 * PAGE_SIZE)
-/* Same as Haswell, but 72064 bytes now. */
-#define GEN8_CXT_TOTAL_SIZE (18 * PAGE_SIZE)
enum {
INTEL_ADVANCED_CONTEXT = 0,
#define CURSOR_MODE_128_ARGB_AX ((1 << 5) | CURSOR_MODE_128_32B_AX)
#define CURSOR_MODE_256_ARGB_AX ((1 << 5) | CURSOR_MODE_256_32B_AX)
#define CURSOR_MODE_64_ARGB_AX ((1 << 5) | CURSOR_MODE_64_32B_AX)
-#define MCURSOR_PIPE_SELECT (1 << 28)
-#define MCURSOR_PIPE_A 0x00
-#define MCURSOR_PIPE_B (1 << 28)
+#define MCURSOR_PIPE_SELECT(pipe) ((pipe) << 28)
#define MCURSOR_GAMMA_ENABLE (1 << 26)
#define CURSOR_ROTATE_180 (1<<15)
#define CURSOR_TRICKLE_FEED_DISABLE (1 << 14)
#define CURSOR_POS_SIGN 0x8000
#define CURSOR_X_SHIFT 0
#define CURSOR_Y_SHIFT 16
-#define CURSIZE _MMIO(0x700a0)
+#define CURSIZE _MMIO(0x700a0) /* 845/865 */
+#define _CUR_FBC_CTL_A 0x700a0 /* ivb+ */
+#define CUR_FBC_CTL_EN (1 << 31)
#define _CURBCNTR 0x700c0
#define _CURBBASE 0x700c4
#define _CURBPOS 0x700c8
#define CURCNTR(pipe) _CURSOR2(pipe, _CURACNTR)
#define CURBASE(pipe) _CURSOR2(pipe, _CURABASE)
#define CURPOS(pipe) _CURSOR2(pipe, _CURAPOS)
+#define CUR_FBC_CTL(pipe) _CURSOR2(pipe, _CUR_FBC_CTL_A)
#define CURSOR_A_OFFSET 0x70080
#define CURSOR_B_OFFSET 0x700c0
#define DISPPLANE_PIPE_CSC_ENABLE (1<<24)
#define DISPPLANE_SEL_PIPE_SHIFT 24
#define DISPPLANE_SEL_PIPE_MASK (3<<DISPPLANE_SEL_PIPE_SHIFT)
-#define DISPPLANE_SEL_PIPE_A 0
-#define DISPPLANE_SEL_PIPE_B (1<<DISPPLANE_SEL_PIPE_SHIFT)
+#define DISPPLANE_SEL_PIPE(pipe) ((pipe)<<DISPPLANE_SEL_PIPE_SHIFT)
#define DISPPLANE_SRC_KEY_ENABLE (1<<22)
#define DISPPLANE_SRC_KEY_DISABLE 0
#define DISPPLANE_LINE_DOUBLE (1<<20)
/* MIPI DSI registers */
-#define _MIPI_PORT(port, a, c) ((port) ? c : a) /* ports A and C only */
+#define _MIPI_PORT(port, a, c) (((port) == PORT_A) ? a : c) /* ports A and C only */
#define _MMIO_MIPI(port, a, c) _MMIO(_MIPI_PORT(port, a, c))
#define MIPIO_TXESC_CLK_DIV1 _MMIO(0x160004)
#include <linux/reservation.h>
#include "i915_sw_fence.h"
+#include "i915_selftest.h"
#define I915_SW_FENCE_FLAG_ALLOC BIT(3) /* after WQ_FLAG_* for safety */
}
#endif
-static void i915_sw_fence_release(struct kref *kref)
-{
- struct i915_sw_fence *fence = container_of(kref, typeof(*fence), kref);
-
- WARN_ON(atomic_read(&fence->pending) > 0);
- debug_fence_destroy(fence);
-
- if (fence->flags & I915_SW_FENCE_MASK) {
- __i915_sw_fence_notify(fence, FENCE_FREE);
- } else {
- i915_sw_fence_fini(fence);
- kfree(fence);
- }
-}
-
-static void i915_sw_fence_put(struct i915_sw_fence *fence)
-{
- debug_fence_assert(fence);
- kref_put(&fence->kref, i915_sw_fence_release);
-}
-
-static struct i915_sw_fence *i915_sw_fence_get(struct i915_sw_fence *fence)
-{
- debug_fence_assert(fence);
- kref_get(&fence->kref);
- return fence;
-}
-
static void __i915_sw_fence_wake_up_all(struct i915_sw_fence *fence,
struct list_head *continuation)
{
debug_fence_set_state(fence, DEBUG_FENCE_IDLE, DEBUG_FENCE_NOTIFY);
- if (fence->flags & I915_SW_FENCE_MASK &&
- __i915_sw_fence_notify(fence, FENCE_COMPLETE) != NOTIFY_DONE)
+ if (__i915_sw_fence_notify(fence, FENCE_COMPLETE) != NOTIFY_DONE)
return;
debug_fence_set_state(fence, DEBUG_FENCE_NOTIFY, DEBUG_FENCE_IDLE);
__i915_sw_fence_wake_up_all(fence, continuation);
+
+ debug_fence_destroy(fence);
+ __i915_sw_fence_notify(fence, FENCE_FREE);
}
static void i915_sw_fence_complete(struct i915_sw_fence *fence)
const char *name,
struct lock_class_key *key)
{
- BUG_ON((unsigned long)fn & ~I915_SW_FENCE_MASK);
+ BUG_ON(!fn || (unsigned long)fn & ~I915_SW_FENCE_MASK);
debug_fence_init(fence);
__init_waitqueue_head(&fence->wait, name, key);
- kref_init(&fence->kref);
atomic_set(&fence->pending, 1);
fence->flags = (unsigned long)fn;
}
-static void __i915_sw_fence_commit(struct i915_sw_fence *fence)
-{
- i915_sw_fence_complete(fence);
- i915_sw_fence_put(fence);
-}
-
void i915_sw_fence_commit(struct i915_sw_fence *fence)
{
debug_fence_activate(fence);
- __i915_sw_fence_commit(fence);
+ i915_sw_fence_complete(fence);
}
static int i915_sw_fence_wake(wait_queue_t *wq, unsigned mode, int flags, void *key)
{
list_del(&wq->task_list);
__i915_sw_fence_complete(wq->private, key);
- i915_sw_fence_put(wq->private);
+
if (wq->flags & I915_SW_FENCE_FLAG_ALLOC)
kfree(wq);
return 0;
unsigned long flags;
bool err;
- if (!IS_ENABLED(CONFIG_I915_SW_FENCE_CHECK_DAG))
+ if (!IS_ENABLED(CONFIG_DRM_I915_SW_FENCE_CHECK_DAG))
return false;
spin_lock_irqsave(&i915_sw_fence_lock, flags);
INIT_LIST_HEAD(&wq->task_list);
wq->flags = pending;
wq->func = i915_sw_fence_wake;
- wq->private = i915_sw_fence_get(fence);
+ wq->private = fence;
i915_sw_fence_await(fence);
dma_fence_put(cb->dma);
cb->dma = NULL;
- __i915_sw_fence_commit(cb->fence);
+ i915_sw_fence_complete(cb->fence);
cb->timer.function = NULL;
}
del_timer_sync(&cb->timer);
if (cb->timer.function)
- __i915_sw_fence_commit(cb->fence);
+ i915_sw_fence_complete(cb->fence);
dma_fence_put(cb->dma);
kfree(cb);
return dma_fence_wait(dma, false);
}
- cb->fence = i915_sw_fence_get(fence);
+ cb->fence = fence;
i915_sw_fence_await(fence);
cb->dma = NULL;
return ret;
}
+
+#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
+#include "selftests/i915_sw_fence.c"
+#endif
struct i915_sw_fence {
wait_queue_head_t wait;
unsigned long flags;
- struct kref kref;
atomic_t pending;
};
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include <linux/slab.h>
+
+#include "i915_syncmap.h"
+
+#include "i915_gem.h" /* GEM_BUG_ON() */
+#include "i915_selftest.h"
+
+#define SHIFT ilog2(KSYNCMAP)
+#define MASK (KSYNCMAP - 1)
+
+/*
+ * struct i915_syncmap is a layer of a radixtree that maps a u64 fence
+ * context id to the last u32 fence seqno waited upon from that context.
+ * Unlike lib/radixtree it uses a parent pointer that allows traversal back to
+ * the root. This allows us to access the whole tree via a single pointer
+ * to the most recently used layer. We expect fence contexts to be dense
+ * and most reuse to be on the same i915_gem_context but on neighbouring
+ * engines (i.e. on adjacent contexts) and reuse the same leaf, a very
+ * effective lookup cache. If the new lookup is not on the same leaf, we
+ * expect it to be on the neighbouring branch.
+ *
+ * A leaf holds an array of u32 seqno, and has height 0. The bitmap field
+ * allows us to store whether a particular seqno is valid (i.e. allows us
+ * to distinguish unset from 0).
+ *
+ * A branch holds an array of layer pointers, and has height > 0, and always
+ * has at least 2 layers (either branches or leaves) below it.
+ *
+ * For example,
+ * for x in
+ * 0 1 2 0x10 0x11 0x200 0x201
+ * 0x500000 0x500001 0x503000 0x503001
+ * 0xE<<60:
+ * i915_syncmap_set(&sync, x, lower_32_bits(x));
+ * will build a tree like:
+ * 0xXXXXXXXXXXXXXXXX
+ * 0-> 0x0000000000XXXXXX
+ * | 0-> 0x0000000000000XXX
+ * | | 0-> 0x00000000000000XX
+ * | | | 0-> 0x000000000000000X 0:0, 1:1, 2:2
+ * | | | 1-> 0x000000000000001X 0:10, 1:11
+ * | | 2-> 0x000000000000020X 0:200, 1:201
+ * | 5-> 0x000000000050XXXX
+ * | 0-> 0x000000000050000X 0:500000, 1:500001
+ * | 3-> 0x000000000050300X 0:503000, 1:503001
+ * e-> 0xe00000000000000X e:e
+ */
+
+struct i915_syncmap {
+ u64 prefix;
+ unsigned int height;
+ unsigned int bitmap;
+ struct i915_syncmap *parent;
+ /*
+ * Following this header is an array of either seqno or child pointers:
+ * union {
+ * u32 seqno[KSYNCMAP];
+ * struct i915_syncmap *child[KSYNCMAP];
+ * };
+ */
+};
+
+/**
+ * i915_syncmap_init -- initialise the #i915_syncmap
+ * @root - pointer to the #i915_syncmap
+ */
+void i915_syncmap_init(struct i915_syncmap **root)
+{
+ BUILD_BUG_ON_NOT_POWER_OF_2(KSYNCMAP);
+ BUILD_BUG_ON_NOT_POWER_OF_2(SHIFT);
+ BUILD_BUG_ON(KSYNCMAP > BITS_PER_BYTE * sizeof((*root)->bitmap));
+ *root = NULL;
+}
+
+static inline u32 *__sync_seqno(struct i915_syncmap *p)
+{
+ GEM_BUG_ON(p->height);
+ return (u32 *)(p + 1);
+}
+
+static inline struct i915_syncmap **__sync_child(struct i915_syncmap *p)
+{
+ GEM_BUG_ON(!p->height);
+ return (struct i915_syncmap **)(p + 1);
+}
+
+static inline unsigned int
+__sync_branch_idx(const struct i915_syncmap *p, u64 id)
+{
+ return (id >> p->height) & MASK;
+}
+
+static inline unsigned int
+__sync_leaf_idx(const struct i915_syncmap *p, u64 id)
+{
+ GEM_BUG_ON(p->height);
+ return id & MASK;
+}
+
+static inline u64 __sync_branch_prefix(const struct i915_syncmap *p, u64 id)
+{
+ return id >> p->height >> SHIFT;
+}
+
+static inline u64 __sync_leaf_prefix(const struct i915_syncmap *p, u64 id)
+{
+ GEM_BUG_ON(p->height);
+ return id >> SHIFT;
+}
+
+static inline bool seqno_later(u32 a, u32 b)
+{
+ return (s32)(a - b) >= 0;
+}
+
+/**
+ * i915_syncmap_is_later -- compare against the last know sync point
+ * @root - pointer to the #i915_syncmap
+ * @id - the context id (other timeline) we are synchronising to
+ * @seqno - the sequence number along the other timeline
+ *
+ * If we have already synchronised this @root timeline with another (@id) then
+ * we can omit any repeated or earlier synchronisation requests. If the two
+ * timelines are already coupled, we can also omit the dependency between the
+ * two as that is already known via the timeline.
+ *
+ * Returns true if the two timelines are already synchronised wrt to @seqno,
+ * false if not and the synchronisation must be emitted.
+ */
+bool i915_syncmap_is_later(struct i915_syncmap **root, u64 id, u32 seqno)
+{
+ struct i915_syncmap *p;
+ unsigned int idx;
+
+ p = *root;
+ if (!p)
+ return false;
+
+ if (likely(__sync_leaf_prefix(p, id) == p->prefix))
+ goto found;
+
+ /* First climb the tree back to a parent branch */
+ do {
+ p = p->parent;
+ if (!p)
+ return false;
+
+ if (__sync_branch_prefix(p, id) == p->prefix)
+ break;
+ } while (1);
+
+ /* And then descend again until we find our leaf */
+ do {
+ if (!p->height)
+ break;
+
+ p = __sync_child(p)[__sync_branch_idx(p, id)];
+ if (!p)
+ return false;
+
+ if (__sync_branch_prefix(p, id) != p->prefix)
+ return false;
+ } while (1);
+
+ *root = p;
+found:
+ idx = __sync_leaf_idx(p, id);
+ if (!(p->bitmap & BIT(idx)))
+ return false;
+
+ return seqno_later(__sync_seqno(p)[idx], seqno);
+}
+
+static struct i915_syncmap *
+__sync_alloc_leaf(struct i915_syncmap *parent, u64 id)
+{
+ struct i915_syncmap *p;
+
+ p = kmalloc(sizeof(*p) + KSYNCMAP * sizeof(u32), GFP_KERNEL);
+ if (unlikely(!p))
+ return NULL;
+
+ p->parent = parent;
+ p->height = 0;
+ p->bitmap = 0;
+ p->prefix = __sync_leaf_prefix(p, id);
+ return p;
+}
+
+static inline void __sync_set_seqno(struct i915_syncmap *p, u64 id, u32 seqno)
+{
+ unsigned int idx = __sync_leaf_idx(p, id);
+
+ p->bitmap |= BIT(idx);
+ __sync_seqno(p)[idx] = seqno;
+}
+
+static inline void __sync_set_child(struct i915_syncmap *p,
+ unsigned int idx,
+ struct i915_syncmap *child)
+{
+ p->bitmap |= BIT(idx);
+ __sync_child(p)[idx] = child;
+}
+
+static noinline int __sync_set(struct i915_syncmap **root, u64 id, u32 seqno)
+{
+ struct i915_syncmap *p = *root;
+ unsigned int idx;
+
+ if (!p) {
+ p = __sync_alloc_leaf(NULL, id);
+ if (unlikely(!p))
+ return -ENOMEM;
+
+ goto found;
+ }
+
+ /* Caller handled the likely cached case */
+ GEM_BUG_ON(__sync_leaf_prefix(p, id) == p->prefix);
+
+ /* Climb back up the tree until we find a common prefix */
+ do {
+ if (!p->parent)
+ break;
+
+ p = p->parent;
+
+ if (__sync_branch_prefix(p, id) == p->prefix)
+ break;
+ } while (1);
+
+ /*
+ * No shortcut, we have to descend the tree to find the right layer
+ * containing this fence.
+ *
+ * Each layer in the tree holds 16 (KSYNCMAP) pointers, either fences
+ * or lower layers. Leaf nodes (height = 0) contain the fences, all
+ * other nodes (height > 0) are internal layers that point to a lower
+ * node. Each internal layer has at least 2 descendents.
+ *
+ * Starting at the top, we check whether the current prefix matches. If
+ * it doesn't, we have gone past our target and need to insert a join
+ * into the tree, and a new leaf node for the target as a descendent
+ * of the join, as well as the original layer.
+ *
+ * The matching prefix means we are still following the right branch
+ * of the tree. If it has height 0, we have found our leaf and just
+ * need to replace the fence slot with ourselves. If the height is
+ * not zero, our slot contains the next layer in the tree (unless
+ * it is empty, in which case we can add ourselves as a new leaf).
+ * As descend the tree the prefix grows (and height decreases).
+ */
+ do {
+ struct i915_syncmap *next;
+
+ if (__sync_branch_prefix(p, id) != p->prefix) {
+ unsigned int above;
+
+ /* Insert a join above the current layer */
+ next = kzalloc(sizeof(*next) + KSYNCMAP * sizeof(next),
+ GFP_KERNEL);
+ if (unlikely(!next))
+ return -ENOMEM;
+
+ /* Compute the height at which these two diverge */
+ above = fls64(__sync_branch_prefix(p, id) ^ p->prefix);
+ above = round_up(above, SHIFT);
+ next->height = above + p->height;
+ next->prefix = __sync_branch_prefix(next, id);
+
+ /* Insert the join into the parent */
+ if (p->parent) {
+ idx = __sync_branch_idx(p->parent, id);
+ __sync_child(p->parent)[idx] = next;
+ GEM_BUG_ON(!(p->parent->bitmap & BIT(idx)));
+ }
+ next->parent = p->parent;
+
+ /* Compute the idx of the other branch, not our id! */
+ idx = p->prefix >> (above - SHIFT) & MASK;
+ __sync_set_child(next, idx, p);
+ p->parent = next;
+
+ /* Ascend to the join */
+ p = next;
+ } else {
+ if (!p->height)
+ break;
+ }
+
+ /* Descend into the next layer */
+ GEM_BUG_ON(!p->height);
+ idx = __sync_branch_idx(p, id);
+ next = __sync_child(p)[idx];
+ if (!next) {
+ next = __sync_alloc_leaf(p, id);
+ if (unlikely(!next))
+ return -ENOMEM;
+
+ __sync_set_child(p, idx, next);
+ p = next;
+ break;
+ }
+
+ p = next;
+ } while (1);
+
+found:
+ GEM_BUG_ON(p->prefix != __sync_leaf_prefix(p, id));
+ __sync_set_seqno(p, id, seqno);
+ *root = p;
+ return 0;
+}
+
+/**
+ * i915_syncmap_set -- mark the most recent syncpoint between contexts
+ * @root - pointer to the #i915_syncmap
+ * @id - the context id (other timeline) we have synchronised to
+ * @seqno - the sequence number along the other timeline
+ *
+ * When we synchronise this @root timeline with another (@id), we also know
+ * that we have synchronized with all previous seqno along that timeline. If
+ * we then have a request to synchronise with the same seqno or older, we can
+ * omit it, see i915_syncmap_is_later()
+ *
+ * Returns 0 on success, or a negative error code.
+ */
+int i915_syncmap_set(struct i915_syncmap **root, u64 id, u32 seqno)
+{
+ struct i915_syncmap *p = *root;
+
+ /*
+ * We expect to be called in sequence following is_later(id), which
+ * should have preloaded the root for us.
+ */
+ if (likely(p && __sync_leaf_prefix(p, id) == p->prefix)) {
+ __sync_set_seqno(p, id, seqno);
+ return 0;
+ }
+
+ return __sync_set(root, id, seqno);
+}
+
+static void __sync_free(struct i915_syncmap *p)
+{
+ if (p->height) {
+ unsigned int i;
+
+ while ((i = ffs(p->bitmap))) {
+ p->bitmap &= ~0u << i;
+ __sync_free(__sync_child(p)[i - 1]);
+ }
+ }
+
+ kfree(p);
+}
+
+/**
+ * i915_syncmap_free -- free all memory associated with the syncmap
+ * @root - pointer to the #i915_syncmap
+ *
+ * Either when the timeline is to be freed and we no longer need the sync
+ * point tracking, or when the fences are all known to be signaled and the
+ * sync point tracking is redundant, we can free the #i915_syncmap to recover
+ * its allocations.
+ *
+ * Will reinitialise the @root pointer so that the #i915_syncmap is ready for
+ * reuse.
+ */
+void i915_syncmap_free(struct i915_syncmap **root)
+{
+ struct i915_syncmap *p;
+
+ p = *root;
+ if (!p)
+ return;
+
+ while (p->parent)
+ p = p->parent;
+
+ __sync_free(p);
+ *root = NULL;
+}
+
+#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
+#include "selftests/i915_syncmap.c"
+#endif
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __I915_SYNCMAP_H__
+#define __I915_SYNCMAP_H__
+
+#include <linux/types.h>
+
+struct i915_syncmap;
+#define KSYNCMAP 16 /* radix of the tree, how many slots in each layer */
+
+void i915_syncmap_init(struct i915_syncmap **root);
+int i915_syncmap_set(struct i915_syncmap **root, u64 id, u32 seqno);
+bool i915_syncmap_is_later(struct i915_syncmap **root, u64 id, u32 seqno);
+void i915_syncmap_free(struct i915_syncmap **root);
+
+#endif /* __I915_SYNCMAP_H__ */
struct drm_i915_private *dev_priv = kdev_minor_to_i915(kdev);
struct drm_device *dev = &dev_priv->drm;
struct i915_gem_context *ctx;
- u32 *temp = NULL; /* Just here to make handling failures easy */
int slice = (int)(uintptr_t)attr->private;
+ u32 **remap_info;
int ret;
- if (!HAS_HW_CONTEXTS(dev_priv))
- return -ENXIO;
-
ret = l3_access_valid(dev_priv, offset);
if (ret)
return ret;
if (ret)
return ret;
- if (!dev_priv->l3_parity.remap_info[slice]) {
- temp = kzalloc(GEN7_L3LOG_SIZE, GFP_KERNEL);
- if (!temp) {
- mutex_unlock(&dev->struct_mutex);
- return -ENOMEM;
+ remap_info = &dev_priv->l3_parity.remap_info[slice];
+ if (!*remap_info) {
+ *remap_info = kzalloc(GEN7_L3LOG_SIZE, GFP_KERNEL);
+ if (!*remap_info) {
+ ret = -ENOMEM;
+ goto out;
}
}
* aren't propagated. Since I cannot find a stable way to reset the GPU
* at this point it is left as a TODO.
*/
- if (temp)
- dev_priv->l3_parity.remap_info[slice] = temp;
-
- memcpy(dev_priv->l3_parity.remap_info[slice] + (offset/4), buf, count);
+ memcpy(*remap_info + (offset/4), buf, count);
/* NB: We defer the remapping until we switch to the context */
list_for_each_entry(ctx, &dev_priv->context_list, link)
ctx->remap_slice |= (1<<slice);
+ ret = count;
+
+out:
mutex_unlock(&dev->struct_mutex);
- return count;
+ return ret;
}
static struct bin_attribute dpf_attrs = {
__entry->frame[PIPE_C], __entry->scanline[PIPE_C])
);
+TRACE_EVENT(g4x_wm,
+ TP_PROTO(struct intel_crtc *crtc, const struct g4x_wm_values *wm),
+ TP_ARGS(crtc, wm),
+
+ TP_STRUCT__entry(
+ __field(enum pipe, pipe)
+ __field(u32, frame)
+ __field(u32, scanline)
+ __field(u16, primary)
+ __field(u16, sprite)
+ __field(u16, cursor)
+ __field(u16, sr_plane)
+ __field(u16, sr_cursor)
+ __field(u16, sr_fbc)
+ __field(u16, hpll_plane)
+ __field(u16, hpll_cursor)
+ __field(u16, hpll_fbc)
+ __field(bool, cxsr)
+ __field(bool, hpll)
+ __field(bool, fbc)
+ ),
+
+ TP_fast_assign(
+ __entry->pipe = crtc->pipe;
+ __entry->frame = crtc->base.dev->driver->get_vblank_counter(crtc->base.dev,
+ crtc->pipe);
+ __entry->scanline = intel_get_crtc_scanline(crtc);
+ __entry->primary = wm->pipe[crtc->pipe].plane[PLANE_PRIMARY];
+ __entry->sprite = wm->pipe[crtc->pipe].plane[PLANE_SPRITE0];
+ __entry->cursor = wm->pipe[crtc->pipe].plane[PLANE_CURSOR];
+ __entry->sr_plane = wm->sr.plane;
+ __entry->sr_cursor = wm->sr.cursor;
+ __entry->sr_fbc = wm->sr.fbc;
+ __entry->hpll_plane = wm->hpll.plane;
+ __entry->hpll_cursor = wm->hpll.cursor;
+ __entry->hpll_fbc = wm->hpll.fbc;
+ __entry->cxsr = wm->cxsr;
+ __entry->hpll = wm->hpll_en;
+ __entry->fbc = wm->fbc_en;
+ ),
+
+ TP_printk("pipe %c, frame=%u, scanline=%u, wm %d/%d/%d, sr %s/%d/%d/%d, hpll %s/%d/%d/%d, fbc %s",
+ pipe_name(__entry->pipe), __entry->frame, __entry->scanline,
+ __entry->primary, __entry->sprite, __entry->cursor,
+ yesno(__entry->cxsr), __entry->sr_plane, __entry->sr_cursor, __entry->sr_fbc,
+ yesno(__entry->hpll), __entry->hpll_plane, __entry->hpll_cursor, __entry->hpll_fbc,
+ yesno(__entry->fbc))
+);
+
TRACE_EVENT(vlv_wm,
TP_PROTO(struct intel_crtc *crtc, const struct vlv_wm_values *wm),
TP_ARGS(crtc, wm),
#define overflows_type(x, T) \
(sizeof(x) > sizeof(T) && (x) >> (sizeof(T) * BITS_PER_BYTE))
-#define ptr_mask_bits(ptr) ({ \
+#define ptr_mask_bits(ptr, n) ({ \
unsigned long __v = (unsigned long)(ptr); \
- (typeof(ptr))(__v & PAGE_MASK); \
+ (typeof(ptr))(__v & -BIT(n)); \
})
-#define ptr_unpack_bits(ptr, bits) ({ \
+#define ptr_unmask_bits(ptr, n) ((unsigned long)(ptr) & (BIT(n) - 1))
+
+#define ptr_unpack_bits(ptr, bits, n) ({ \
unsigned long __v = (unsigned long)(ptr); \
- (bits) = __v & ~PAGE_MASK; \
- (typeof(ptr))(__v & PAGE_MASK); \
+ *(bits) = __v & (BIT(n) - 1); \
+ (typeof(ptr))(__v & -BIT(n)); \
})
-#define ptr_pack_bits(ptr, bits) \
+#define ptr_pack_bits(ptr, bits, n) \
((typeof(ptr))((unsigned long)(ptr) | (bits)))
+#define page_mask_bits(ptr) ptr_mask_bits(ptr, PAGE_SHIFT)
+#define page_unmask_bits(ptr) ptr_unmask_bits(ptr, PAGE_SHIFT)
+#define page_pack_bits(ptr, bits) ptr_pack_bits(ptr, bits, PAGE_SHIFT)
+#define page_unpack_bits(ptr, bits) ptr_unpack_bits(ptr, bits, PAGE_SHIFT)
+
#define ptr_offset(ptr, member) offsetof(typeof(*(ptr)), member)
#define fetch_and_zero(ptr) ({ \
__T; \
})
+#define __mask_next_bit(mask) ({ \
+ int __idx = ffs(mask) - 1; \
+ mask &= ~BIT(__idx); \
+ __idx; \
+})
+
+#include <linux/list.h>
+
+static inline void __list_del_many(struct list_head *head,
+ struct list_head *first)
+{
+ first->prev = head;
+ WRITE_ONCE(head->next, first);
+}
+
#endif /* !__I915_UTILS_H */
intel_plane_destroy_state(struct drm_plane *plane,
struct drm_plane_state *state)
{
- struct i915_vma *vma;
-
- vma = fetch_and_zero(&to_intel_plane_state(state)->vma);
-
- /*
- * FIXME: Normally intel_cleanup_plane_fb handles destruction of vma.
- * We currently don't clear all planes during driver unload, so we have
- * to be able to unpin vma here for now.
- *
- * Normally this can only happen during unload when kmscon is disabled
- * and userspace doesn't attempt to set a framebuffer at all.
- */
- if (vma) {
- mutex_lock(&plane->dev->struct_mutex);
- intel_unpin_fb_vma(vma);
- mutex_unlock(&plane->dev->struct_mutex);
- }
+ WARN_ON(to_intel_plane_state(state)->vma);
drm_atomic_helper_plane_destroy_state(plane, state);
}
}
intel_state->base.visible = false;
- ret = intel_plane->check_plane(plane, crtc_state, intel_state);
+ ret = intel_plane->check_plane(intel_plane, crtc_state, intel_state);
if (ret)
return ret;
trace_intel_update_plane(plane,
to_intel_crtc(crtc));
- intel_plane->update_plane(plane,
+ intel_plane->update_plane(intel_plane,
to_intel_crtc_state(crtc->state),
intel_state);
} else {
trace_intel_disable_plane(plane,
to_intel_crtc(crtc));
- intel_plane->disable_plane(plane, crtc);
+ intel_plane->disable_plane(intel_plane, to_intel_crtc(crtc));
}
}
(int) port, (int) pipe);
}
- switch (intel_encoder->type) {
- case INTEL_OUTPUT_HDMI:
- intel_lpe_audio_notify(dev_priv, connector->eld, port, pipe,
- crtc_state->port_clock,
- false, 0);
- break;
- case INTEL_OUTPUT_DP:
- intel_lpe_audio_notify(dev_priv, connector->eld, port, pipe,
- adjusted_mode->crtc_clock,
- true, crtc_state->port_clock);
- break;
- default:
- break;
- }
+ intel_lpe_audio_notify(dev_priv, pipe, port, connector->eld,
+ crtc_state->port_clock,
+ intel_encoder->type == INTEL_OUTPUT_DP);
}
/**
(int) port, (int) pipe);
}
- intel_lpe_audio_notify(dev_priv, NULL, port, pipe, 0, false, 0);
+ intel_lpe_audio_notify(dev_priv, pipe, port, NULL, 0, false);
}
/**
static noinline void missed_breadcrumb(struct intel_engine_cs *engine)
{
- DRM_DEBUG_DRIVER("%s missed breadcrumb at %pF, irq posted? %s\n",
+ DRM_DEBUG_DRIVER("%s missed breadcrumb at %pF, irq posted? %s, current seqno=%x, last=%x\n",
engine->name, __builtin_return_address(0),
yesno(test_bit(ENGINE_IRQ_BREADCRUMB,
- &engine->irq_posted)));
+ &engine->irq_posted)),
+ intel_engine_get_seqno(engine),
+ intel_engine_last_submit(engine));
set_bit(engine->id, &engine->i915->gpu_error.missed_irq_rings);
}
return 0;
}
-void intel_engine_enable_signaling(struct drm_i915_gem_request *request)
+void intel_engine_enable_signaling(struct drm_i915_gem_request *request,
+ bool wakeup)
{
struct intel_engine_cs *engine = request->engine;
struct intel_breadcrumbs *b = &engine->breadcrumbs;
struct rb_node *parent, **p;
- bool first, wakeup;
+ bool first;
u32 seqno;
/* Note that we may be called from an interrupt handler on another
* If we are the oldest waiter, enable the irq (after which we
* must double check that the seqno did not complete).
*/
- wakeup = __intel_engine_add_wait(engine, &request->signaling.wait);
+ wakeup &= __intel_engine_add_wait(engine, &request->signaling.wait);
/* Now insert ourselves into the retirement ordered list of signals
* on this engine. We track the oldest seqno as that will be the
static int glk_calc_cdclk(int max_pixclk)
{
- if (max_pixclk > 2 * 158400)
+ /*
+ * FIXME: Avoid using a pixel clock that is more than 99% of the cdclk
+ * as a temporary workaround. Use a higher cdclk instead. (Note that
+ * intel_compute_max_dotclk() limits the max pixel clock to 99% of max
+ * cdclk.)
+ */
+ if (max_pixclk > DIV_ROUND_UP(2 * 158400 * 99, 100))
return 316800;
- else if (max_pixclk > 2 * 79200)
+ else if (max_pixclk > DIV_ROUND_UP(2 * 79200 * 99, 100))
return 158400;
else
return 79200;
int max_cdclk_freq = dev_priv->max_cdclk_freq;
if (IS_GEMINILAKE(dev_priv))
- return 2 * max_cdclk_freq;
+ /*
+ * FIXME: Limiting to 99% as a temporary workaround. See
+ * glk_calc_cdclk() for details.
+ */
+ return 2 * max_cdclk_freq * 99 / 100;
else if (INTEL_INFO(dev_priv)->gen >= 9 ||
IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv))
return max_cdclk_freq;
case CLKCFG_FSB_800:
return 200000;
case CLKCFG_FSB_1067:
+ case CLKCFG_FSB_1067_ALT:
return 266667;
case CLKCFG_FSB_1333:
+ case CLKCFG_FSB_1333_ALT:
return 333333;
- /* these two are just a guess; one of them might be right */
- case CLKCFG_FSB_1600:
- case CLKCFG_FSB_1600_ALT:
- return 400000;
default:
return 133333;
}
return ret;
}
-static int intel_crt_set_property(struct drm_connector *connector,
- struct drm_property *property,
- uint64_t value)
-{
- return 0;
-}
-
void intel_crt_reset(struct drm_encoder *encoder)
{
struct drm_i915_private *dev_priv = to_i915(encoder->dev);
.late_register = intel_connector_register,
.early_unregister = intel_connector_unregister,
.destroy = intel_crt_destroy,
- .set_property = intel_crt_set_property,
+ .set_property = drm_atomic_helper_connector_set_property,
.atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
.atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
- .atomic_get_property = intel_connector_atomic_get_property,
};
static const struct drm_connector_helper_funcs intel_crt_connector_helper_funcs = {
} else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
for_each_pipe(dev_priv, pipe)
info->num_sprites[pipe] = 2;
- } else if (INTEL_GEN(dev_priv) >= 5) {
+ } else if (INTEL_GEN(dev_priv) >= 5 || IS_G4X(dev_priv)) {
for_each_pipe(dev_priv, pipe)
info->num_sprites[pipe] = 1;
}
I915_STATE_WARN(val & SPRITE_ENABLE,
"sprite %c assertion failure, should be off on pipe %c but is still active\n",
plane_name(pipe), pipe_name(pipe));
- } else if (INTEL_GEN(dev_priv) >= 5) {
+ } else if (INTEL_GEN(dev_priv) >= 5 || IS_G4X(dev_priv)) {
u32 val = I915_READ(DVSCNTR(pipe));
I915_STATE_WARN(val & DVS_ENABLE,
"sprite %c assertion failure, should be off on pipe %c but is still active\n",
}
}
+static unsigned int intel_cursor_alignment(const struct drm_i915_private *dev_priv)
+{
+ if (IS_I830(dev_priv))
+ return 16 * 1024;
+ else if (IS_I85X(dev_priv))
+ return 256;
+ else if (IS_I845G(dev_priv) || IS_I865G(dev_priv))
+ return 32;
+ else
+ return 4 * 1024;
+}
+
static unsigned int intel_linear_alignment(const struct drm_i915_private *dev_priv)
{
if (INTEL_INFO(dev_priv)->gen >= 9)
const struct intel_plane_state *state,
int plane)
{
- const struct drm_i915_private *dev_priv = to_i915(state->base.plane->dev);
+ struct intel_plane *intel_plane = to_intel_plane(state->base.plane);
+ struct drm_i915_private *dev_priv = to_i915(intel_plane->base.dev);
const struct drm_framebuffer *fb = state->base.fb;
unsigned int rotation = state->base.rotation;
int pitch = intel_fb_pitch(fb, plane, rotation);
- u32 alignment = intel_surf_alignment(fb, plane);
+ u32 alignment;
+
+ if (intel_plane->id == PLANE_CURSOR)
+ alignment = intel_cursor_alignment(dev_priv);
+ else
+ alignment = intel_surf_alignment(fb, plane);
return _intel_compute_tile_offset(dev_priv, x, y, fb, plane, pitch,
rotation, alignment);
false);
intel_pre_disable_primary_noatomic(&intel_crtc->base);
trace_intel_disable_plane(primary, intel_crtc);
- intel_plane->disable_plane(primary, &intel_crtc->base);
+ intel_plane->disable_plane(intel_plane, intel_crtc);
return;
if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv))
dspcntr |= DISPPLANE_PIPE_CSC_ENABLE;
- if (INTEL_GEN(dev_priv) < 4) {
- if (crtc->pipe == PIPE_B)
- dspcntr |= DISPPLANE_SEL_PIPE_B;
- }
+ if (INTEL_GEN(dev_priv) < 4)
+ dspcntr |= DISPPLANE_SEL_PIPE(crtc->pipe);
switch (fb->format->format) {
case DRM_FORMAT_C8:
return 0;
}
-static void i9xx_update_primary_plane(struct drm_plane *primary,
+static void i9xx_update_primary_plane(struct intel_plane *primary,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_i915_private *dev_priv = to_i915(primary->dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc_state->base.crtc);
- struct drm_framebuffer *fb = plane_state->base.fb;
- int plane = intel_crtc->plane;
+ struct drm_i915_private *dev_priv = to_i915(primary->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum plane plane = primary->plane;
u32 linear_offset;
u32 dspcntr = plane_state->ctl;
i915_reg_t reg = DSPCNTR(plane);
linear_offset = intel_fb_xy_to_linear(x, y, plane_state, 0);
if (INTEL_GEN(dev_priv) >= 4)
- intel_crtc->dspaddr_offset = plane_state->main.offset;
+ crtc->dspaddr_offset = plane_state->main.offset;
else
- intel_crtc->dspaddr_offset = linear_offset;
+ crtc->dspaddr_offset = linear_offset;
- intel_crtc->adjusted_x = x;
- intel_crtc->adjusted_y = y;
+ crtc->adjusted_x = x;
+ crtc->adjusted_y = y;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv)) {
I915_WRITE_FW(DSPSURF(plane),
intel_plane_ggtt_offset(plane_state) +
- intel_crtc->dspaddr_offset);
+ crtc->dspaddr_offset);
I915_WRITE_FW(DSPOFFSET(plane), (y << 16) | x);
} else if (INTEL_GEN(dev_priv) >= 4) {
I915_WRITE_FW(DSPSURF(plane),
intel_plane_ggtt_offset(plane_state) +
- intel_crtc->dspaddr_offset);
+ crtc->dspaddr_offset);
I915_WRITE_FW(DSPTILEOFF(plane), (y << 16) | x);
I915_WRITE_FW(DSPLINOFF(plane), linear_offset);
} else {
I915_WRITE_FW(DSPADDR(plane),
intel_plane_ggtt_offset(plane_state) +
- intel_crtc->dspaddr_offset);
+ crtc->dspaddr_offset);
}
POSTING_READ_FW(reg);
spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
}
-static void i9xx_disable_primary_plane(struct drm_plane *primary,
- struct drm_crtc *crtc)
+static void i9xx_disable_primary_plane(struct intel_plane *primary,
+ struct intel_crtc *crtc)
{
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- int plane = intel_crtc->plane;
+ struct drm_i915_private *dev_priv = to_i915(primary->base.dev);
+ enum plane plane = primary->plane;
unsigned long irqflags;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
return plane_ctl;
}
-static void skylake_update_primary_plane(struct drm_plane *plane,
+static void skylake_update_primary_plane(struct intel_plane *plane,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = plane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc_state->base.crtc);
- struct drm_framebuffer *fb = plane_state->base.fb;
- enum plane_id plane_id = to_intel_plane(plane)->id;
- enum pipe pipe = to_intel_plane(plane)->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum plane_id plane_id = plane->id;
+ enum pipe pipe = plane->pipe;
u32 plane_ctl = plane_state->ctl;
unsigned int rotation = plane_state->base.rotation;
u32 stride = skl_plane_stride(fb, 0, rotation);
dst_w--;
dst_h--;
- intel_crtc->dspaddr_offset = surf_addr;
+ crtc->dspaddr_offset = surf_addr;
- intel_crtc->adjusted_x = src_x;
- intel_crtc->adjusted_y = src_y;
+ crtc->adjusted_x = src_x;
+ crtc->adjusted_y = src_y;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
}
-static void skylake_disable_primary_plane(struct drm_plane *primary,
- struct drm_crtc *crtc)
+static void skylake_disable_primary_plane(struct intel_plane *primary,
+ struct intel_crtc *crtc)
{
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- enum plane_id plane_id = to_intel_plane(primary)->id;
- enum pipe pipe = to_intel_plane(primary)->pipe;
+ struct drm_i915_private *dev_priv = to_i915(primary->base.dev);
+ enum plane_id plane_id = primary->id;
+ enum pipe pipe = primary->pipe;
unsigned long irqflags;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
trace_intel_update_plane(&plane->base,
to_intel_crtc(crtc));
- plane->update_plane(&plane->base,
+ plane->update_plane(plane,
to_intel_crtc_state(crtc->state),
plane_state);
}
{
if (intel_crtc->overlay) {
struct drm_device *dev = intel_crtc->base.dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
mutex_lock(&dev->struct_mutex);
- dev_priv->mm.interruptible = false;
(void) intel_overlay_switch_off(intel_crtc->overlay);
- dev_priv->mm.interruptible = true;
mutex_unlock(&dev->struct_mutex);
}
intel_crtc_dpms_overlay_disable(intel_crtc);
drm_for_each_plane_mask(p, dev, plane_mask)
- to_intel_plane(p)->disable_plane(p, crtc);
+ to_intel_plane(p)->disable_plane(to_intel_plane(p), intel_crtc);
/*
* FIXME: Once we grow proper nuclear flip support out of this we need
static void i9xx_crtc_enable(struct intel_crtc_state *pipe_config,
struct drm_atomic_state *old_state)
{
+ struct intel_atomic_state *old_intel_state =
+ to_intel_atomic_state(old_state);
struct drm_crtc *crtc = pipe_config->base.crtc;
struct drm_device *dev = crtc->dev;
struct drm_i915_private *dev_priv = to_i915(dev);
intel_color_load_luts(&pipe_config->base);
- intel_update_watermarks(intel_crtc);
+ if (dev_priv->display.initial_watermarks != NULL)
+ dev_priv->display.initial_watermarks(old_intel_state,
+ intel_crtc->config);
+ else
+ intel_update_watermarks(intel_crtc);
intel_enable_pipe(intel_crtc);
assert_vblank_disabled(crtc);
/* Cross check the actual hw state with our own modeset state tracking (and it's
* internal consistency). */
-static void intel_connector_verify_state(struct intel_connector *connector)
+static void intel_connector_verify_state(struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
{
- struct drm_crtc *crtc = connector->base.state->crtc;
+ struct intel_connector *connector = to_intel_connector(conn_state->connector);
DRM_DEBUG_KMS("[CONNECTOR:%d:%s]\n",
connector->base.base.id,
if (connector->get_hw_state(connector)) {
struct intel_encoder *encoder = connector->encoder;
- struct drm_connector_state *conn_state = connector->base.state;
- I915_STATE_WARN(!crtc,
+ I915_STATE_WARN(!crtc_state,
"connector enabled without attached crtc\n");
- if (!crtc)
+ if (!crtc_state)
return;
- I915_STATE_WARN(!crtc->state->active,
+ I915_STATE_WARN(!crtc_state->active,
"connector is active, but attached crtc isn't\n");
if (!encoder || encoder->type == INTEL_OUTPUT_DP_MST)
I915_STATE_WARN(conn_state->crtc != encoder->base.crtc,
"attached encoder crtc differs from connector crtc\n");
} else {
- I915_STATE_WARN(crtc && crtc->state->active,
+ I915_STATE_WARN(crtc_state && crtc_state->active,
"attached crtc is active, but connector isn't\n");
- I915_STATE_WARN(!crtc && connector->base.state->best_encoder,
+ I915_STATE_WARN(!crtc_state && conn_state->best_encoder,
"best encoder set without crtc!\n");
}
}
vlv_dpio_write(dev_priv, pipe, VLV_PLL_DW9(1), reg_val);
reg_val = vlv_dpio_read(dev_priv, pipe, VLV_REF_DW13);
- reg_val &= 0x8cffffff;
- reg_val = 0x8c000000;
+ reg_val &= 0x00ffffff;
+ reg_val |= 0x8c000000;
vlv_dpio_write(dev_priv, pipe, VLV_REF_DW13, reg_val);
reg_val = vlv_dpio_read(dev_priv, pipe, VLV_PLL_DW9(1));
{
struct drm_device *dev = crtc->base.dev;
struct drm_i915_private *dev_priv = to_i915(dev);
- struct dpll reduced_clock;
- bool has_reduced_clock = false;
- struct intel_shared_dpll *pll;
const struct intel_limit *limit;
int refclk = 120000;
return -EINVAL;
}
- ironlake_compute_dpll(crtc, crtc_state,
- has_reduced_clock ? &reduced_clock : NULL);
+ ironlake_compute_dpll(crtc, crtc_state, NULL);
- pll = intel_get_shared_dpll(crtc, crtc_state, NULL);
- if (pll == NULL) {
+ if (!intel_get_shared_dpll(crtc, crtc_state, NULL)) {
DRM_DEBUG_DRIVER("failed to find PLL for pipe %c\n",
pipe_name(crtc->pipe));
return -EINVAL;
}
- if (intel_crtc_has_type(crtc_state, INTEL_OUTPUT_LVDS) &&
- has_reduced_clock)
- crtc->lowfreq_avail = true;
-
return 0;
}
return active;
}
+static u32 intel_cursor_base(const struct intel_plane_state *plane_state)
+{
+ struct drm_i915_private *dev_priv =
+ to_i915(plane_state->base.plane->dev);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ const struct drm_i915_gem_object *obj = intel_fb_obj(fb);
+ u32 base;
+
+ if (INTEL_INFO(dev_priv)->cursor_needs_physical)
+ base = obj->phys_handle->busaddr;
+ else
+ base = intel_plane_ggtt_offset(plane_state);
+
+ base += plane_state->main.offset;
+
+ /* ILK+ do this automagically */
+ if (HAS_GMCH_DISPLAY(dev_priv) &&
+ plane_state->base.rotation & DRM_MODE_ROTATE_180)
+ base += (plane_state->base.crtc_h *
+ plane_state->base.crtc_w - 1) * fb->format->cpp[0];
+
+ return base;
+}
+
+static u32 intel_cursor_position(const struct intel_plane_state *plane_state)
+{
+ int x = plane_state->base.crtc_x;
+ int y = plane_state->base.crtc_y;
+ u32 pos = 0;
+
+ if (x < 0) {
+ pos |= CURSOR_POS_SIGN << CURSOR_X_SHIFT;
+ x = -x;
+ }
+ pos |= x << CURSOR_X_SHIFT;
+
+ if (y < 0) {
+ pos |= CURSOR_POS_SIGN << CURSOR_Y_SHIFT;
+ y = -y;
+ }
+ pos |= y << CURSOR_Y_SHIFT;
+
+ return pos;
+}
+
+static bool intel_cursor_size_ok(const struct intel_plane_state *plane_state)
+{
+ const struct drm_mode_config *config =
+ &plane_state->base.plane->dev->mode_config;
+ int width = plane_state->base.crtc_w;
+ int height = plane_state->base.crtc_h;
+
+ return width > 0 && width <= config->cursor_width &&
+ height > 0 && height <= config->cursor_height;
+}
+
+static int intel_check_cursor(struct intel_crtc_state *crtc_state,
+ struct intel_plane_state *plane_state)
+{
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ int src_x, src_y;
+ u32 offset;
+ int ret;
+
+ ret = drm_plane_helper_check_state(&plane_state->base,
+ &plane_state->clip,
+ DRM_PLANE_HELPER_NO_SCALING,
+ DRM_PLANE_HELPER_NO_SCALING,
+ true, true);
+ if (ret)
+ return ret;
+
+ if (!fb)
+ return 0;
+
+ if (fb->modifier != DRM_FORMAT_MOD_LINEAR) {
+ DRM_DEBUG_KMS("cursor cannot be tiled\n");
+ return -EINVAL;
+ }
+
+ src_x = plane_state->base.src_x >> 16;
+ src_y = plane_state->base.src_y >> 16;
+
+ intel_add_fb_offsets(&src_x, &src_y, plane_state, 0);
+ offset = intel_compute_tile_offset(&src_x, &src_y, plane_state, 0);
+
+ if (src_x != 0 || src_y != 0) {
+ DRM_DEBUG_KMS("Arbitrary cursor panning not supported\n");
+ return -EINVAL;
+ }
+
+ plane_state->main.offset = offset;
+
+ return 0;
+}
+
static u32 i845_cursor_ctl(const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- unsigned int width = plane_state->base.crtc_w;
- unsigned int stride = roundup_pow_of_two(width) * 4;
+ const struct drm_framebuffer *fb = plane_state->base.fb;
- switch (stride) {
- default:
- WARN_ONCE(1, "Invalid cursor width/stride, width=%u, stride=%u\n",
- width, stride);
- stride = 256;
- /* fallthrough */
+ return CURSOR_ENABLE |
+ CURSOR_GAMMA_ENABLE |
+ CURSOR_FORMAT_ARGB |
+ CURSOR_STRIDE(fb->pitches[0]);
+}
+
+static bool i845_cursor_size_ok(const struct intel_plane_state *plane_state)
+{
+ int width = plane_state->base.crtc_w;
+
+ /*
+ * 845g/865g are only limited by the width of their cursors,
+ * the height is arbitrary up to the precision of the register.
+ */
+ return intel_cursor_size_ok(plane_state) && IS_ALIGNED(width, 64);
+}
+
+static int i845_check_cursor(struct intel_plane *plane,
+ struct intel_crtc_state *crtc_state,
+ struct intel_plane_state *plane_state)
+{
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ int ret;
+
+ ret = intel_check_cursor(crtc_state, plane_state);
+ if (ret)
+ return ret;
+
+ /* if we want to turn off the cursor ignore width and height */
+ if (!fb)
+ return 0;
+
+ /* Check for which cursor types we support */
+ if (!i845_cursor_size_ok(plane_state)) {
+ DRM_DEBUG("Cursor dimension %dx%d not supported\n",
+ plane_state->base.crtc_w,
+ plane_state->base.crtc_h);
+ return -EINVAL;
+ }
+
+ switch (fb->pitches[0]) {
case 256:
case 512:
case 1024:
case 2048:
break;
+ default:
+ DRM_DEBUG_KMS("Invalid cursor stride (%u)\n",
+ fb->pitches[0]);
+ return -EINVAL;
}
- return CURSOR_ENABLE |
- CURSOR_GAMMA_ENABLE |
- CURSOR_FORMAT_ARGB |
- CURSOR_STRIDE(stride);
+ plane_state->ctl = i845_cursor_ctl(crtc_state, plane_state);
+
+ return 0;
}
-static void i845_update_cursor(struct drm_crtc *crtc, u32 base,
+static void i845_update_cursor(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- uint32_t cntl = 0, size = 0;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ u32 cntl = 0, base = 0, pos = 0, size = 0;
+ unsigned long irqflags;
if (plane_state && plane_state->base.visible) {
unsigned int width = plane_state->base.crtc_w;
cntl = plane_state->ctl;
size = (height << 12) | width;
- }
- if (intel_crtc->cursor_cntl != 0 &&
- (intel_crtc->cursor_base != base ||
- intel_crtc->cursor_size != size ||
- intel_crtc->cursor_cntl != cntl)) {
- /* On these chipsets we can only modify the base/size/stride
- * whilst the cursor is disabled.
- */
- I915_WRITE_FW(CURCNTR(PIPE_A), 0);
- POSTING_READ_FW(CURCNTR(PIPE_A));
- intel_crtc->cursor_cntl = 0;
+ base = intel_cursor_base(plane_state);
+ pos = intel_cursor_position(plane_state);
}
- if (intel_crtc->cursor_base != base) {
- I915_WRITE_FW(CURBASE(PIPE_A), base);
- intel_crtc->cursor_base = base;
- }
+ spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
- if (intel_crtc->cursor_size != size) {
+ /* On these chipsets we can only modify the base/size/stride
+ * whilst the cursor is disabled.
+ */
+ if (plane->cursor.base != base ||
+ plane->cursor.size != size ||
+ plane->cursor.cntl != cntl) {
+ I915_WRITE_FW(CURCNTR(PIPE_A), 0);
+ I915_WRITE_FW(CURBASE(PIPE_A), base);
I915_WRITE_FW(CURSIZE, size);
- intel_crtc->cursor_size = size;
- }
-
- if (intel_crtc->cursor_cntl != cntl) {
+ I915_WRITE_FW(CURPOS(PIPE_A), pos);
I915_WRITE_FW(CURCNTR(PIPE_A), cntl);
- POSTING_READ_FW(CURCNTR(PIPE_A));
- intel_crtc->cursor_cntl = cntl;
+
+ plane->cursor.base = base;
+ plane->cursor.size = size;
+ plane->cursor.cntl = cntl;
+ } else {
+ I915_WRITE_FW(CURPOS(PIPE_A), pos);
}
+
+ POSTING_READ_FW(CURCNTR(PIPE_A));
+
+ spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+}
+
+static void i845_disable_cursor(struct intel_plane *plane,
+ struct intel_crtc *crtc)
+{
+ i845_update_cursor(plane, NULL, NULL);
}
static u32 i9xx_cursor_ctl(const struct intel_crtc_state *crtc_state,
struct drm_i915_private *dev_priv =
to_i915(plane_state->base.plane->dev);
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
- enum pipe pipe = crtc->pipe;
u32 cntl;
cntl = MCURSOR_GAMMA_ENABLE;
if (HAS_DDI(dev_priv))
cntl |= CURSOR_PIPE_CSC_ENABLE;
- cntl |= pipe << 28; /* Connect to correct pipe */
+ cntl |= MCURSOR_PIPE_SELECT(crtc->pipe);
switch (plane_state->base.crtc_w) {
case 64:
return cntl;
}
-static void i9xx_update_cursor(struct drm_crtc *crtc, u32 base,
- const struct intel_plane_state *plane_state)
+static bool i9xx_cursor_size_ok(const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- int pipe = intel_crtc->pipe;
- uint32_t cntl = 0;
+ struct drm_i915_private *dev_priv =
+ to_i915(plane_state->base.plane->dev);
+ int width = plane_state->base.crtc_w;
+ int height = plane_state->base.crtc_h;
- if (plane_state && plane_state->base.visible)
- cntl = plane_state->ctl;
+ if (!intel_cursor_size_ok(plane_state))
+ return false;
- if (intel_crtc->cursor_cntl != cntl) {
- I915_WRITE_FW(CURCNTR(pipe), cntl);
- POSTING_READ_FW(CURCNTR(pipe));
- intel_crtc->cursor_cntl = cntl;
+ /* Cursor width is limited to a few power-of-two sizes */
+ switch (width) {
+ case 256:
+ case 128:
+ case 64:
+ break;
+ default:
+ return false;
}
- /* and commit changes on next vblank */
- I915_WRITE_FW(CURBASE(pipe), base);
- POSTING_READ_FW(CURBASE(pipe));
+ /*
+ * IVB+ have CUR_FBC_CTL which allows an arbitrary cursor
+ * height from 8 lines up to the cursor width, when the
+ * cursor is not rotated. Everything else requires square
+ * cursors.
+ */
+ if (HAS_CUR_FBC(dev_priv) &&
+ plane_state->base.rotation & DRM_MODE_ROTATE_0) {
+ if (height < 8 || height > width)
+ return false;
+ } else {
+ if (height != width)
+ return false;
+ }
- intel_crtc->cursor_base = base;
+ return true;
}
-/* If no-part of the cursor is visible on the framebuffer, then the GPU may hang... */
-static void intel_crtc_update_cursor(struct drm_crtc *crtc,
- const struct intel_plane_state *plane_state)
+static int i9xx_check_cursor(struct intel_plane *plane,
+ struct intel_crtc_state *crtc_state,
+ struct intel_plane_state *plane_state)
{
- struct drm_device *dev = crtc->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- int pipe = intel_crtc->pipe;
- u32 base = intel_crtc->cursor_addr;
- unsigned long irqflags;
- u32 pos = 0;
-
- if (plane_state) {
- int x = plane_state->base.crtc_x;
- int y = plane_state->base.crtc_y;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum pipe pipe = plane->pipe;
+ int ret;
- if (x < 0) {
- pos |= CURSOR_POS_SIGN << CURSOR_X_SHIFT;
- x = -x;
- }
- pos |= x << CURSOR_X_SHIFT;
+ ret = intel_check_cursor(crtc_state, plane_state);
+ if (ret)
+ return ret;
- if (y < 0) {
- pos |= CURSOR_POS_SIGN << CURSOR_Y_SHIFT;
- y = -y;
- }
- pos |= y << CURSOR_Y_SHIFT;
+ /* if we want to turn off the cursor ignore width and height */
+ if (!fb)
+ return 0;
- /* ILK+ do this automagically */
- if (HAS_GMCH_DISPLAY(dev_priv) &&
- plane_state->base.rotation & DRM_MODE_ROTATE_180) {
- base += (plane_state->base.crtc_h *
- plane_state->base.crtc_w - 1) * 4;
- }
+ /* Check for which cursor types we support */
+ if (!i9xx_cursor_size_ok(plane_state)) {
+ DRM_DEBUG("Cursor dimension %dx%d not supported\n",
+ plane_state->base.crtc_w,
+ plane_state->base.crtc_h);
+ return -EINVAL;
}
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+ if (fb->pitches[0] != plane_state->base.crtc_w * fb->format->cpp[0]) {
+ DRM_DEBUG_KMS("Invalid cursor stride (%u) (cursor width %d)\n",
+ fb->pitches[0], plane_state->base.crtc_w);
+ return -EINVAL;
+ }
- I915_WRITE_FW(CURPOS(pipe), pos);
+ /*
+ * There's something wrong with the cursor on CHV pipe C.
+ * If it straddles the left edge of the screen then
+ * moving it away from the edge or disabling it often
+ * results in a pipe underrun, and often that can lead to
+ * dead pipe (constant underrun reported, and it scans
+ * out just a solid color). To recover from that, the
+ * display power well must be turned off and on again.
+ * Refuse the put the cursor into that compromised position.
+ */
+ if (IS_CHERRYVIEW(dev_priv) && pipe == PIPE_C &&
+ plane_state->base.visible && plane_state->base.crtc_x < 0) {
+ DRM_DEBUG_KMS("CHV cursor C not allowed to straddle the left screen edge\n");
+ return -EINVAL;
+ }
- if (IS_I845G(dev_priv) || IS_I865G(dev_priv))
- i845_update_cursor(crtc, base, plane_state);
- else
- i9xx_update_cursor(crtc, base, plane_state);
+ plane_state->ctl = i9xx_cursor_ctl(crtc_state, plane_state);
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+ return 0;
}
-static bool cursor_size_ok(struct drm_i915_private *dev_priv,
- uint32_t width, uint32_t height)
+static void i9xx_update_cursor(struct intel_plane *plane,
+ const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state)
{
- if (width == 0 || height == 0)
- return false;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ enum pipe pipe = plane->pipe;
+ u32 cntl = 0, base = 0, pos = 0, fbc_ctl = 0;
+ unsigned long irqflags;
- /*
- * 845g/865g are special in that they are only limited by
- * the width of their cursors, the height is arbitrary up to
- * the precision of the register. Everything else requires
- * square cursors, limited to a few power-of-two sizes.
- */
- if (IS_I845G(dev_priv) || IS_I865G(dev_priv)) {
- if ((width & 63) != 0)
- return false;
+ if (plane_state && plane_state->base.visible) {
+ cntl = plane_state->ctl;
- if (width > (IS_I845G(dev_priv) ? 64 : 512))
- return false;
+ if (plane_state->base.crtc_h != plane_state->base.crtc_w)
+ fbc_ctl = CUR_FBC_CTL_EN | (plane_state->base.crtc_h - 1);
- if (height > 1023)
- return false;
+ base = intel_cursor_base(plane_state);
+ pos = intel_cursor_position(plane_state);
+ }
+
+ spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
+
+ /*
+ * On some platforms writing CURCNTR first will also
+ * cause CURPOS to be armed by the CURBASE write.
+ * Without the CURCNTR write the CURPOS write would
+ * arm itself.
+ *
+ * CURCNTR and CUR_FBC_CTL are always
+ * armed by the CURBASE write only.
+ */
+ if (plane->cursor.base != base ||
+ plane->cursor.size != fbc_ctl ||
+ plane->cursor.cntl != cntl) {
+ I915_WRITE_FW(CURCNTR(pipe), cntl);
+ if (HAS_CUR_FBC(dev_priv))
+ I915_WRITE_FW(CUR_FBC_CTL(pipe), fbc_ctl);
+ I915_WRITE_FW(CURPOS(pipe), pos);
+ I915_WRITE_FW(CURBASE(pipe), base);
+
+ plane->cursor.base = base;
+ plane->cursor.size = fbc_ctl;
+ plane->cursor.cntl = cntl;
} else {
- switch (width | height) {
- case 256:
- case 128:
- if (IS_GEN2(dev_priv))
- return false;
- case 64:
- break;
- default:
- return false;
- }
+ I915_WRITE_FW(CURPOS(pipe), pos);
}
- return true;
+ POSTING_READ_FW(CURBASE(pipe));
+
+ spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
+}
+
+static void i9xx_disable_cursor(struct intel_plane *plane,
+ struct intel_crtc *crtc)
+{
+ i9xx_update_cursor(plane, NULL, NULL);
}
+
/* VESA 640x480x72Hz mode to set on the pipe */
static struct drm_display_mode load_detect_mode = {
DRM_MODE("640x480", DRM_MODE_TYPE_DEFAULT, 31500, 640, 664,
*/
if (!crtc) {
DRM_DEBUG_KMS("no pipe available for load-detect\n");
+ ret = -ENODEV;
goto fail;
}
DRM_DEBUG_KMS("reusing fbdev for load-detection framebuffer\n");
if (IS_ERR(fb)) {
DRM_DEBUG_KMS("failed to allocate framebuffer for load-detection\n");
+ ret = PTR_ERR(fb);
goto fail;
}
turn_off, turn_on, mode_changed);
if (turn_on) {
- if (INTEL_GEN(dev_priv) < 5)
+ if (INTEL_GEN(dev_priv) < 5 && !IS_G4X(dev_priv))
pipe_config->update_wm_pre = true;
/* must disable cxsr around plane enable/disable */
if (plane->id != PLANE_CURSOR)
pipe_config->disable_cxsr = true;
} else if (turn_off) {
- if (INTEL_GEN(dev_priv) < 5)
+ if (INTEL_GEN(dev_priv) < 5 && !IS_G4X(dev_priv))
pipe_config->update_wm_post = true;
/* must disable cxsr around plane enable/disable */
if (plane->id != PLANE_CURSOR)
pipe_config->disable_cxsr = true;
} else if (intel_wm_need_update(&plane->base, plane_state)) {
- if (INTEL_GEN(dev_priv) < 5) {
+ if (INTEL_GEN(dev_priv) < 5 && !IS_G4X(dev_priv)) {
/* FIXME bollocks */
pipe_config->update_wm_pre = true;
pipe_config->update_wm_post = true;
shared_dpll = crtc_state->shared_dpll;
dpll_hw_state = crtc_state->dpll_hw_state;
force_thru = crtc_state->pch_pfit.force_thru;
- if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
+ if (IS_G4X(dev_priv) ||
+ IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
wm_state = crtc_state->wm;
/* Keep base drm_crtc_state intact, only clear our extended struct */
crtc_state->shared_dpll = shared_dpll;
crtc_state->dpll_hw_state = dpll_hw_state;
crtc_state->pch_pfit.force_thru = force_thru;
- if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
+ if (IS_G4X(dev_priv) ||
+ IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
crtc_state->wm = wm_state;
}
* allocation. In that case since the ddb allocation will be updated
* once the plane becomes visible, we can skip this check
*/
- if (intel_crtc->cursor_addr) {
+ if (1) {
hw_plane_wm = &hw_wm.planes[PLANE_CURSOR];
sw_plane_wm = &sw_wm->planes[PLANE_CURSOR];
for_each_new_connector_in_state(state, connector, new_conn_state, i) {
struct drm_encoder *encoder = connector->encoder;
+ struct drm_crtc_state *crtc_state = NULL;
if (new_conn_state->crtc != crtc)
continue;
- intel_connector_verify_state(to_intel_connector(connector));
+ if (crtc)
+ crtc_state = drm_atomic_get_new_crtc_state(state, new_conn_state->crtc);
+
+ intel_connector_verify_state(crtc_state, new_conn_state);
I915_STATE_WARN(new_conn_state->best_encoder != encoder,
"connector's atomic encoder doesn't match legacy encoder\n");
intel_pipe_config_sanity_check(dev_priv, pipe_config);
- sw_config = to_intel_crtc_state(crtc->state);
+ sw_config = to_intel_crtc_state(new_crtc_state);
if (!intel_pipe_config_compare(dev_priv, sw_config,
pipe_config, false)) {
I915_STATE_WARN(1, "pipe state doesn't match!\n");
if (obj) {
if (plane->type == DRM_PLANE_TYPE_CURSOR &&
INTEL_INFO(dev_priv)->cursor_needs_physical) {
- const int align = IS_I830(dev_priv) ? 16 * 1024 : 256;
+ const int align = intel_cursor_alignment(dev_priv);
ret = i915_gem_object_attach_phys(obj, align);
if (ret) {
}
static int
-intel_check_primary_plane(struct drm_plane *plane,
+intel_check_primary_plane(struct intel_plane *plane,
struct intel_crtc_state *crtc_state,
struct intel_plane_state *state)
{
- struct drm_i915_private *dev_priv = to_i915(plane->dev);
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
struct drm_crtc *crtc = state->base.crtc;
int min_scale = DRM_PLANE_HELPER_NO_SCALING;
int max_scale = DRM_PLANE_HELPER_NO_SCALING;
goto out_free;
if (INTEL_INFO(dev_priv)->cursor_needs_physical) {
- int align = IS_I830(dev_priv) ? 16 * 1024 : 256;
+ int align = intel_cursor_alignment(dev_priv);
ret = i915_gem_object_attach_phys(intel_fb_obj(fb), align);
if (ret) {
if (plane->state->visible) {
trace_intel_update_plane(plane, to_intel_crtc(crtc));
- intel_plane->update_plane(plane,
+ intel_plane->update_plane(intel_plane,
to_intel_crtc_state(crtc->state),
to_intel_plane_state(plane->state));
} else {
trace_intel_disable_plane(plane, to_intel_crtc(crtc));
- intel_plane->disable_plane(plane, crtc);
+ intel_plane->disable_plane(intel_plane, to_intel_crtc(crtc));
}
intel_cleanup_plane_fb(plane, new_plane_state);
return ERR_PTR(ret);
}
-static int
-intel_check_cursor_plane(struct drm_plane *plane,
- struct intel_crtc_state *crtc_state,
- struct intel_plane_state *state)
-{
- struct drm_i915_private *dev_priv = to_i915(plane->dev);
- struct drm_framebuffer *fb = state->base.fb;
- struct drm_i915_gem_object *obj = intel_fb_obj(fb);
- enum pipe pipe = to_intel_plane(plane)->pipe;
- unsigned stride;
- int ret;
-
- ret = drm_plane_helper_check_state(&state->base,
- &state->clip,
- DRM_PLANE_HELPER_NO_SCALING,
- DRM_PLANE_HELPER_NO_SCALING,
- true, true);
- if (ret)
- return ret;
-
- /* if we want to turn off the cursor ignore width and height */
- if (!obj)
- return 0;
-
- /* Check for which cursor types we support */
- if (!cursor_size_ok(dev_priv, state->base.crtc_w,
- state->base.crtc_h)) {
- DRM_DEBUG("Cursor dimension %dx%d not supported\n",
- state->base.crtc_w, state->base.crtc_h);
- return -EINVAL;
- }
-
- stride = roundup_pow_of_two(state->base.crtc_w) * 4;
- if (obj->base.size < stride * state->base.crtc_h) {
- DRM_DEBUG_KMS("buffer is too small\n");
- return -ENOMEM;
- }
-
- if (fb->modifier != DRM_FORMAT_MOD_LINEAR) {
- DRM_DEBUG_KMS("cursor cannot be tiled\n");
- return -EINVAL;
- }
-
- /*
- * There's something wrong with the cursor on CHV pipe C.
- * If it straddles the left edge of the screen then
- * moving it away from the edge or disabling it often
- * results in a pipe underrun, and often that can lead to
- * dead pipe (constant underrun reported, and it scans
- * out just a solid color). To recover from that, the
- * display power well must be turned off and on again.
- * Refuse the put the cursor into that compromised position.
- */
- if (IS_CHERRYVIEW(dev_priv) && pipe == PIPE_C &&
- state->base.visible && state->base.crtc_x < 0) {
- DRM_DEBUG_KMS("CHV cursor C not allowed to straddle the left screen edge\n");
- return -EINVAL;
- }
-
- if (IS_I845G(dev_priv) || IS_I865G(dev_priv))
- state->ctl = i845_cursor_ctl(crtc_state, state);
- else
- state->ctl = i9xx_cursor_ctl(crtc_state, state);
-
- return 0;
-}
-
-static void
-intel_disable_cursor_plane(struct drm_plane *plane,
- struct drm_crtc *crtc)
-{
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
-
- intel_crtc->cursor_addr = 0;
- intel_crtc_update_cursor(crtc, NULL);
-}
-
-static void
-intel_update_cursor_plane(struct drm_plane *plane,
- const struct intel_crtc_state *crtc_state,
- const struct intel_plane_state *state)
-{
- struct drm_crtc *crtc = crtc_state->base.crtc;
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- struct drm_i915_private *dev_priv = to_i915(plane->dev);
- struct drm_i915_gem_object *obj = intel_fb_obj(state->base.fb);
- uint32_t addr;
-
- if (!obj)
- addr = 0;
- else if (!INTEL_INFO(dev_priv)->cursor_needs_physical)
- addr = intel_plane_ggtt_offset(state);
- else
- addr = obj->phys_handle->busaddr;
-
- intel_crtc->cursor_addr = addr;
- intel_crtc_update_cursor(crtc, state);
-}
-
static struct intel_plane *
-intel_cursor_plane_create(struct drm_i915_private *dev_priv, enum pipe pipe)
+intel_cursor_plane_create(struct drm_i915_private *dev_priv,
+ enum pipe pipe)
{
struct intel_plane *cursor = NULL;
struct intel_plane_state *state = NULL;
cursor->plane = pipe;
cursor->id = PLANE_CURSOR;
cursor->frontbuffer_bit = INTEL_FRONTBUFFER_CURSOR(pipe);
- cursor->check_plane = intel_check_cursor_plane;
- cursor->update_plane = intel_update_cursor_plane;
- cursor->disable_plane = intel_disable_cursor_plane;
+
+ if (IS_I845G(dev_priv) || IS_I865G(dev_priv)) {
+ cursor->update_plane = i845_update_cursor;
+ cursor->disable_plane = i845_disable_cursor;
+ cursor->check_plane = i845_check_cursor;
+ } else {
+ cursor->update_plane = i9xx_update_cursor;
+ cursor->disable_plane = i9xx_disable_cursor;
+ cursor->check_plane = i9xx_check_cursor;
+ }
+
+ cursor->cursor.base = ~0;
+ cursor->cursor.cntl = ~0;
+
+ if (IS_I845G(dev_priv) || IS_I865G(dev_priv) || HAS_CUR_FBC(dev_priv))
+ cursor->cursor.size = ~0;
ret = drm_universal_plane_init(&dev_priv->drm, &cursor->base,
0, &intel_cursor_plane_funcs,
intel_crtc->pipe = pipe;
intel_crtc->plane = primary->plane;
- intel_crtc->cursor_base = ~0;
- intel_crtc->cursor_cntl = ~0;
- intel_crtc->cursor_size = ~0;
-
/* initialize shared scalers */
intel_crtc_init_scalers(intel_crtc, crtc_state);
case DRM_FORMAT_UYVY:
case DRM_FORMAT_YVYU:
case DRM_FORMAT_VYUY:
- if (INTEL_GEN(dev_priv) < 5) {
+ if (INTEL_GEN(dev_priv) < 5 && !IS_G4X(dev_priv)) {
DRM_DEBUG_KMS("unsupported pixel format: %s\n",
drm_get_format_name(mode_cmd->pixel_format, &format_name));
goto err;
dev->mode_config.funcs = &intel_mode_funcs;
+ init_llist_head(&dev_priv->atomic_helper.free_list);
INIT_WORK(&dev_priv->atomic_helper.free_work,
intel_atomic_helper_free_state_worker);
continue;
trace_intel_disable_plane(&plane->base, crtc);
- plane->disable_plane(&plane->base, &crtc->base);
+ plane->disable_plane(plane, crtc);
}
}
pll->on = false;
}
- if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
+ if (IS_G4X(dev_priv)) {
+ g4x_wm_get_hw_state(dev);
+ g4x_wm_sanitize(dev_priv);
+ } else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
vlv_wm_get_hw_state(dev);
vlv_wm_sanitize(dev_priv);
} else if (IS_GEN9(dev_priv)) {
if (state)
state->acquire_ctx = &ctx;
- /*
- * This is a cludge because with real atomic modeset mode_config.mutex
- * won't be taken. Unfortunately some probed state like
- * audio_codec_enable is still protected by mode_config.mutex, so lock
- * it here for now.
- */
- mutex_lock(&dev->mode_config.mutex);
drm_modeset_acquire_init(&ctx, 0);
while (1) {
drm_modeset_drop_locks(&ctx);
drm_modeset_acquire_fini(&ctx);
- mutex_unlock(&dev->mode_config.mutex);
if (ret)
DRM_ERROR("Restoring old state failed with %i\n", ret);
enum pipe pipe);
static void intel_dp_unset_edid(struct intel_dp *intel_dp);
-static int
-intel_dp_max_link_bw(struct intel_dp *intel_dp)
+static int intel_dp_num_rates(u8 link_bw_code)
{
- int max_link_bw = intel_dp->dpcd[DP_MAX_LINK_RATE];
-
- switch (max_link_bw) {
+ switch (link_bw_code) {
+ default:
+ WARN(1, "invalid max DP link bw val %x, using 1.62Gbps\n",
+ link_bw_code);
case DP_LINK_BW_1_62:
+ return 1;
case DP_LINK_BW_2_7:
+ return 2;
case DP_LINK_BW_5_4:
- break;
- default:
- WARN(1, "invalid max DP link bw val %x, using 1.62Gbps\n",
- max_link_bw);
- max_link_bw = DP_LINK_BW_1_62;
- break;
+ return 3;
}
- return max_link_bw;
}
-static u8 intel_dp_max_lane_count(struct intel_dp *intel_dp)
+/* update sink rates from dpcd */
+static void intel_dp_set_sink_rates(struct intel_dp *intel_dp)
{
- struct intel_digital_port *intel_dig_port = dp_to_dig_port(intel_dp);
- u8 source_max, sink_max;
+ int i, num_rates;
+
+ num_rates = intel_dp_num_rates(intel_dp->dpcd[DP_MAX_LINK_RATE]);
+
+ for (i = 0; i < num_rates; i++)
+ intel_dp->sink_rates[i] = default_rates[i];
- source_max = intel_dig_port->max_lanes;
- sink_max = intel_dp->max_sink_lane_count;
+ intel_dp->num_sink_rates = num_rates;
+}
+
+/* Theoretical max between source and sink */
+static int intel_dp_max_common_rate(struct intel_dp *intel_dp)
+{
+ return intel_dp->common_rates[intel_dp->num_common_rates - 1];
+}
+
+/* Theoretical max between source and sink */
+static int intel_dp_max_common_lane_count(struct intel_dp *intel_dp)
+{
+ struct intel_digital_port *intel_dig_port = dp_to_dig_port(intel_dp);
+ int source_max = intel_dig_port->max_lanes;
+ int sink_max = drm_dp_max_lane_count(intel_dp->dpcd);
return min(source_max, sink_max);
}
+int intel_dp_max_lane_count(struct intel_dp *intel_dp)
+{
+ return intel_dp->max_link_lane_count;
+}
+
int
intel_dp_link_required(int pixel_clock, int bpp)
{
return max_dotclk;
}
-static int
-intel_dp_sink_rates(struct intel_dp *intel_dp, const int **sink_rates)
-{
- if (intel_dp->num_sink_rates) {
- *sink_rates = intel_dp->sink_rates;
- return intel_dp->num_sink_rates;
- }
-
- *sink_rates = default_rates;
-
- return (intel_dp->max_sink_link_bw >> 3) + 1;
-}
-
-static int
-intel_dp_source_rates(struct intel_dp *intel_dp, const int **source_rates)
+static void
+intel_dp_set_source_rates(struct intel_dp *intel_dp)
{
struct intel_digital_port *dig_port = dp_to_dig_port(intel_dp);
struct drm_i915_private *dev_priv = to_i915(dig_port->base.base.dev);
+ const int *source_rates;
int size;
+ /* This should only be done once */
+ WARN_ON(intel_dp->source_rates || intel_dp->num_source_rates);
+
if (IS_GEN9_LP(dev_priv)) {
- *source_rates = bxt_rates;
+ source_rates = bxt_rates;
size = ARRAY_SIZE(bxt_rates);
} else if (IS_GEN9_BC(dev_priv)) {
- *source_rates = skl_rates;
+ source_rates = skl_rates;
size = ARRAY_SIZE(skl_rates);
} else {
- *source_rates = default_rates;
+ source_rates = default_rates;
size = ARRAY_SIZE(default_rates);
}
if (!intel_dp_source_supports_hbr2(intel_dp))
size--;
- return size;
+ intel_dp->source_rates = source_rates;
+ intel_dp->num_source_rates = size;
}
static int intersect_rates(const int *source_rates, int source_len,
return k;
}
-static int intel_dp_common_rates(struct intel_dp *intel_dp,
- int *common_rates)
+/* return index of rate in rates array, or -1 if not found */
+static int intel_dp_rate_index(const int *rates, int len, int rate)
{
- const int *source_rates, *sink_rates;
- int source_len, sink_len;
+ int i;
- sink_len = intel_dp_sink_rates(intel_dp, &sink_rates);
- source_len = intel_dp_source_rates(intel_dp, &source_rates);
+ for (i = 0; i < len; i++)
+ if (rate == rates[i])
+ return i;
- return intersect_rates(source_rates, source_len,
- sink_rates, sink_len,
- common_rates);
+ return -1;
}
-static int intel_dp_link_rate_index(struct intel_dp *intel_dp,
- int *common_rates, int link_rate)
+static void intel_dp_set_common_rates(struct intel_dp *intel_dp)
{
- int common_len;
- int index;
+ WARN_ON(!intel_dp->num_source_rates || !intel_dp->num_sink_rates);
+
+ intel_dp->num_common_rates = intersect_rates(intel_dp->source_rates,
+ intel_dp->num_source_rates,
+ intel_dp->sink_rates,
+ intel_dp->num_sink_rates,
+ intel_dp->common_rates);
- common_len = intel_dp_common_rates(intel_dp, common_rates);
- for (index = 0; index < common_len; index++) {
- if (link_rate == common_rates[common_len - index - 1])
- return common_len - index - 1;
+ /* Paranoia, there should always be something in common. */
+ if (WARN_ON(intel_dp->num_common_rates == 0)) {
+ intel_dp->common_rates[0] = default_rates[0];
+ intel_dp->num_common_rates = 1;
}
+}
- return -1;
+/* get length of common rates potentially limited by max_rate */
+static int intel_dp_common_len_rate_limit(struct intel_dp *intel_dp,
+ int max_rate)
+{
+ const int *common_rates = intel_dp->common_rates;
+ int i, common_len = intel_dp->num_common_rates;
+
+ /* Limit results by potentially reduced max rate */
+ for (i = 0; i < common_len; i++) {
+ if (common_rates[common_len - i - 1] <= max_rate)
+ return common_len - i;
+ }
+
+ return 0;
+}
+
+static bool intel_dp_link_params_valid(struct intel_dp *intel_dp)
+{
+ /*
+ * FIXME: we need to synchronize the current link parameters with
+ * hardware readout. Currently fast link training doesn't work on
+ * boot-up.
+ */
+ if (intel_dp->link_rate == 0 ||
+ intel_dp->link_rate > intel_dp->max_link_rate)
+ return false;
+
+ if (intel_dp->lane_count == 0 ||
+ intel_dp->lane_count > intel_dp_max_lane_count(intel_dp))
+ return false;
+
+ return true;
}
int intel_dp_get_link_train_fallback_values(struct intel_dp *intel_dp,
int link_rate, uint8_t lane_count)
{
- int common_rates[DP_MAX_SUPPORTED_RATES];
- int link_rate_index;
+ int index;
- link_rate_index = intel_dp_link_rate_index(intel_dp,
- common_rates,
- link_rate);
- if (link_rate_index > 0) {
- intel_dp->max_sink_link_bw = drm_dp_link_rate_to_bw_code(common_rates[link_rate_index - 1]);
- intel_dp->max_sink_lane_count = lane_count;
+ index = intel_dp_rate_index(intel_dp->common_rates,
+ intel_dp->num_common_rates,
+ link_rate);
+ if (index > 0) {
+ intel_dp->max_link_rate = intel_dp->common_rates[index - 1];
+ intel_dp->max_link_lane_count = lane_count;
} else if (lane_count > 1) {
- intel_dp->max_sink_link_bw = intel_dp_max_link_bw(intel_dp);
- intel_dp->max_sink_lane_count = lane_count >> 1;
+ intel_dp->max_link_rate = intel_dp_max_common_rate(intel_dp);
+ intel_dp->max_link_lane_count = lane_count >> 1;
} else {
DRM_ERROR("Link Training Unsuccessful\n");
return -1;
static void intel_dp_print_rates(struct intel_dp *intel_dp)
{
- const int *source_rates, *sink_rates;
- int source_len, sink_len, common_len;
- int common_rates[DP_MAX_SUPPORTED_RATES];
char str[128]; /* FIXME: too big for stack? */
if ((drm_debug & DRM_UT_KMS) == 0)
return;
- source_len = intel_dp_source_rates(intel_dp, &source_rates);
- snprintf_int_array(str, sizeof(str), source_rates, source_len);
+ snprintf_int_array(str, sizeof(str),
+ intel_dp->source_rates, intel_dp->num_source_rates);
DRM_DEBUG_KMS("source rates: %s\n", str);
- sink_len = intel_dp_sink_rates(intel_dp, &sink_rates);
- snprintf_int_array(str, sizeof(str), sink_rates, sink_len);
+ snprintf_int_array(str, sizeof(str),
+ intel_dp->sink_rates, intel_dp->num_sink_rates);
DRM_DEBUG_KMS("sink rates: %s\n", str);
- common_len = intel_dp_common_rates(intel_dp, common_rates);
- snprintf_int_array(str, sizeof(str), common_rates, common_len);
+ snprintf_int_array(str, sizeof(str),
+ intel_dp->common_rates, intel_dp->num_common_rates);
DRM_DEBUG_KMS("common rates: %s\n", str);
}
return true;
}
-static int rate_to_index(int find, const int *rates)
-{
- int i = 0;
-
- for (i = 0; i < DP_MAX_SUPPORTED_RATES; ++i)
- if (find == rates[i])
- break;
-
- return i;
-}
-
int
intel_dp_max_link_rate(struct intel_dp *intel_dp)
{
- int rates[DP_MAX_SUPPORTED_RATES] = {};
int len;
- len = intel_dp_common_rates(intel_dp, rates);
+ len = intel_dp_common_len_rate_limit(intel_dp, intel_dp->max_link_rate);
if (WARN_ON(len <= 0))
return 162000;
- return rates[len - 1];
+ return intel_dp->common_rates[len - 1];
}
int intel_dp_rate_select(struct intel_dp *intel_dp, int rate)
{
- return rate_to_index(rate, intel_dp->sink_rates);
+ int i = intel_dp_rate_index(intel_dp->sink_rates,
+ intel_dp->num_sink_rates, rate);
+
+ if (WARN_ON(i < 0))
+ i = 0;
+
+ return i;
}
void intel_dp_compute_rate(struct intel_dp *intel_dp, int port_clock,
uint8_t *link_bw, uint8_t *rate_select)
{
- if (intel_dp->num_sink_rates) {
+ /* eDP 1.4 rate select method. */
+ if (intel_dp->use_rate_select) {
*link_bw = 0;
*rate_select =
intel_dp_rate_select(intel_dp, port_clock);
/* Conveniently, the link BW constants become indices with a shift...*/
int min_clock = 0;
int max_clock;
- int link_rate_index;
int bpp, mode_rate;
int link_avail, link_clock;
- int common_rates[DP_MAX_SUPPORTED_RATES] = {};
int common_len;
uint8_t link_bw, rate_select;
- common_len = intel_dp_common_rates(intel_dp, common_rates);
+ common_len = intel_dp_common_len_rate_limit(intel_dp,
+ intel_dp->max_link_rate);
/* No common link rates between source and sink */
WARN_ON(common_len <= 0);
/* Use values requested by Compliance Test Request */
if (intel_dp->compliance.test_type == DP_TEST_LINK_TRAINING) {
- link_rate_index = intel_dp_link_rate_index(intel_dp,
- common_rates,
- intel_dp->compliance.test_link_rate);
- if (link_rate_index >= 0)
- min_clock = max_clock = link_rate_index;
+ int index;
+
+ index = intel_dp_rate_index(intel_dp->common_rates,
+ intel_dp->num_common_rates,
+ intel_dp->compliance.test_link_rate);
+ if (index >= 0)
+ min_clock = max_clock = index;
min_lane_count = max_lane_count = intel_dp->compliance.test_lane_count;
}
DRM_DEBUG_KMS("DP link computation with max lane count %i "
"max bw %d pixel clock %iKHz\n",
- max_lane_count, common_rates[max_clock],
+ max_lane_count, intel_dp->common_rates[max_clock],
adjusted_mode->crtc_clock);
/* Walk through all bpp values. Luckily they're all nicely spaced with 2
lane_count <= max_lane_count;
lane_count <<= 1) {
- link_clock = common_rates[clock];
+ link_clock = intel_dp->common_rates[clock];
link_avail = intel_dp_max_data_rate(link_clock,
lane_count);
pipe_config->lane_count = lane_count;
pipe_config->pipe_bpp = bpp;
- pipe_config->port_clock = common_rates[clock];
+ pipe_config->port_clock = intel_dp->common_rates[clock];
intel_dp_compute_rate(intel_dp, pipe_config->port_clock,
&link_bw, &rate_select);
{
uint8_t psr_caps = 0;
- drm_dp_dpcd_readb(&intel_dp->aux, DP_PSR_CAPS, &psr_caps);
+ if (drm_dp_dpcd_readb(&intel_dp->aux, DP_PSR_CAPS, &psr_caps) != 1)
+ return false;
return psr_caps & DP_PSR2_SU_Y_COORDINATE_REQUIRED;
}
{
uint8_t dprx = 0;
- drm_dp_dpcd_readb(&intel_dp->aux,
- DP_DPRX_FEATURE_ENUMERATION_LIST,
- &dprx);
+ if (drm_dp_dpcd_readb(&intel_dp->aux, DP_DPRX_FEATURE_ENUMERATION_LIST,
+ &dprx) != 1)
+ return false;
return dprx & DP_VSC_SDP_EXT_FOR_COLORIMETRY_SUPPORTED;
}
{
uint8_t alpm_caps = 0;
- drm_dp_dpcd_readb(&intel_dp->aux, DP_RECEIVER_ALPM_CAP, &alpm_caps);
+ if (drm_dp_dpcd_readb(&intel_dp->aux, DP_RECEIVER_ALPM_CAP,
+ &alpm_caps) != 1)
+ return false;
return alpm_caps & DP_ALPM_CAP;
}
uint8_t frame_sync_cap;
dev_priv->psr.sink_support = true;
- drm_dp_dpcd_read(&intel_dp->aux,
- DP_SINK_DEVICE_AUX_FRAME_SYNC_CAP,
- &frame_sync_cap, 1);
+ if (drm_dp_dpcd_readb(&intel_dp->aux,
+ DP_SINK_DEVICE_AUX_FRAME_SYNC_CAP,
+ &frame_sync_cap) != 1)
+ frame_sync_cap = 0;
dev_priv->psr.aux_frame_sync = frame_sync_cap ? true : false;
/* PSR2 needs frame sync as well */
dev_priv->psr.psr2_support = dev_priv->psr.aux_frame_sync;
intel_dp->num_sink_rates = i;
}
+ if (intel_dp->num_sink_rates)
+ intel_dp->use_rate_select = true;
+ else
+ intel_dp_set_sink_rates(intel_dp);
+
+ intel_dp_set_common_rates(intel_dp);
+
return true;
}
static bool
intel_dp_get_dpcd(struct intel_dp *intel_dp)
{
+ u8 sink_count;
+
if (!intel_dp_read_dpcd(intel_dp))
return false;
- if (drm_dp_dpcd_read(&intel_dp->aux, DP_SINK_COUNT,
- &intel_dp->sink_count, 1) < 0)
+ /* Don't clobber cached eDP rates. */
+ if (!is_edp(intel_dp)) {
+ intel_dp_set_sink_rates(intel_dp);
+ intel_dp_set_common_rates(intel_dp);
+ }
+
+ if (drm_dp_dpcd_readb(&intel_dp->aux, DP_SINK_COUNT, &sink_count) <= 0)
return false;
/*
* a member variable in intel_dp will track any changes
* between short pulse interrupts.
*/
- intel_dp->sink_count = DP_GET_SINK_COUNT(intel_dp->sink_count);
+ intel_dp->sink_count = DP_GET_SINK_COUNT(sink_count);
/*
* SINK_COUNT == 0 and DOWNSTREAM_PORT_PRESENT == 1 implies that
static bool
intel_dp_can_mst(struct intel_dp *intel_dp)
{
- u8 buf[1];
+ u8 mstm_cap;
if (!i915.enable_dp_mst)
return false;
if (intel_dp->dpcd[DP_DPCD_REV] < 0x12)
return false;
- if (drm_dp_dpcd_read(&intel_dp->aux, DP_MSTM_CAP, buf, 1) != 1)
+ if (drm_dp_dpcd_readb(&intel_dp->aux, DP_MSTM_CAP, &mstm_cap) != 1)
return false;
- return buf[0] & DP_MST_CAP;
+ return mstm_cap & DP_MST_CAP;
}
static void
static bool
intel_dp_get_sink_irq(struct intel_dp *intel_dp, u8 *sink_irq_vector)
{
- return drm_dp_dpcd_read(&intel_dp->aux,
- DP_DEVICE_SERVICE_IRQ_VECTOR,
- sink_irq_vector, 1) == 1;
+ return drm_dp_dpcd_readb(&intel_dp->aux, DP_DEVICE_SERVICE_IRQ_VECTOR,
+ sink_irq_vector) == 1;
}
static bool
{
int status = 0;
int min_lane_count = 1;
- int common_rates[DP_MAX_SUPPORTED_RATES] = {};
int link_rate_index, test_link_rate;
uint8_t test_lane_count, test_link_bw;
/* (DP CTS 1.2)
test_lane_count &= DP_MAX_LANE_COUNT_MASK;
/* Validate the requested lane count */
if (test_lane_count < min_lane_count ||
- test_lane_count > intel_dp->max_sink_lane_count)
+ test_lane_count > intel_dp->max_link_lane_count)
return DP_TEST_NAK;
status = drm_dp_dpcd_readb(&intel_dp->aux, DP_TEST_LINK_RATE,
}
/* Validate the requested link rate */
test_link_rate = drm_dp_bw_code_to_link_rate(test_link_bw);
- link_rate_index = intel_dp_link_rate_index(intel_dp,
- common_rates,
- test_link_rate);
+ link_rate_index = intel_dp_rate_index(intel_dp->common_rates,
+ intel_dp->num_common_rates,
+ test_link_rate);
if (link_rate_index < 0)
return DP_TEST_NAK;
static uint8_t intel_dp_autotest_video_pattern(struct intel_dp *intel_dp)
{
uint8_t test_pattern;
- uint16_t test_misc;
+ uint8_t test_misc;
__be16 h_width, v_height;
int status = 0;
/* Read the TEST_PATTERN (DP CTS 3.1.5) */
- status = drm_dp_dpcd_read(&intel_dp->aux, DP_TEST_PATTERN,
- &test_pattern, 1);
+ status = drm_dp_dpcd_readb(&intel_dp->aux, DP_TEST_PATTERN,
+ &test_pattern);
if (status <= 0) {
DRM_DEBUG_KMS("Test pattern read failed\n");
return DP_TEST_NAK;
return DP_TEST_NAK;
}
- status = drm_dp_dpcd_read(&intel_dp->aux, DP_TEST_MISC0,
- &test_misc, 1);
+ status = drm_dp_dpcd_readb(&intel_dp->aux, DP_TEST_MISC0,
+ &test_misc);
if (status <= 0) {
DRM_DEBUG_KMS("TEST MISC read failed\n");
return DP_TEST_NAK;
*/
block += intel_connector->detect_edid->extensions;
- if (!drm_dp_dpcd_write(&intel_dp->aux,
- DP_TEST_EDID_CHECKSUM,
- &block->checksum,
- 1))
+ if (drm_dp_dpcd_writeb(&intel_dp->aux, DP_TEST_EDID_CHECKSUM,
+ block->checksum) <= 0)
DRM_DEBUG_KMS("Failed to write EDID checksum\n");
test_result = DP_TEST_ACK | DP_TEST_EDID_CHECKSUM_WRITE;
if (!to_intel_crtc(intel_encoder->base.crtc)->active)
return;
- /* FIXME: we need to synchronize this sort of stuff with hardware
- * readout. Currently fast link training doesn't work on boot-up. */
- if (!intel_dp->lane_count)
+ /*
+ * Validate the cached values of intel_dp->link_rate and
+ * intel_dp->lane_count before attempting to retrain.
+ */
+ if (!intel_dp_link_params_valid(intel_dp))
return;
/* Retrain if Channel EQ or CR not ok */
yesno(drm_dp_tps3_supported(intel_dp->dpcd)));
if (intel_dp->reset_link_params) {
- /* Set the max lane count for sink */
- intel_dp->max_sink_lane_count = drm_dp_max_lane_count(intel_dp->dpcd);
+ /* Initial max link lane count */
+ intel_dp->max_link_lane_count = intel_dp_max_common_lane_count(intel_dp);
- /* Set the max link BW for sink */
- intel_dp->max_sink_link_bw = intel_dp_max_link_bw(intel_dp);
+ /* Initial max link rate */
+ intel_dp->max_link_rate = intel_dp_max_common_rate(intel_dp);
intel_dp->reset_link_params = false;
}
return intel_bios_is_port_edp(dev_priv, port);
}
-void
+static void
intel_dp_add_properties(struct intel_dp *intel_dp, struct drm_connector *connector)
{
struct intel_connector *intel_connector = to_intel_connector(connector);
}
}
+static void intel_dp_modeset_retry_work_fn(struct work_struct *work)
+{
+ struct intel_connector *intel_connector;
+ struct drm_connector *connector;
+
+ intel_connector = container_of(work, typeof(*intel_connector),
+ modeset_retry_work);
+ connector = &intel_connector->base;
+ DRM_DEBUG_KMS("[CONNECTOR:%d:%s]\n", connector->base.id,
+ connector->name);
+
+ /* Grab the locks before changing connector property*/
+ mutex_lock(&connector->dev->mode_config.mutex);
+ /* Set connector link status to BAD and send a Uevent to notify
+ * userspace to do a modeset.
+ */
+ drm_mode_connector_set_link_status_property(connector,
+ DRM_MODE_LINK_STATUS_BAD);
+ mutex_unlock(&connector->dev->mode_config.mutex);
+ /* Send Hotplug uevent so userspace can reprobe */
+ drm_kms_helper_hotplug_event(connector->dev);
+}
+
bool
intel_dp_init_connector(struct intel_digital_port *intel_dig_port,
struct intel_connector *intel_connector)
enum port port = intel_dig_port->port;
int type;
+ /* Initialize the work for modeset in case of link train failure */
+ INIT_WORK(&intel_connector->modeset_retry_work,
+ intel_dp_modeset_retry_work_fn);
+
if (WARN(intel_dig_port->max_lanes < 1,
"Not enough lanes (%d) for DP on port %c\n",
intel_dig_port->max_lanes, port_name(port)))
return false;
+ intel_dp_set_source_rates(intel_dp);
+
intel_dp->reset_link_params = true;
intel_dp->pps_pipe = INVALID_PIPE;
intel_dp->active_pipe = INVALID_PIPE;
{
uint8_t reg_val = 0;
+ /* Early return when display use other mechanism to enable backlight. */
+ if (!(intel_dp->edp_dpcd[1] & DP_EDP_BACKLIGHT_AUX_ENABLE_CAP))
+ return;
+
if (drm_dp_dpcd_readb(&intel_dp->aux, DP_EDP_DISPLAY_CONTROL_REGISTER,
®_val) < 0) {
DRM_DEBUG_KMS("Failed to read DPCD register 0x%x\n",
{
struct intel_dp *intel_dp = enc_to_intel_dp(&connector->encoder->base);
uint8_t dpcd_buf = 0;
+ uint8_t edp_backlight_mode = 0;
- set_aux_backlight_enable(intel_dp, true);
+ if (drm_dp_dpcd_readb(&intel_dp->aux,
+ DP_EDP_BACKLIGHT_MODE_SET_REGISTER, &dpcd_buf) != 1) {
+ DRM_DEBUG_KMS("Failed to read DPCD register 0x%x\n",
+ DP_EDP_BACKLIGHT_MODE_SET_REGISTER);
+ return;
+ }
+
+ edp_backlight_mode = dpcd_buf & DP_EDP_BACKLIGHT_CONTROL_MODE_MASK;
+
+ switch (edp_backlight_mode) {
+ case DP_EDP_BACKLIGHT_CONTROL_MODE_PWM:
+ case DP_EDP_BACKLIGHT_CONTROL_MODE_PRESET:
+ case DP_EDP_BACKLIGHT_CONTROL_MODE_PRODUCT:
+ dpcd_buf &= ~DP_EDP_BACKLIGHT_CONTROL_MODE_MASK;
+ dpcd_buf |= DP_EDP_BACKLIGHT_CONTROL_MODE_DPCD;
+ if (drm_dp_dpcd_writeb(&intel_dp->aux,
+ DP_EDP_BACKLIGHT_MODE_SET_REGISTER, dpcd_buf) < 0) {
+ DRM_DEBUG_KMS("Failed to write aux backlight mode\n");
+ }
+ break;
+
+ /* Do nothing when it is already DPCD mode */
+ case DP_EDP_BACKLIGHT_CONTROL_MODE_DPCD:
+ default:
+ break;
+ }
- if ((drm_dp_dpcd_readb(&intel_dp->aux,
- DP_EDP_BACKLIGHT_MODE_SET_REGISTER, &dpcd_buf) == 1) &&
- ((dpcd_buf & DP_EDP_BACKLIGHT_CONTROL_MODE_MASK) ==
- DP_EDP_BACKLIGHT_CONTROL_MODE_PRESET))
- drm_dp_dpcd_writeb(&intel_dp->aux, DP_EDP_BACKLIGHT_MODE_SET_REGISTER,
- (dpcd_buf | DP_EDP_BACKLIGHT_CONTROL_MODE_DPCD));
+ set_aux_backlight_enable(intel_dp, true);
+ intel_dp_aux_set_backlight(connector, connector->panel.backlight.level);
}
static void intel_dp_aux_disable_backlight(struct intel_connector *connector)
* the panel can support backlight control over the aux channel
*/
if (intel_dp->edp_dpcd[1] & DP_EDP_TCON_BACKLIGHT_ADJUSTMENT_CAP &&
- (intel_dp->edp_dpcd[1] & DP_EDP_BACKLIGHT_AUX_ENABLE_CAP) &&
- !((intel_dp->edp_dpcd[1] & DP_EDP_BACKLIGHT_PIN_ENABLE_CAP) ||
- (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP))) {
+ (intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_AUX_SET_CAP) &&
+ !(intel_dp->edp_dpcd[2] & DP_EDP_BACKLIGHT_BRIGHTNESS_PWM_PIN_CAP)) {
DRM_DEBUG_KMS("AUX Backlight Control Supported!\n");
return true;
}
link_config[1] |= DP_LANE_COUNT_ENHANCED_FRAME_EN;
drm_dp_dpcd_write(&intel_dp->aux, DP_LINK_BW_SET, link_config, 2);
- if (intel_dp->num_sink_rates)
+ /* eDP 1.4 rate select method. */
+ if (!link_bw)
drm_dp_dpcd_write(&intel_dp->aux, DP_LINK_RATE_SET,
&rate_select, 1);
void
intel_dp_start_link_train(struct intel_dp *intel_dp)
{
- intel_dp_link_training_clock_recovery(intel_dp);
- intel_dp_link_training_channel_equalization(intel_dp);
+ struct intel_connector *intel_connector = intel_dp->attached_connector;
+
+ if (!intel_dp_link_training_clock_recovery(intel_dp))
+ goto failure_handling;
+ if (!intel_dp_link_training_channel_equalization(intel_dp))
+ goto failure_handling;
+
+ DRM_DEBUG_KMS("Link Training Passed at Link Rate = %d, Lane count = %d",
+ intel_dp->link_rate, intel_dp->lane_count);
+ return;
+
+ failure_handling:
+ DRM_DEBUG_KMS("Link Training failed at link rate = %d, lane count = %d",
+ intel_dp->link_rate, intel_dp->lane_count);
+ if (!intel_dp_get_link_train_fallback_values(intel_dp,
+ intel_dp->link_rate,
+ intel_dp->lane_count))
+ /* Schedule a Hotplug Uevent to userspace to start modeset */
+ schedule_work(&intel_connector->modeset_retry_work);
+ return;
}
* for MST we always configure max link bw - the spec doesn't
* seem to suggest we should do otherwise.
*/
- lane_count = drm_dp_max_lane_count(intel_dp->dpcd);
+ lane_count = intel_dp_max_lane_count(intel_dp);
+
pipe_config->lane_count = lane_count;
pipe_config->pipe_bpp = bpp;
return drm_dp_mst_detect_port(connector, &intel_dp->mst_mgr, intel_connector->port);
}
-static int
-intel_dp_mst_set_property(struct drm_connector *connector,
- struct drm_property *property,
- uint64_t val)
-{
- return 0;
-}
-
static void
intel_dp_mst_connector_destroy(struct drm_connector *connector)
{
.dpms = drm_atomic_helper_connector_dpms,
.detect = intel_dp_mst_detect,
.fill_modes = drm_helper_probe_single_connector_modes,
- .set_property = intel_dp_mst_set_property,
- .atomic_get_property = intel_connector_atomic_get_property,
+ .set_property = drm_atomic_helper_connector_set_property,
.late_register = intel_connector_register,
.early_unregister = intel_connector_unregister,
.destroy = intel_dp_mst_connector_destroy,
int max_rate, mode_rate, max_lanes, max_link_clock;
max_link_clock = intel_dp_max_link_rate(intel_dp);
- max_lanes = drm_dp_max_lane_count(intel_dp->dpcd);
+ max_lanes = intel_dp_max_lane_count(intel_dp);
max_rate = intel_dp_max_data_rate(max_link_clock, max_lanes);
mode_rate = intel_dp_link_required(mode->clock, bpp);
drm_mode_connector_attach_encoder(&intel_connector->base,
&intel_dp->mst_encoders[i]->base.base);
}
- intel_dp_add_properties(intel_dp, connector);
drm_object_attach_property(&connector->base, dev->mode_config.path_property, 0);
drm_object_attach_property(&connector->base, dev->mode_config.tile_property, 0);
int cpu, ret, timeout = (US) * 1000; \
u64 base; \
_WAIT_FOR_ATOMIC_CHECK(ATOMIC); \
- BUILD_BUG_ON((US) > 50000); \
if (!(ATOMIC)) { \
preempt_disable(); \
cpu = smp_processor_id(); \
ret__; \
})
-#define wait_for_atomic(COND, MS) _wait_for_atomic((COND), (MS) * 1000, 1)
-#define wait_for_atomic_us(COND, US) _wait_for_atomic((COND), (US), 1)
+#define wait_for_atomic_us(COND, US) \
+({ \
+ BUILD_BUG_ON(!__builtin_constant_p(US)); \
+ BUILD_BUG_ON((US) > 50000); \
+ _wait_for_atomic((COND), (US), 1); \
+})
+
+#define wait_for_atomic(COND, MS) wait_for_atomic_us((COND), (MS) * 1000)
#define KHz(x) (1000 * (x))
#define MHz(x) KHz(1000 * (x))
void *port; /* store this opaque as its illegal to dereference it */
struct intel_dp *mst_port;
+
+ /* Work struct to schedule a uevent on link train failure */
+ struct work_struct modeset_retry_work;
};
struct dpll {
};
struct vlv_wm_state {
- struct vlv_pipe_wm wm[NUM_VLV_WM_LEVELS];
- struct vlv_sr_wm sr[NUM_VLV_WM_LEVELS];
+ struct g4x_pipe_wm wm[NUM_VLV_WM_LEVELS];
+ struct g4x_sr_wm sr[NUM_VLV_WM_LEVELS];
uint8_t num_levels;
bool cxsr;
};
u16 plane[I915_MAX_PLANES];
};
+enum g4x_wm_level {
+ G4X_WM_LEVEL_NORMAL,
+ G4X_WM_LEVEL_SR,
+ G4X_WM_LEVEL_HPLL,
+ NUM_G4X_WM_LEVELS,
+};
+
+struct g4x_wm_state {
+ struct g4x_pipe_wm wm;
+ struct g4x_sr_wm sr;
+ struct g4x_sr_wm hpll;
+ bool cxsr;
+ bool hpll_en;
+ bool fbc_en;
+};
+
struct intel_crtc_wm_state {
union {
struct {
struct {
/* "raw" watermarks (not inverted) */
- struct vlv_pipe_wm raw[NUM_VLV_WM_LEVELS];
+ struct g4x_pipe_wm raw[NUM_VLV_WM_LEVELS];
/* intermediate watermarks (inverted) */
struct vlv_wm_state intermediate;
/* optimal watermarks (inverted) */
/* display FIFO split */
struct vlv_fifo_state fifo_state;
} vlv;
+
+ struct {
+ /* "raw" watermarks */
+ struct g4x_pipe_wm raw[NUM_G4X_WM_LEVELS];
+ /* intermediate watermarks */
+ struct g4x_wm_state intermediate;
+ /* optimal watermarks */
+ struct g4x_wm_state optimal;
+ } g4x;
};
/*
int adjusted_x;
int adjusted_y;
- uint32_t cursor_addr;
- uint32_t cursor_cntl;
- uint32_t cursor_size;
- uint32_t cursor_base;
-
struct intel_crtc_state *config;
/* global reset count when the last flip was submitted */
union {
struct intel_pipe_wm ilk;
struct vlv_wm_state vlv;
+ struct g4x_wm_state g4x;
} active;
} wm;
int max_downscale;
uint32_t frontbuffer_bit;
+ struct {
+ u32 base, cntl, size;
+ } cursor;
+
/*
* NOTE: Do not place new plane state fields here (e.g., when adding
* new plane properties). New runtime state should now be placed in
* the intel_plane_state structure and accessed via plane_state.
*/
- void (*update_plane)(struct drm_plane *plane,
+ void (*update_plane)(struct intel_plane *plane,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state);
- void (*disable_plane)(struct drm_plane *plane,
- struct drm_crtc *crtc);
- int (*check_plane)(struct drm_plane *plane,
+ void (*disable_plane)(struct intel_plane *plane,
+ struct intel_crtc *crtc);
+ int (*check_plane)(struct intel_plane *plane,
struct intel_crtc_state *crtc_state,
struct intel_plane_state *state);
};
uint8_t psr_dpcd[EDP_PSR_RECEIVER_CAP_SIZE];
uint8_t downstream_ports[DP_MAX_DOWNSTREAM_PORTS];
uint8_t edp_dpcd[EDP_DISPLAY_CTL_CAP_SIZE];
- /* sink rates as reported by DP_SUPPORTED_LINK_RATES */
- uint8_t num_sink_rates;
+ /* source rates */
+ int num_source_rates;
+ const int *source_rates;
+ /* sink rates as reported by DP_MAX_LINK_RATE/DP_SUPPORTED_LINK_RATES */
+ int num_sink_rates;
int sink_rates[DP_MAX_SUPPORTED_RATES];
- /* Max lane count for the sink as per DPCD registers */
- uint8_t max_sink_lane_count;
- /* Max link BW for the sink as per DPCD registers */
- int max_sink_link_bw;
+ bool use_rate_select;
+ /* intersection of source and sink rates */
+ int num_common_rates;
+ int common_rates[DP_MAX_SUPPORTED_RATES];
+ /* Max lane count for the current link */
+ int max_link_lane_count;
+ /* Max rate for the current link */
+ int max_link_rate;
/* sink or branch descriptor */
struct intel_dp_desc desc;
struct drm_dp_aux aux;
void intel_edp_panel_vdd_on(struct intel_dp *intel_dp);
void intel_edp_panel_on(struct intel_dp *intel_dp);
void intel_edp_panel_off(struct intel_dp *intel_dp);
-void intel_dp_add_properties(struct intel_dp *intel_dp, struct drm_connector *connector);
void intel_dp_mst_suspend(struct drm_device *dev);
void intel_dp_mst_resume(struct drm_device *dev);
int intel_dp_max_link_rate(struct intel_dp *intel_dp);
+int intel_dp_max_lane_count(struct intel_dp *intel_dp);
int intel_dp_rate_select(struct intel_dp *intel_dp, int rate);
void intel_dp_hot_plug(struct intel_encoder *intel_encoder);
void intel_power_sequencer_reset(struct drm_i915_private *dev_priv);
struct intel_rps_client *rps,
unsigned long submitted);
void intel_queue_rps_boost_for_request(struct drm_i915_gem_request *req);
+void g4x_wm_get_hw_state(struct drm_device *dev);
void vlv_wm_get_hw_state(struct drm_device *dev);
void ilk_wm_get_hw_state(struct drm_device *dev);
void skl_wm_get_hw_state(struct drm_device *dev);
struct skl_ddb_allocation *ddb /* out */);
void skl_pipe_wm_get_hw_state(struct drm_crtc *crtc,
struct skl_pipe_wm *out);
+void g4x_wm_sanitize(struct drm_i915_private *dev_priv);
void vlv_wm_sanitize(struct drm_i915_private *dev_priv);
bool intel_can_enable_sagv(struct drm_atomic_state *state);
int intel_enable_sagv(struct drm_i915_private *dev_priv);
val |= (ULPS_STATE_ENTER | DEVICE_READY);
I915_WRITE(MIPI_DEVICE_READY(port), val);
- /* Wait for ULPS Not active */
+ /* Wait for ULPS active */
if (intel_wait_for_register(dev_priv,
- MIPI_CTRL(port), GLK_ULPS_NOT_ACTIVE,
- GLK_ULPS_NOT_ACTIVE, 20))
- DRM_ERROR("ULPS is still active\n");
+ MIPI_CTRL(port), GLK_ULPS_NOT_ACTIVE, 0, 20))
+ DRM_ERROR("ULPS not active\n");
/* Exit ULPS */
val = I915_READ(MIPI_DEVICE_READY(port));
clk_zero_cnt << 8 | prepare_cnt;
/*
- * LP to HS switch count = 4TLPX + PREP_COUNT * 2 + EXIT_ZERO_COUNT * 2
- * + 10UI + Extra Byte Count
+ * LP to HS switch count = 4TLPX + PREP_COUNT * mul + EXIT_ZERO_COUNT *
+ * mul + 10UI + Extra Byte Count
*
* HS to LP switch count = THS-TRAIL + 2TLPX + Extra Byte Count
* Extra Byte Count is calculated according to number of lanes.
/* B044 */
/* FIXME:
* The comment above does not match with the code */
- lp_to_hs_switch = DIV_ROUND_UP(4 * tlpx_ui + prepare_cnt * 2 +
- exit_zero_cnt * 2 + 10, 8);
+ lp_to_hs_switch = DIV_ROUND_UP(4 * tlpx_ui + prepare_cnt * mul +
+ exit_zero_cnt * mul + 10, 8);
hs_to_lp_switch = DIV_ROUND_UP(mipi_config->ths_trail + 2 * tlpx_ui, 8);
.early_unregister = intel_connector_unregister,
.destroy = intel_dvo_destroy,
.fill_modes = drm_helper_probe_single_connector_modes,
- .atomic_get_property = intel_connector_atomic_get_property,
+ .set_property = drm_atomic_helper_connector_set_property,
.atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
.atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
};
#include "intel_ringbuffer.h"
#include "intel_lrc.h"
-static const struct engine_info {
+/* Haswell does have the CXT_SIZE register however it does not appear to be
+ * valid. Now, docs explain in dwords what is in the context object. The full
+ * size is 70720 bytes, however, the power context and execlist context will
+ * never be saved (power context is stored elsewhere, and execlists don't work
+ * on HSW) - so the final size, including the extra state required for the
+ * Resource Streamer, is 66944 bytes, which rounds to 17 pages.
+ */
+#define HSW_CXT_TOTAL_SIZE (17 * PAGE_SIZE)
+/* Same as Haswell, but 72064 bytes now. */
+#define GEN8_CXT_TOTAL_SIZE (18 * PAGE_SIZE)
+
+#define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE)
+#define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE)
+
+#define GEN8_LR_CONTEXT_OTHER_SIZE ( 2 * PAGE_SIZE)
+
+struct engine_class_info {
const char *name;
- unsigned int exec_id;
+ int (*init_legacy)(struct intel_engine_cs *engine);
+ int (*init_execlists)(struct intel_engine_cs *engine);
+};
+
+static const struct engine_class_info intel_engine_classes[] = {
+ [RENDER_CLASS] = {
+ .name = "rcs",
+ .init_execlists = logical_render_ring_init,
+ .init_legacy = intel_init_render_ring_buffer,
+ },
+ [COPY_ENGINE_CLASS] = {
+ .name = "bcs",
+ .init_execlists = logical_xcs_ring_init,
+ .init_legacy = intel_init_blt_ring_buffer,
+ },
+ [VIDEO_DECODE_CLASS] = {
+ .name = "vcs",
+ .init_execlists = logical_xcs_ring_init,
+ .init_legacy = intel_init_bsd_ring_buffer,
+ },
+ [VIDEO_ENHANCEMENT_CLASS] = {
+ .name = "vecs",
+ .init_execlists = logical_xcs_ring_init,
+ .init_legacy = intel_init_vebox_ring_buffer,
+ },
+};
+
+struct engine_info {
unsigned int hw_id;
+ unsigned int uabi_id;
+ u8 class;
+ u8 instance;
u32 mmio_base;
unsigned irq_shift;
- int (*init_legacy)(struct intel_engine_cs *engine);
- int (*init_execlists)(struct intel_engine_cs *engine);
-} intel_engines[] = {
+};
+
+static const struct engine_info intel_engines[] = {
[RCS] = {
- .name = "rcs",
.hw_id = RCS_HW,
- .exec_id = I915_EXEC_RENDER,
+ .uabi_id = I915_EXEC_RENDER,
+ .class = RENDER_CLASS,
+ .instance = 0,
.mmio_base = RENDER_RING_BASE,
.irq_shift = GEN8_RCS_IRQ_SHIFT,
- .init_execlists = logical_render_ring_init,
- .init_legacy = intel_init_render_ring_buffer,
},
[BCS] = {
- .name = "bcs",
.hw_id = BCS_HW,
- .exec_id = I915_EXEC_BLT,
+ .uabi_id = I915_EXEC_BLT,
+ .class = COPY_ENGINE_CLASS,
+ .instance = 0,
.mmio_base = BLT_RING_BASE,
.irq_shift = GEN8_BCS_IRQ_SHIFT,
- .init_execlists = logical_xcs_ring_init,
- .init_legacy = intel_init_blt_ring_buffer,
},
[VCS] = {
- .name = "vcs",
.hw_id = VCS_HW,
- .exec_id = I915_EXEC_BSD,
+ .uabi_id = I915_EXEC_BSD,
+ .class = VIDEO_DECODE_CLASS,
+ .instance = 0,
.mmio_base = GEN6_BSD_RING_BASE,
.irq_shift = GEN8_VCS1_IRQ_SHIFT,
- .init_execlists = logical_xcs_ring_init,
- .init_legacy = intel_init_bsd_ring_buffer,
},
[VCS2] = {
- .name = "vcs2",
.hw_id = VCS2_HW,
- .exec_id = I915_EXEC_BSD,
+ .uabi_id = I915_EXEC_BSD,
+ .class = VIDEO_DECODE_CLASS,
+ .instance = 1,
.mmio_base = GEN8_BSD2_RING_BASE,
.irq_shift = GEN8_VCS2_IRQ_SHIFT,
- .init_execlists = logical_xcs_ring_init,
- .init_legacy = intel_init_bsd2_ring_buffer,
},
[VECS] = {
- .name = "vecs",
.hw_id = VECS_HW,
- .exec_id = I915_EXEC_VEBOX,
+ .uabi_id = I915_EXEC_VEBOX,
+ .class = VIDEO_ENHANCEMENT_CLASS,
+ .instance = 0,
.mmio_base = VEBOX_RING_BASE,
.irq_shift = GEN8_VECS_IRQ_SHIFT,
- .init_execlists = logical_xcs_ring_init,
- .init_legacy = intel_init_vebox_ring_buffer,
},
};
+/**
+ * ___intel_engine_context_size() - return the size of the context for an engine
+ * @dev_priv: i915 device private
+ * @class: engine class
+ *
+ * Each engine class may require a different amount of space for a context
+ * image.
+ *
+ * Return: size (in bytes) of an engine class specific context image
+ *
+ * Note: this size includes the HWSP, which is part of the context image
+ * in LRC mode, but does not include the "shared data page" used with
+ * GuC submission. The caller should account for this if using the GuC.
+ */
+static u32
+__intel_engine_context_size(struct drm_i915_private *dev_priv, u8 class)
+{
+ u32 cxt_size;
+
+ BUILD_BUG_ON(I915_GTT_PAGE_SIZE != PAGE_SIZE);
+
+ switch (class) {
+ case RENDER_CLASS:
+ switch (INTEL_GEN(dev_priv)) {
+ default:
+ MISSING_CASE(INTEL_GEN(dev_priv));
+ case 9:
+ return GEN9_LR_CONTEXT_RENDER_SIZE;
+ case 8:
+ return i915.enable_execlists ?
+ GEN8_LR_CONTEXT_RENDER_SIZE :
+ GEN8_CXT_TOTAL_SIZE;
+ case 7:
+ if (IS_HASWELL(dev_priv))
+ return HSW_CXT_TOTAL_SIZE;
+
+ cxt_size = I915_READ(GEN7_CXT_SIZE);
+ return round_up(GEN7_CXT_TOTAL_SIZE(cxt_size) * 64,
+ PAGE_SIZE);
+ case 6:
+ cxt_size = I915_READ(CXT_SIZE);
+ return round_up(GEN6_CXT_TOTAL_SIZE(cxt_size) * 64,
+ PAGE_SIZE);
+ case 5:
+ case 4:
+ case 3:
+ case 2:
+ /* For the special day when i810 gets merged. */
+ case 1:
+ return 0;
+ }
+ break;
+ default:
+ MISSING_CASE(class);
+ case VIDEO_DECODE_CLASS:
+ case VIDEO_ENHANCEMENT_CLASS:
+ case COPY_ENGINE_CLASS:
+ if (INTEL_GEN(dev_priv) < 8)
+ return 0;
+ return GEN8_LR_CONTEXT_OTHER_SIZE;
+ }
+}
+
static int
intel_engine_setup(struct drm_i915_private *dev_priv,
enum intel_engine_id id)
{
const struct engine_info *info = &intel_engines[id];
+ const struct engine_class_info *class_info;
struct intel_engine_cs *engine;
+ GEM_BUG_ON(info->class >= ARRAY_SIZE(intel_engine_classes));
+ class_info = &intel_engine_classes[info->class];
+
GEM_BUG_ON(dev_priv->engine[id]);
engine = kzalloc(sizeof(*engine), GFP_KERNEL);
if (!engine)
engine->id = id;
engine->i915 = dev_priv;
- engine->name = info->name;
- engine->exec_id = info->exec_id;
+ WARN_ON(snprintf(engine->name, sizeof(engine->name), "%s%u",
+ class_info->name, info->instance) >=
+ sizeof(engine->name));
+ engine->uabi_id = info->uabi_id;
engine->hw_id = engine->guc_id = info->hw_id;
engine->mmio_base = info->mmio_base;
engine->irq_shift = info->irq_shift;
+ engine->class = info->class;
+ engine->instance = info->instance;
+
+ engine->context_size = __intel_engine_context_size(dev_priv,
+ engine->class);
+ if (WARN_ON(engine->context_size > BIT(20)))
+ engine->context_size = 0;
/* Nothing to do here, execute in order of dependencies */
engine->schedule = NULL;
}
/**
- * intel_engines_init_early() - allocate the Engine Command Streamers
+ * intel_engines_init_mmio() - allocate and prepare the Engine Command Streamers
* @dev_priv: i915 device private
*
* Return: non-zero if the initialization failed.
*/
-int intel_engines_init_early(struct drm_i915_private *dev_priv)
+int intel_engines_init_mmio(struct drm_i915_private *dev_priv)
{
struct intel_device_info *device_info = mkwrite_device_info(dev_priv);
- unsigned int ring_mask = INTEL_INFO(dev_priv)->ring_mask;
- unsigned int mask = 0;
+ const unsigned int ring_mask = INTEL_INFO(dev_priv)->ring_mask;
struct intel_engine_cs *engine;
enum intel_engine_id id;
+ unsigned int mask = 0;
unsigned int i;
int err;
if (WARN_ON(mask != ring_mask))
device_info->ring_mask = mask;
+ /* We always presume we have at least RCS available for later probing */
+ if (WARN_ON(!HAS_ENGINE(dev_priv, RCS))) {
+ err = -ENODEV;
+ goto cleanup;
+ }
+
device_info->num_rings = hweight32(mask);
return 0;
}
/**
- * intel_engines_init() - allocate, populate and init the Engine Command Streamers
+ * intel_engines_init() - init the Engine Command Streamers
* @dev_priv: i915 device private
*
* Return: non-zero if the initialization failed.
int err = 0;
for_each_engine(engine, dev_priv, id) {
+ const struct engine_class_info *class_info =
+ &intel_engine_classes[engine->class];
int (*init)(struct intel_engine_cs *engine);
if (i915.enable_execlists)
- init = intel_engines[id].init_execlists;
+ init = class_info->init_execlists;
else
- init = intel_engines[id].init_legacy;
+ init = class_info->init_legacy;
if (!init) {
kfree(engine);
dev_priv->engine[id] = NULL;
{
struct drm_i915_private *dev_priv = engine->i915;
+ GEM_BUG_ON(!intel_engine_is_idle(engine));
+ GEM_BUG_ON(i915_gem_active_isset(&engine->timeline->last_request));
+
/* Our semaphore implementation is strictly monotonic (i.e. we proceed
* so long as the semaphore value in the register/page is greater
* than the sync value), so whenever we reset the seqno,
intel_write_status_page(engine, I915_GEM_HWS_INDEX, seqno);
clear_bit(ENGINE_IRQ_BREADCRUMB, &engine->irq_posted);
- GEM_BUG_ON(i915_gem_active_isset(&engine->timeline->last_request));
- engine->hangcheck.seqno = seqno;
-
/* After manually advancing the seqno, fake the interrupt in case
* there are any waiters for that seqno.
*/
intel_engine_wakeup(engine);
+
+ GEM_BUG_ON(intel_engine_get_seqno(engine) != seqno);
}
static void intel_engine_init_timeline(struct intel_engine_cs *engine)
*/
int intel_engine_init_common(struct intel_engine_cs *engine)
{
+ struct intel_ring *ring;
int ret;
engine->set_default_submission(engine);
* be available. To avoid this we always pin the default
* context.
*/
- ret = engine->context_pin(engine, engine->i915->kernel_context);
- if (ret)
- return ret;
+ ring = engine->context_pin(engine, engine->i915->kernel_context);
+ if (IS_ERR(ring))
+ return PTR_ERR(ring);
ret = intel_engine_init_breadcrumbs(engine);
if (ret)
*/
}
+ /* WaEnableYV12BugFixInHalfSliceChicken7:skl,bxt,kbl,glk */
/* WaEnableSamplerGPGPUPreemptionSupport:skl,bxt,kbl */
WA_SET_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN7,
+ GEN9_ENABLE_YV12_BUGFIX |
GEN9_ENABLE_GPGPU_PREEMPTION);
/* Wa4x4STCOptimizationDisable:skl,bxt,kbl,glk */
{
struct drm_i915_private *dev_priv = engine->i915;
+ /* More white lies, if wedged, hw state is inconsistent */
+ if (i915_terminally_wedged(&dev_priv->gpu_error))
+ return true;
+
/* Any inflight/incomplete requests? */
if (!i915_seqno_passed(intel_engine_get_seqno(engine),
intel_engine_last_submit(engine)))
return false;
+ if (I915_SELFTEST_ONLY(engine->breadcrumbs.mock))
+ return true;
+
/* Interrupt/tasklet pending? */
if (test_bit(ENGINE_IRQ_EXECLIST, &engine->irq_posted))
return false;
/* Both ports drained, no more ELSP submission? */
- if (engine->execlist_port[0].request)
+ if (port_request(&engine->execlist_port[0]))
return false;
/* Ring stopped? */
engine->set_default_submission(engine);
}
+void intel_engines_mark_idle(struct drm_i915_private *i915)
+{
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+
+ for_each_engine(engine, i915, id) {
+ intel_engine_disarm_breadcrumbs(engine);
+ i915_gem_batch_pool_fini(&engine->batch_pool);
+ engine->no_priolist = false;
+ }
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/mock_engine.c"
#endif
static bool need_fbc_vtd_wa(struct drm_i915_private *dev_priv)
{
-#ifdef CONFIG_INTEL_IOMMU
/* WaFbcTurnOffFbcWhenHyperVisorIsUsed:skl,bxt */
- if (intel_iommu_gfx_mapped &&
+ if (intel_vtd_active() &&
(IS_SKYLAKE(dev_priv) || IS_BROXTON(dev_priv))) {
DRM_INFO("Disabling framebuffer compression (FBC) to prevent screen flicker with VT-d enabled\n");
return true;
}
-#endif
return false;
}
--- /dev/null
+/*
+ * Copyright © 2016-2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#include "i915_drv.h"
+#include "intel_guc_ct.h"
+
+enum { CTB_SEND = 0, CTB_RECV = 1 };
+
+enum { CTB_OWNER_HOST = 0 };
+
+void intel_guc_ct_init_early(struct intel_guc_ct *ct)
+{
+ /* we're using static channel owners */
+ ct->host_channel.owner = CTB_OWNER_HOST;
+}
+
+static inline const char *guc_ct_buffer_type_to_str(u32 type)
+{
+ switch (type) {
+ case INTEL_GUC_CT_BUFFER_TYPE_SEND:
+ return "SEND";
+ case INTEL_GUC_CT_BUFFER_TYPE_RECV:
+ return "RECV";
+ default:
+ return "<invalid>";
+ }
+}
+
+static void guc_ct_buffer_desc_init(struct guc_ct_buffer_desc *desc,
+ u32 cmds_addr, u32 size, u32 owner)
+{
+ DRM_DEBUG_DRIVER("CT: desc %p init addr=%#x size=%u owner=%u\n",
+ desc, cmds_addr, size, owner);
+ memset(desc, 0, sizeof(*desc));
+ desc->addr = cmds_addr;
+ desc->size = size;
+ desc->owner = owner;
+}
+
+static void guc_ct_buffer_desc_reset(struct guc_ct_buffer_desc *desc)
+{
+ DRM_DEBUG_DRIVER("CT: desc %p reset head=%u tail=%u\n",
+ desc, desc->head, desc->tail);
+ desc->head = 0;
+ desc->tail = 0;
+ desc->is_in_error = 0;
+}
+
+static int guc_action_register_ct_buffer(struct intel_guc *guc,
+ u32 desc_addr,
+ u32 type)
+{
+ u32 action[] = {
+ INTEL_GUC_ACTION_REGISTER_COMMAND_TRANSPORT_BUFFER,
+ desc_addr,
+ sizeof(struct guc_ct_buffer_desc),
+ type
+ };
+ int err;
+
+ /* Can't use generic send(), CT registration must go over MMIO */
+ err = intel_guc_send_mmio(guc, action, ARRAY_SIZE(action));
+ if (err)
+ DRM_ERROR("CT: register %s buffer failed; err=%d\n",
+ guc_ct_buffer_type_to_str(type), err);
+ return err;
+}
+
+static int guc_action_deregister_ct_buffer(struct intel_guc *guc,
+ u32 owner,
+ u32 type)
+{
+ u32 action[] = {
+ INTEL_GUC_ACTION_DEREGISTER_COMMAND_TRANSPORT_BUFFER,
+ owner,
+ type
+ };
+ int err;
+
+ /* Can't use generic send(), CT deregistration must go over MMIO */
+ err = intel_guc_send_mmio(guc, action, ARRAY_SIZE(action));
+ if (err)
+ DRM_ERROR("CT: deregister %s buffer failed; owner=%d err=%d\n",
+ guc_ct_buffer_type_to_str(type), owner, err);
+ return err;
+}
+
+static bool ctch_is_open(struct intel_guc_ct_channel *ctch)
+{
+ return ctch->vma != NULL;
+}
+
+static int ctch_init(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch)
+{
+ struct i915_vma *vma;
+ void *blob;
+ int err;
+ int i;
+
+ GEM_BUG_ON(ctch->vma);
+
+ /* We allocate 1 page to hold both descriptors and both buffers.
+ * ___________.....................
+ * |desc (SEND)| :
+ * |___________| PAGE/4
+ * :___________....................:
+ * |desc (RECV)| :
+ * |___________| PAGE/4
+ * :_______________________________:
+ * |cmds (SEND) |
+ * | PAGE/4
+ * |_______________________________|
+ * |cmds (RECV) |
+ * | PAGE/4
+ * |_______________________________|
+ *
+ * Each message can use a maximum of 32 dwords and we don't expect to
+ * have more than 1 in flight at any time, so we have enough space.
+ * Some logic further ahead will rely on the fact that there is only 1
+ * page and that it is always mapped, so if the size is changed the
+ * other code will need updating as well.
+ */
+
+ /* allocate vma */
+ vma = intel_guc_allocate_vma(guc, PAGE_SIZE);
+ if (IS_ERR(vma)) {
+ err = PTR_ERR(vma);
+ goto err_out;
+ }
+ ctch->vma = vma;
+
+ /* map first page */
+ blob = i915_gem_object_pin_map(vma->obj, I915_MAP_WB);
+ if (IS_ERR(blob)) {
+ err = PTR_ERR(blob);
+ goto err_vma;
+ }
+ DRM_DEBUG_DRIVER("CT: vma base=%#x\n", guc_ggtt_offset(ctch->vma));
+
+ /* store pointers to desc and cmds */
+ for (i = 0; i < ARRAY_SIZE(ctch->ctbs); i++) {
+ GEM_BUG_ON((i != CTB_SEND) && (i != CTB_RECV));
+ ctch->ctbs[i].desc = blob + PAGE_SIZE/4 * i;
+ ctch->ctbs[i].cmds = blob + PAGE_SIZE/4 * i + PAGE_SIZE/2;
+ }
+
+ return 0;
+
+err_vma:
+ i915_vma_unpin_and_release(&ctch->vma);
+err_out:
+ DRM_DEBUG_DRIVER("CT: channel %d initialization failed; err=%d\n",
+ ctch->owner, err);
+ return err;
+}
+
+static void ctch_fini(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch)
+{
+ GEM_BUG_ON(!ctch->vma);
+
+ i915_gem_object_unpin_map(ctch->vma->obj);
+ i915_vma_unpin_and_release(&ctch->vma);
+}
+
+static int ctch_open(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch)
+{
+ u32 base;
+ int err;
+ int i;
+
+ DRM_DEBUG_DRIVER("CT: channel %d reopen=%s\n",
+ ctch->owner, yesno(ctch_is_open(ctch)));
+
+ if (!ctch->vma) {
+ err = ctch_init(guc, ctch);
+ if (unlikely(err))
+ goto err_out;
+ }
+
+ /* vma should be already allocated and map'ed */
+ base = guc_ggtt_offset(ctch->vma);
+
+ /* (re)initialize descriptors
+ * cmds buffers are in the second half of the blob page
+ */
+ for (i = 0; i < ARRAY_SIZE(ctch->ctbs); i++) {
+ GEM_BUG_ON((i != CTB_SEND) && (i != CTB_RECV));
+ guc_ct_buffer_desc_init(ctch->ctbs[i].desc,
+ base + PAGE_SIZE/4 * i + PAGE_SIZE/2,
+ PAGE_SIZE/4,
+ ctch->owner);
+ }
+
+ /* register buffers, starting wirh RECV buffer
+ * descriptors are in first half of the blob
+ */
+ err = guc_action_register_ct_buffer(guc,
+ base + PAGE_SIZE/4 * CTB_RECV,
+ INTEL_GUC_CT_BUFFER_TYPE_RECV);
+ if (unlikely(err))
+ goto err_fini;
+
+ err = guc_action_register_ct_buffer(guc,
+ base + PAGE_SIZE/4 * CTB_SEND,
+ INTEL_GUC_CT_BUFFER_TYPE_SEND);
+ if (unlikely(err))
+ goto err_deregister;
+
+ return 0;
+
+err_deregister:
+ guc_action_deregister_ct_buffer(guc,
+ ctch->owner,
+ INTEL_GUC_CT_BUFFER_TYPE_RECV);
+err_fini:
+ ctch_fini(guc, ctch);
+err_out:
+ DRM_ERROR("CT: can't open channel %d; err=%d\n", ctch->owner, err);
+ return err;
+}
+
+static void ctch_close(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch)
+{
+ GEM_BUG_ON(!ctch_is_open(ctch));
+
+ guc_action_deregister_ct_buffer(guc,
+ ctch->owner,
+ INTEL_GUC_CT_BUFFER_TYPE_SEND);
+ guc_action_deregister_ct_buffer(guc,
+ ctch->owner,
+ INTEL_GUC_CT_BUFFER_TYPE_RECV);
+ ctch_fini(guc, ctch);
+}
+
+static u32 ctch_get_next_fence(struct intel_guc_ct_channel *ctch)
+{
+ /* For now it's trivial */
+ return ++ctch->next_fence;
+}
+
+static int ctb_write(struct intel_guc_ct_buffer *ctb,
+ const u32 *action,
+ u32 len /* in dwords */,
+ u32 fence)
+{
+ struct guc_ct_buffer_desc *desc = ctb->desc;
+ u32 head = desc->head / 4; /* in dwords */
+ u32 tail = desc->tail / 4; /* in dwords */
+ u32 size = desc->size / 4; /* in dwords */
+ u32 used; /* in dwords */
+ u32 header;
+ u32 *cmds = ctb->cmds;
+ unsigned int i;
+
+ GEM_BUG_ON(desc->size % 4);
+ GEM_BUG_ON(desc->head % 4);
+ GEM_BUG_ON(desc->tail % 4);
+ GEM_BUG_ON(tail >= size);
+
+ /*
+ * tail == head condition indicates empty. GuC FW does not support
+ * using up the entire buffer to get tail == head meaning full.
+ */
+ if (tail < head)
+ used = (size - head) + tail;
+ else
+ used = tail - head;
+
+ /* make sure there is a space including extra dw for the fence */
+ if (unlikely(used + len + 1 >= size))
+ return -ENOSPC;
+
+ /* Write the message. The format is the following:
+ * DW0: header (including action code)
+ * DW1: fence
+ * DW2+: action data
+ */
+ header = (len << GUC_CT_MSG_LEN_SHIFT) |
+ (GUC_CT_MSG_WRITE_FENCE_TO_DESC) |
+ (action[0] << GUC_CT_MSG_ACTION_SHIFT);
+
+ cmds[tail] = header;
+ tail = (tail + 1) % size;
+
+ cmds[tail] = fence;
+ tail = (tail + 1) % size;
+
+ for (i = 1; i < len; i++) {
+ cmds[tail] = action[i];
+ tail = (tail + 1) % size;
+ }
+
+ /* now update desc tail (back in bytes) */
+ desc->tail = tail * 4;
+ GEM_BUG_ON(desc->tail > desc->size);
+
+ return 0;
+}
+
+/* Wait for the response from the GuC.
+ * @fence: response fence
+ * @status: placeholder for status
+ * return: 0 response received (status is valid)
+ * -ETIMEDOUT no response within hardcoded timeout
+ * -EPROTO no response, ct buffer was in error
+ */
+static int wait_for_response(struct guc_ct_buffer_desc *desc,
+ u32 fence,
+ u32 *status)
+{
+ int err;
+
+ /*
+ * Fast commands should complete in less than 10us, so sample quickly
+ * up to that length of time, then switch to a slower sleep-wait loop.
+ * No GuC command should ever take longer than 10ms.
+ */
+#define done (READ_ONCE(desc->fence) == fence)
+ err = wait_for_us(done, 10);
+ if (err)
+ err = wait_for(done, 10);
+#undef done
+
+ if (unlikely(err)) {
+ DRM_ERROR("CT: fence %u failed; reported fence=%u\n",
+ fence, desc->fence);
+
+ if (WARN_ON(desc->is_in_error)) {
+ /* Something went wrong with the messaging, try to reset
+ * the buffer and hope for the best
+ */
+ guc_ct_buffer_desc_reset(desc);
+ err = -EPROTO;
+ }
+ }
+
+ *status = desc->status;
+ return err;
+}
+
+static int ctch_send(struct intel_guc *guc,
+ struct intel_guc_ct_channel *ctch,
+ const u32 *action,
+ u32 len,
+ u32 *status)
+{
+ struct intel_guc_ct_buffer *ctb = &ctch->ctbs[CTB_SEND];
+ struct guc_ct_buffer_desc *desc = ctb->desc;
+ u32 fence;
+ int err;
+
+ GEM_BUG_ON(!ctch_is_open(ctch));
+ GEM_BUG_ON(!len);
+ GEM_BUG_ON(len & ~GUC_CT_MSG_LEN_MASK);
+
+ fence = ctch_get_next_fence(ctch);
+ err = ctb_write(ctb, action, len, fence);
+ if (unlikely(err))
+ return err;
+
+ intel_guc_notify(guc);
+
+ err = wait_for_response(desc, fence, status);
+ if (unlikely(err))
+ return err;
+ if (*status != INTEL_GUC_STATUS_SUCCESS)
+ return -EIO;
+ return 0;
+}
+
+/*
+ * Command Transport (CT) buffer based GuC send function.
+ */
+static int intel_guc_send_ct(struct intel_guc *guc, const u32 *action, u32 len)
+{
+ struct intel_guc_ct_channel *ctch = &guc->ct.host_channel;
+ u32 status = ~0; /* undefined */
+ int err;
+
+ mutex_lock(&guc->send_mutex);
+
+ err = ctch_send(guc, ctch, action, len, &status);
+ if (unlikely(err)) {
+ DRM_ERROR("CT: send action %#X failed; err=%d status=%#X\n",
+ action[0], err, status);
+ }
+
+ mutex_unlock(&guc->send_mutex);
+ return err;
+}
+
+/**
+ * Enable buffer based command transport
+ * Shall only be called for platforms with HAS_GUC_CT.
+ * @guc: the guc
+ * return: 0 on success
+ * non-zero on failure
+ */
+int intel_guc_enable_ct(struct intel_guc *guc)
+{
+ struct drm_i915_private *dev_priv = guc_to_i915(guc);
+ struct intel_guc_ct_channel *ctch = &guc->ct.host_channel;
+ int err;
+
+ GEM_BUG_ON(!HAS_GUC_CT(dev_priv));
+
+ err = ctch_open(guc, ctch);
+ if (unlikely(err))
+ return err;
+
+ /* Switch into cmd transport buffer based send() */
+ guc->send = intel_guc_send_ct;
+ DRM_INFO("CT: %s\n", enableddisabled(true));
+ return 0;
+}
+
+/**
+ * Disable buffer based command transport.
+ * Shall only be called for platforms with HAS_GUC_CT.
+ * @guc: the guc
+ */
+void intel_guc_disable_ct(struct intel_guc *guc)
+{
+ struct drm_i915_private *dev_priv = guc_to_i915(guc);
+ struct intel_guc_ct_channel *ctch = &guc->ct.host_channel;
+
+ GEM_BUG_ON(!HAS_GUC_CT(dev_priv));
+
+ if (!ctch_is_open(ctch))
+ return;
+
+ ctch_close(guc, ctch);
+
+ /* Disable send */
+ guc->send = intel_guc_send_nop;
+ DRM_INFO("CT: %s\n", enableddisabled(false));
+}
--- /dev/null
+/*
+ * Copyright © 2016-2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#ifndef _INTEL_GUC_CT_H_
+#define _INTEL_GUC_CT_H_
+
+struct intel_guc;
+struct i915_vma;
+
+#include "intel_guc_fwif.h"
+
+/**
+ * DOC: Command Transport (CT).
+ *
+ * Buffer based command transport is a replacement for MMIO based mechanism.
+ * It can be used to perform both host-2-guc and guc-to-host communication.
+ */
+
+/** Represents single command transport buffer.
+ *
+ * A single command transport buffer consists of two parts, the header
+ * record (command transport buffer descriptor) and the actual buffer which
+ * holds the commands.
+ *
+ * @desc: pointer to the buffer descriptor
+ * @cmds: pointer to the commands buffer
+ */
+struct intel_guc_ct_buffer {
+ struct guc_ct_buffer_desc *desc;
+ u32 *cmds;
+};
+
+/** Represents pair of command transport buffers.
+ *
+ * Buffers go in pairs to allow bi-directional communication.
+ * To simplify the code we place both of them in the same vma.
+ * Buffers from the same pair must share unique owner id.
+ *
+ * @vma: pointer to the vma with pair of CT buffers
+ * @ctbs: buffers for sending(0) and receiving(1) commands
+ * @owner: unique identifier
+ * @next_fence: fence to be used with next send command
+ */
+struct intel_guc_ct_channel {
+ struct i915_vma *vma;
+ struct intel_guc_ct_buffer ctbs[2];
+ u32 owner;
+ u32 next_fence;
+};
+
+/** Holds all command transport channels.
+ *
+ * @host_channel: main channel used by the host
+ */
+struct intel_guc_ct {
+ struct intel_guc_ct_channel host_channel;
+ /* other channels are tbd */
+};
+
+void intel_guc_ct_init_early(struct intel_guc_ct *ct);
+
+/* XXX: move to intel_uc.h ? don't fit there either */
+int intel_guc_enable_ct(struct intel_guc *guc);
+void intel_guc_disable_ct(struct intel_guc *guc);
+
+#endif /* _INTEL_GUC_CT_H_ */
#ifndef _INTEL_GUC_FWIF_H
#define _INTEL_GUC_FWIF_H
-#define GFXCORE_FAMILY_GEN9 12
-#define GFXCORE_FAMILY_UNKNOWN 0x7fffffff
+#define GUC_CORE_FAMILY_GEN9 12
+#define GUC_CORE_FAMILY_UNKNOWN 0x7fffffff
#define GUC_CLIENT_PRIORITY_KMD_HIGH 0
#define GUC_CLIENT_PRIORITY_HIGH 1
u64 desc_private;
} __packed;
+/*
+ * Describes single command transport buffer.
+ * Used by both guc-master and clients.
+ */
+struct guc_ct_buffer_desc {
+ u32 addr; /* gfx address */
+ u64 host_private; /* host private data */
+ u32 size; /* size in bytes */
+ u32 head; /* offset updated by GuC*/
+ u32 tail; /* offset updated by owner */
+ u32 is_in_error; /* error indicator */
+ u32 fence; /* fence updated by GuC */
+ u32 status; /* status updated by GuC */
+ u32 owner; /* id of the channel owner */
+ u32 owner_sub_id; /* owner-defined field for extra tracking */
+ u32 reserved[5];
+} __packed;
+
+/* Type of command transport buffer */
+#define INTEL_GUC_CT_BUFFER_TYPE_SEND 0x0u
+#define INTEL_GUC_CT_BUFFER_TYPE_RECV 0x1u
+
+/*
+ * Definition of the command transport message header (DW0)
+ *
+ * bit[4..0] message len (in dwords)
+ * bit[7..5] reserved
+ * bit[8] write fence to desc
+ * bit[9] write status to H2G buff
+ * bit[10] send status (via G2H)
+ * bit[15..11] reserved
+ * bit[31..16] action code
+ */
+#define GUC_CT_MSG_LEN_SHIFT 0
+#define GUC_CT_MSG_LEN_MASK 0x1F
+#define GUC_CT_MSG_WRITE_FENCE_TO_DESC (1 << 8)
+#define GUC_CT_MSG_WRITE_STATUS_TO_BUFF (1 << 9)
+#define GUC_CT_MSG_SEND_STATUS (1 << 10)
+#define GUC_CT_MSG_ACTION_SHIFT 16
+#define GUC_CT_MSG_ACTION_MASK 0xFFFF
+
#define GUC_FORCEWAKE_RENDER (1 << 0)
#define GUC_FORCEWAKE_MEDIA (1 << 1)
INTEL_GUC_ACTION_EXIT_S_STATE = 0x502,
INTEL_GUC_ACTION_SLPC_REQUEST = 0x3003,
INTEL_GUC_ACTION_AUTHENTICATE_HUC = 0x4000,
+ INTEL_GUC_ACTION_REGISTER_COMMAND_TRANSPORT_BUFFER = 0x4505,
+ INTEL_GUC_ACTION_DEREGISTER_COMMAND_TRANSPORT_BUFFER = 0x4506,
INTEL_GUC_ACTION_UK_LOG_ENABLE_LOGGING = 0x0E000,
INTEL_GUC_ACTION_LIMIT
};
#define KBL_FW_MAJOR 9
#define KBL_FW_MINOR 14
+#define GLK_FW_MAJOR 10
+#define GLK_FW_MINOR 56
+
#define GUC_FW_PATH(platform, major, minor) \
"i915/" __stringify(platform) "_guc_ver" __stringify(major) "_" __stringify(minor) ".bin"
#define I915_KBL_GUC_UCODE GUC_FW_PATH(kbl, KBL_FW_MAJOR, KBL_FW_MINOR)
MODULE_FIRMWARE(I915_KBL_GUC_UCODE);
+#define I915_GLK_GUC_UCODE GUC_FW_PATH(glk, GLK_FW_MAJOR, GLK_FW_MINOR)
+
static u32 get_gttype(struct drm_i915_private *dev_priv)
{
switch (gen) {
case 9:
- return GFXCORE_FAMILY_GEN9;
+ return GUC_CORE_FAMILY_GEN9;
default:
- WARN(1, "GEN%d does not support GuC operation!\n", gen);
- return GFXCORE_FAMILY_UNKNOWN;
+ MISSING_CASE(gen);
+ return GUC_CORE_FAMILY_UNKNOWN;
}
}
intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
- /* init WOPCM */
- I915_WRITE(GUC_WOPCM_SIZE, intel_guc_wopcm_size(dev_priv));
- I915_WRITE(DMA_GUC_WOPCM_OFFSET, GUC_WOPCM_OFFSET_VALUE);
-
/* Enable MIA caching. GuC clock gating is disabled. */
I915_WRITE(GUC_SHIM_CONTROL, GUC_SHIM_CONTROL_VALUE);
guc->fw.path = I915_KBL_GUC_UCODE;
guc->fw.major_ver_wanted = KBL_FW_MAJOR;
guc->fw.minor_ver_wanted = KBL_FW_MINOR;
+ } else if (IS_GEMINILAKE(dev_priv)) {
+ guc->fw.path = I915_GLK_GUC_UCODE;
+ guc->fw.major_ver_wanted = GLK_FW_MAJOR;
+ guc->fw.minor_ver_wanted = GLK_FW_MINOR;
} else {
DRM_ERROR("No GuC firmware known for platform with GuC!\n");
return -ENOENT;
void *vaddr;
struct rchan *guc_log_relay_chan;
size_t n_subbufs, subbuf_size;
- int ret = 0;
+ int ret;
lockdep_assert_held(&dev_priv->drm.struct_mutex);
GEM_BUG_ON(guc_log_has_runtime(guc));
+ ret = i915_gem_object_set_to_wc_domain(guc->log.vma->obj, true);
+ if (ret)
+ return ret;
+
/* Create a WC (Uncached for read) vmalloc mapping of log
* buffer pages, so that we can directly get the data
* (up-to-date) from memory.
"%s, ", engine->name);
msg[len-2] = '\0';
- return i915_handle_error(i915, hung, msg);
+ return i915_handle_error(i915, hung, "%s", msg);
}
/*
return false;
}
+ /* Display Wa #1139 */
+ if (IS_GLK_REVID(dev_priv, 0, GLK_REVID_A1) &&
+ crtc_state->base.adjusted_mode.htotal > 5460)
+ return false;
+
return true;
}
}
if (!pipe_config->bw_constrained) {
- DRM_DEBUG_KMS("forcing pipe bpc to %i for HDMI\n", desired_bpp);
+ DRM_DEBUG_KMS("forcing pipe bpp to %i for HDMI\n", desired_bpp);
pipe_config->pipe_bpp = desired_bpp;
}
#define KBL_HUC_FW_MINOR 00
#define KBL_BLD_NUM 1810
+#define GLK_HUC_FW_MAJOR 02
+#define GLK_HUC_FW_MINOR 00
+#define GLK_BLD_NUM 1748
+
#define HUC_FW_PATH(platform, major, minor, bld_num) \
"i915/" __stringify(platform) "_huc_ver" __stringify(major) "_" \
__stringify(minor) "_" __stringify(bld_num) ".bin"
KBL_HUC_FW_MINOR, KBL_BLD_NUM)
MODULE_FIRMWARE(I915_KBL_HUC_UCODE);
+#define I915_GLK_HUC_UCODE HUC_FW_PATH(glk, GLK_HUC_FW_MAJOR, \
+ GLK_HUC_FW_MINOR, GLK_BLD_NUM)
+
/**
* huc_ucode_xfer() - DMA's the firmware
* @dev_priv: the drm_i915_private device
intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
- /* init WOPCM */
- I915_WRITE(GUC_WOPCM_SIZE, intel_guc_wopcm_size(dev_priv));
- I915_WRITE(DMA_GUC_WOPCM_OFFSET, GUC_WOPCM_OFFSET_VALUE |
- HUC_LOADING_AGENT_GUC);
-
/* Set the source address for the uCode */
offset = guc_ggtt_offset(vma) + huc_fw->header_offset;
I915_WRITE(DMA_ADDR_0_LOW, lower_32_bits(offset));
huc->fw.path = I915_KBL_HUC_UCODE;
huc->fw.major_ver_wanted = KBL_HUC_FW_MAJOR;
huc->fw.minor_ver_wanted = KBL_HUC_FW_MINOR;
+ } else if (IS_GEMINILAKE(dev_priv)) {
+ huc->fw.path = I915_GLK_HUC_UCODE;
+ huc->fw.major_ver_wanted = GLK_HUC_FW_MAJOR;
+ huc->fw.minor_ver_wanted = GLK_HUC_FW_MINOR;
} else {
DRM_ERROR("No HuC firmware known for platform with HuC!\n");
return;
* earlier call to intel_huc_init(), so here we need only check that
* is succeeded, and then transfer the image to the h/w.
*
- * Return: non-zero code on error
*/
-int intel_huc_init_hw(struct intel_huc *huc)
+void intel_huc_init_hw(struct intel_huc *huc)
{
struct drm_i915_private *dev_priv = huc_to_i915(huc);
int err;
- if (huc->fw.fetch_status == INTEL_UC_FIRMWARE_NONE)
- return 0;
-
DRM_DEBUG_DRIVER("%s fw status: fetch %s, load %s\n",
huc->fw.path,
intel_uc_fw_status_repr(huc->fw.fetch_status),
intel_uc_fw_status_repr(huc->fw.load_status));
- if (huc->fw.fetch_status == INTEL_UC_FIRMWARE_SUCCESS &&
- huc->fw.load_status == INTEL_UC_FIRMWARE_FAIL)
- return -ENOEXEC;
+ if (huc->fw.fetch_status != INTEL_UC_FIRMWARE_SUCCESS)
+ return;
huc->fw.load_status = INTEL_UC_FIRMWARE_PENDING;
- switch (huc->fw.fetch_status) {
- case INTEL_UC_FIRMWARE_FAIL:
- /* something went wrong :( */
- err = -EIO;
- goto fail;
-
- case INTEL_UC_FIRMWARE_NONE:
- case INTEL_UC_FIRMWARE_PENDING:
- default:
- /* "can't happen" */
- WARN_ONCE(1, "HuC fw %s invalid fetch_status %s [%d]\n",
- huc->fw.path,
- intel_uc_fw_status_repr(huc->fw.fetch_status),
- huc->fw.fetch_status);
- err = -ENXIO;
- goto fail;
-
- case INTEL_UC_FIRMWARE_SUCCESS:
- break;
- }
-
err = huc_ucode_xfer(dev_priv);
- if (err)
- goto fail;
- huc->fw.load_status = INTEL_UC_FIRMWARE_SUCCESS;
+ huc->fw.load_status = err ?
+ INTEL_UC_FIRMWARE_FAIL : INTEL_UC_FIRMWARE_SUCCESS;
DRM_DEBUG_DRIVER("%s fw status: fetch %s, load %s\n",
huc->fw.path,
intel_uc_fw_status_repr(huc->fw.fetch_status),
intel_uc_fw_status_repr(huc->fw.load_status));
- return 0;
-
-fail:
- if (huc->fw.load_status == INTEL_UC_FIRMWARE_PENDING)
- huc->fw.load_status = INTEL_UC_FIRMWARE_FAIL;
-
- DRM_ERROR("Failed to complete HuC uCode load with ret %d\n", err);
+ if (huc->fw.load_status != INTEL_UC_FIRMWARE_SUCCESS)
+ DRM_ERROR("Failed to complete HuC uCode load with ret %d\n", err);
- return err;
+ return;
}
/**
#include <linux/acpi.h>
#include <linux/device.h>
#include <linux/pci.h>
+#include <linux/pm_runtime.h>
#include "i915_drv.h"
#include <linux/delay.h>
pinfo.size_data = sizeof(*pdata);
pinfo.dma_mask = DMA_BIT_MASK(32);
+ pdata->num_pipes = INTEL_INFO(dev_priv)->num_pipes;
+ pdata->num_ports = IS_CHERRYVIEW(dev_priv) ? 3 : 2; /* B,C,D or B,C */
+ pdata->port[0].pipe = -1;
+ pdata->port[1].pipe = -1;
+ pdata->port[2].pipe = -1;
spin_lock_init(&pdata->lpe_audio_slock);
platdev = platform_device_register_full(&pinfo);
kfree(rsc);
+ pm_runtime_forbid(&platdev->dev);
+ pm_runtime_set_active(&platdev->dev);
+ pm_runtime_enable(&platdev->dev);
+
return platdev;
err:
static void lpe_audio_irq_unmask(struct irq_data *d)
{
- struct drm_i915_private *dev_priv = d->chip_data;
- unsigned long irqflags;
- u32 val = (I915_LPE_PIPE_A_INTERRUPT |
- I915_LPE_PIPE_B_INTERRUPT);
-
- if (IS_CHERRYVIEW(dev_priv))
- val |= I915_LPE_PIPE_C_INTERRUPT;
-
- spin_lock_irqsave(&dev_priv->irq_lock, irqflags);
-
- dev_priv->irq_mask &= ~val;
- I915_WRITE(VLV_IIR, val);
- I915_WRITE(VLV_IIR, val);
- I915_WRITE(VLV_IMR, dev_priv->irq_mask);
- POSTING_READ(VLV_IMR);
-
- spin_unlock_irqrestore(&dev_priv->irq_lock, irqflags);
}
static void lpe_audio_irq_mask(struct irq_data *d)
{
- struct drm_i915_private *dev_priv = d->chip_data;
- unsigned long irqflags;
- u32 val = (I915_LPE_PIPE_A_INTERRUPT |
- I915_LPE_PIPE_B_INTERRUPT);
-
- if (IS_CHERRYVIEW(dev_priv))
- val |= I915_LPE_PIPE_C_INTERRUPT;
-
- spin_lock_irqsave(&dev_priv->irq_lock, irqflags);
-
- dev_priv->irq_mask |= val;
- I915_WRITE(VLV_IMR, dev_priv->irq_mask);
- I915_WRITE(VLV_IIR, val);
- I915_WRITE(VLV_IIR, val);
- POSTING_READ(VLV_IIR);
-
- spin_unlock_irqrestore(&dev_priv->irq_lock, irqflags);
}
static struct irq_chip lpe_audio_irqchip = {
desc = irq_to_desc(dev_priv->lpe_audio.irq);
- lpe_audio_irq_mask(&desc->irq_data);
-
lpe_audio_platdev_destroy(dev_priv);
irq_free_desc(dev_priv->lpe_audio.irq);
* intel_lpe_audio_notify() - notify lpe audio event
* audio driver and i915
* @dev_priv: the i915 drm device private data
+ * @pipe: pipe
+ * @port: port
* @eld : ELD data
- * @pipe: pipe id
- * @port: port id
- * @tmds_clk_speed: tmds clock frequency in Hz
+ * @ls_clock: Link symbol clock in kHz
+ * @dp_output: Driving a DP output?
*
* Notify lpe audio driver of eld change.
*/
void intel_lpe_audio_notify(struct drm_i915_private *dev_priv,
- void *eld, int port, int pipe, int tmds_clk_speed,
- bool dp_output, int link_rate)
+ enum pipe pipe, enum port port,
+ const void *eld, int ls_clock, bool dp_output)
{
- unsigned long irq_flags;
- struct intel_hdmi_lpe_audio_pdata *pdata = NULL;
+ unsigned long irqflags;
+ struct intel_hdmi_lpe_audio_pdata *pdata;
+ struct intel_hdmi_lpe_audio_port_pdata *ppdata;
u32 audio_enable;
if (!HAS_LPE_AUDIO(dev_priv))
return;
- pdata = dev_get_platdata(
- &(dev_priv->lpe_audio.platdev->dev));
+ pdata = dev_get_platdata(&dev_priv->lpe_audio.platdev->dev);
+ ppdata = &pdata->port[port - PORT_B];
- spin_lock_irqsave(&pdata->lpe_audio_slock, irq_flags);
+ spin_lock_irqsave(&pdata->lpe_audio_slock, irqflags);
audio_enable = I915_READ(VLV_AUD_PORT_EN_DBG(port));
if (eld != NULL) {
- memcpy(pdata->eld.eld_data, eld,
- HDMI_MAX_ELD_BYTES);
- pdata->eld.port_id = port;
- pdata->eld.pipe_id = pipe;
- pdata->hdmi_connected = true;
-
- pdata->dp_output = dp_output;
- if (tmds_clk_speed)
- pdata->tmds_clock_speed = tmds_clk_speed;
- if (link_rate)
- pdata->link_rate = link_rate;
+ memcpy(ppdata->eld, eld, HDMI_MAX_ELD_BYTES);
+ ppdata->pipe = pipe;
+ ppdata->ls_clock = ls_clock;
+ ppdata->dp_output = dp_output;
/* Unmute the amp for both DP and HDMI */
I915_WRITE(VLV_AUD_PORT_EN_DBG(port),
audio_enable & ~VLV_AMP_MUTE);
-
} else {
- memset(pdata->eld.eld_data, 0,
- HDMI_MAX_ELD_BYTES);
- pdata->hdmi_connected = false;
- pdata->dp_output = false;
+ memset(ppdata->eld, 0, HDMI_MAX_ELD_BYTES);
+ ppdata->pipe = -1;
+ ppdata->ls_clock = 0;
+ ppdata->dp_output = false;
/* Mute the amp for both DP and HDMI */
I915_WRITE(VLV_AUD_PORT_EN_DBG(port),
}
if (pdata->notify_audio_lpe)
- pdata->notify_audio_lpe(dev_priv->lpe_audio.platdev);
- else
- pdata->notify_pending = true;
+ pdata->notify_audio_lpe(dev_priv->lpe_audio.platdev, port - PORT_B);
- spin_unlock_irqrestore(&pdata->lpe_audio_slock,
- irq_flags);
+ spin_unlock_irqrestore(&pdata->lpe_audio_slock, irqflags);
}
#include "i915_drv.h"
#include "intel_mocs.h"
-#define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE)
-#define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE)
-#define GEN8_LR_CONTEXT_OTHER_SIZE (2 * PAGE_SIZE)
-
#define RING_EXECLIST_QFULL (1 << 0x2)
#define RING_EXECLIST1_VALID (1 << 0x3)
#define RING_EXECLIST0_VALID (1 << 0x4)
rq->ctx->ppgtt ?: rq->i915->mm.aliasing_ppgtt;
u32 *reg_state = ce->lrc_reg_state;
- assert_ring_tail_valid(rq->ring, rq->tail);
- reg_state[CTX_RING_TAIL+1] = rq->tail;
+ reg_state[CTX_RING_TAIL+1] = intel_ring_set_tail(rq->ring, rq->tail);
/* True 32b PPGTT with dynamic page allocation: update PDP
* registers and point the unallocated PDPs to scratch page.
static void execlists_submit_ports(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
struct execlist_port *port = engine->execlist_port;
u32 __iomem *elsp =
- dev_priv->regs + i915_mmio_reg_offset(RING_ELSP(engine));
- u64 desc[2];
-
- GEM_BUG_ON(port[0].count > 1);
- if (!port[0].count)
- execlists_context_status_change(port[0].request,
- INTEL_CONTEXT_SCHEDULE_IN);
- desc[0] = execlists_update_context(port[0].request);
- GEM_DEBUG_EXEC(port[0].context_id = upper_32_bits(desc[0]));
- port[0].count++;
-
- if (port[1].request) {
- GEM_BUG_ON(port[1].count);
- execlists_context_status_change(port[1].request,
- INTEL_CONTEXT_SCHEDULE_IN);
- desc[1] = execlists_update_context(port[1].request);
- GEM_DEBUG_EXEC(port[1].context_id = upper_32_bits(desc[1]));
- port[1].count = 1;
- } else {
- desc[1] = 0;
- }
- GEM_BUG_ON(desc[0] == desc[1]);
-
- /* You must always write both descriptors in the order below. */
- writel(upper_32_bits(desc[1]), elsp);
- writel(lower_32_bits(desc[1]), elsp);
+ engine->i915->regs + i915_mmio_reg_offset(RING_ELSP(engine));
+ unsigned int n;
+
+ for (n = ARRAY_SIZE(engine->execlist_port); n--; ) {
+ struct drm_i915_gem_request *rq;
+ unsigned int count;
+ u64 desc;
+
+ rq = port_unpack(&port[n], &count);
+ if (rq) {
+ GEM_BUG_ON(count > !n);
+ if (!count++)
+ execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_IN);
+ port_set(&port[n], port_pack(rq, count));
+ desc = execlists_update_context(rq);
+ GEM_DEBUG_EXEC(port[n].context_id = upper_32_bits(desc));
+ } else {
+ GEM_BUG_ON(!n);
+ desc = 0;
+ }
- writel(upper_32_bits(desc[0]), elsp);
- /* The context is automatically loaded after the following */
- writel(lower_32_bits(desc[0]), elsp);
+ writel(upper_32_bits(desc), elsp);
+ writel(lower_32_bits(desc), elsp);
+ }
}
static bool ctx_single_port_submission(const struct i915_gem_context *ctx)
return true;
}
+static void port_assign(struct execlist_port *port,
+ struct drm_i915_gem_request *rq)
+{
+ GEM_BUG_ON(rq == port_request(port));
+
+ if (port_isset(port))
+ i915_gem_request_put(port_request(port));
+
+ port_set(port, port_pack(i915_gem_request_get(rq), port_count(port)));
+}
+
static void execlists_dequeue(struct intel_engine_cs *engine)
{
struct drm_i915_gem_request *last;
struct rb_node *rb;
bool submit = false;
- last = port->request;
+ last = port_request(port);
if (last)
/* WaIdleLiteRestore:bdw,skl
* Apply the wa NOOPs to prevent ring:HEAD == req:TAIL
*/
last->tail = last->wa_tail;
- GEM_BUG_ON(port[1].request);
+ GEM_BUG_ON(port_isset(&port[1]));
/* Hardware submission is through 2 ports. Conceptually each port
* has a (RING_START, RING_HEAD, RING_TAIL) tuple. RING_START is
spin_lock_irq(&engine->timeline->lock);
rb = engine->execlist_first;
+ GEM_BUG_ON(rb_first(&engine->execlist_queue) != rb);
while (rb) {
- struct drm_i915_gem_request *cursor =
- rb_entry(rb, typeof(*cursor), priotree.node);
-
- /* Can we combine this request with the current port? It has to
- * be the same context/ringbuffer and not have any exceptions
- * (e.g. GVT saying never to combine contexts).
- *
- * If we can combine the requests, we can execute both by
- * updating the RING_TAIL to point to the end of the second
- * request, and so we never need to tell the hardware about
- * the first.
- */
- if (last && !can_merge_ctx(cursor->ctx, last->ctx)) {
- /* If we are on the second port and cannot combine
- * this request with the last, then we are done.
- */
- if (port != engine->execlist_port)
- break;
-
- /* If GVT overrides us we only ever submit port[0],
- * leaving port[1] empty. Note that we also have
- * to be careful that we don't queue the same
- * context (even though a different request) to
- * the second port.
+ struct i915_priolist *p = rb_entry(rb, typeof(*p), node);
+ struct drm_i915_gem_request *rq, *rn;
+
+ list_for_each_entry_safe(rq, rn, &p->requests, priotree.link) {
+ /*
+ * Can we combine this request with the current port?
+ * It has to be the same context/ringbuffer and not
+ * have any exceptions (e.g. GVT saying never to
+ * combine contexts).
+ *
+ * If we can combine the requests, we can execute both
+ * by updating the RING_TAIL to point to the end of the
+ * second request, and so we never need to tell the
+ * hardware about the first.
*/
- if (ctx_single_port_submission(last->ctx) ||
- ctx_single_port_submission(cursor->ctx))
- break;
+ if (last && !can_merge_ctx(rq->ctx, last->ctx)) {
+ /*
+ * If we are on the second port and cannot
+ * combine this request with the last, then we
+ * are done.
+ */
+ if (port != engine->execlist_port) {
+ __list_del_many(&p->requests,
+ &rq->priotree.link);
+ goto done;
+ }
+
+ /*
+ * If GVT overrides us we only ever submit
+ * port[0], leaving port[1] empty. Note that we
+ * also have to be careful that we don't queue
+ * the same context (even though a different
+ * request) to the second port.
+ */
+ if (ctx_single_port_submission(last->ctx) ||
+ ctx_single_port_submission(rq->ctx)) {
+ __list_del_many(&p->requests,
+ &rq->priotree.link);
+ goto done;
+ }
+
+ GEM_BUG_ON(last->ctx == rq->ctx);
+
+ if (submit)
+ port_assign(port, last);
+ port++;
+ }
- GEM_BUG_ON(last->ctx == cursor->ctx);
+ INIT_LIST_HEAD(&rq->priotree.link);
+ rq->priotree.priority = INT_MAX;
- i915_gem_request_assign(&port->request, last);
- port++;
+ __i915_gem_request_submit(rq);
+ trace_i915_gem_request_in(rq, port_index(port, engine));
+ last = rq;
+ submit = true;
}
rb = rb_next(rb);
- rb_erase(&cursor->priotree.node, &engine->execlist_queue);
- RB_CLEAR_NODE(&cursor->priotree.node);
- cursor->priotree.priority = INT_MAX;
-
- __i915_gem_request_submit(cursor);
- trace_i915_gem_request_in(cursor, port - engine->execlist_port);
- last = cursor;
- submit = true;
- }
- if (submit) {
- i915_gem_request_assign(&port->request, last);
- engine->execlist_first = rb;
+ rb_erase(&p->node, &engine->execlist_queue);
+ INIT_LIST_HEAD(&p->requests);
+ if (p->priority != I915_PRIORITY_NORMAL)
+ kmem_cache_free(engine->i915->priorities, p);
}
+done:
+ engine->execlist_first = rb;
+ if (submit)
+ port_assign(port, last);
spin_unlock_irq(&engine->timeline->lock);
if (submit)
execlists_submit_ports(engine);
}
-static bool execlists_elsp_idle(struct intel_engine_cs *engine)
-{
- return !engine->execlist_port[0].request;
-}
-
static bool execlists_elsp_ready(const struct intel_engine_cs *engine)
{
const struct execlist_port *port = engine->execlist_port;
- return port[0].count + port[1].count < 2;
+ return port_count(&port[0]) + port_count(&port[1]) < 2;
}
/*
struct execlist_port *port = engine->execlist_port;
struct drm_i915_private *dev_priv = engine->i915;
+ /* We can skip acquiring intel_runtime_pm_get() here as it was taken
+ * on our behalf by the request (see i915_gem_mark_busy()) and it will
+ * not be relinquished until the device is idle (see
+ * i915_gem_idle_work_handler()). As a precaution, we make sure
+ * that all ELSP are drained i.e. we have processed the CSB,
+ * before allowing ourselves to idle and calling intel_runtime_pm_put().
+ */
+ GEM_BUG_ON(!dev_priv->gt.awake);
+
intel_uncore_forcewake_get(dev_priv, engine->fw_domains);
/* Prefer doing test_and_clear_bit() as a two stage operation to avoid
tail = GEN8_CSB_WRITE_PTR(head);
head = GEN8_CSB_READ_PTR(head);
while (head != tail) {
+ struct drm_i915_gem_request *rq;
unsigned int status;
+ unsigned int count;
if (++head == GEN8_CSB_ENTRIES)
head = 0;
/* Check the context/desc id for this event matches */
GEM_DEBUG_BUG_ON(readl(buf + 2 * head + 1) !=
- port[0].context_id);
+ port->context_id);
- GEM_BUG_ON(port[0].count == 0);
- if (--port[0].count == 0) {
+ rq = port_unpack(port, &count);
+ GEM_BUG_ON(count == 0);
+ if (--count == 0) {
GEM_BUG_ON(status & GEN8_CTX_STATUS_PREEMPTED);
- GEM_BUG_ON(!i915_gem_request_completed(port[0].request));
- execlists_context_status_change(port[0].request,
- INTEL_CONTEXT_SCHEDULE_OUT);
+ GEM_BUG_ON(!i915_gem_request_completed(rq));
+ execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_OUT);
+
+ trace_i915_gem_request_out(rq);
+ i915_gem_request_put(rq);
- trace_i915_gem_request_out(port[0].request);
- i915_gem_request_put(port[0].request);
port[0] = port[1];
memset(&port[1], 0, sizeof(port[1]));
+ } else {
+ port_set(port, port_pack(rq, count));
}
- GEM_BUG_ON(port[0].count == 0 &&
+ /* After the final element, the hw should be idle */
+ GEM_BUG_ON(port_count(port) == 0 &&
!(status & GEN8_CTX_STATUS_ACTIVE_IDLE));
}
intel_uncore_forcewake_put(dev_priv, engine->fw_domains);
}
-static bool insert_request(struct i915_priotree *pt, struct rb_root *root)
+static bool
+insert_request(struct intel_engine_cs *engine,
+ struct i915_priotree *pt,
+ int prio)
{
- struct rb_node **p, *rb;
+ struct i915_priolist *p;
+ struct rb_node **parent, *rb;
bool first = true;
+ if (unlikely(engine->no_priolist))
+ prio = I915_PRIORITY_NORMAL;
+
+find_priolist:
/* most positive priority is scheduled first, equal priorities fifo */
rb = NULL;
- p = &root->rb_node;
- while (*p) {
- struct i915_priotree *pos;
-
- rb = *p;
- pos = rb_entry(rb, typeof(*pos), node);
- if (pt->priority > pos->priority) {
- p = &rb->rb_left;
- } else {
- p = &rb->rb_right;
+ parent = &engine->execlist_queue.rb_node;
+ while (*parent) {
+ rb = *parent;
+ p = rb_entry(rb, typeof(*p), node);
+ if (prio > p->priority) {
+ parent = &rb->rb_left;
+ } else if (prio < p->priority) {
+ parent = &rb->rb_right;
first = false;
+ } else {
+ list_add_tail(&pt->link, &p->requests);
+ return false;
}
}
- rb_link_node(&pt->node, rb, p);
- rb_insert_color(&pt->node, root);
+
+ if (prio == I915_PRIORITY_NORMAL) {
+ p = &engine->default_priolist;
+ } else {
+ p = kmem_cache_alloc(engine->i915->priorities, GFP_ATOMIC);
+ /* Convert an allocation failure to a priority bump */
+ if (unlikely(!p)) {
+ prio = I915_PRIORITY_NORMAL; /* recurses just once */
+
+ /* To maintain ordering with all rendering, after an
+ * allocation failure we have to disable all scheduling.
+ * Requests will then be executed in fifo, and schedule
+ * will ensure that dependencies are emitted in fifo.
+ * There will be still some reordering with existing
+ * requests, so if userspace lied about their
+ * dependencies that reordering may be visible.
+ */
+ engine->no_priolist = true;
+ goto find_priolist;
+ }
+ }
+
+ p->priority = prio;
+ rb_link_node(&p->node, rb, parent);
+ rb_insert_color(&p->node, &engine->execlist_queue);
+
+ INIT_LIST_HEAD(&p->requests);
+ list_add_tail(&pt->link, &p->requests);
+
+ if (first)
+ engine->execlist_first = &p->node;
return first;
}
/* Will be called from irq-context when using foreign fences. */
spin_lock_irqsave(&engine->timeline->lock, flags);
- if (insert_request(&request->priotree, &engine->execlist_queue)) {
- engine->execlist_first = &request->priotree.node;
+ if (insert_request(engine,
+ &request->priotree,
+ request->priotree.priority)) {
if (execlists_elsp_ready(engine))
tasklet_hi_schedule(&engine->irq_tasklet);
}
+ GEM_BUG_ON(!engine->execlist_first);
+ GEM_BUG_ON(list_empty(&request->priotree.link));
+
spin_unlock_irqrestore(&engine->timeline->lock, flags);
}
list_safe_reset_next(dep, p, dfs_link);
}
+ /* If we didn't need to bump any existing priorities, and we haven't
+ * yet submitted this request (i.e. there is no potential race with
+ * execlists_submit_request()), we can set our own priority and skip
+ * acquiring the engine locks.
+ */
+ if (request->priotree.priority == INT_MIN) {
+ GEM_BUG_ON(!list_empty(&request->priotree.link));
+ request->priotree.priority = prio;
+ if (stack.dfs_link.next == stack.dfs_link.prev)
+ return;
+ __list_del_entry(&stack.dfs_link);
+ }
+
engine = request->engine;
spin_lock_irq(&engine->timeline->lock);
continue;
pt->priority = prio;
- if (!RB_EMPTY_NODE(&pt->node)) {
- rb_erase(&pt->node, &engine->execlist_queue);
- if (insert_request(pt, &engine->execlist_queue))
- engine->execlist_first = &pt->node;
+ if (!list_empty(&pt->link)) {
+ __list_del_entry(&pt->link);
+ insert_request(engine, pt, prio);
}
}
/* XXX Do we need to preempt to make room for us and our deps? */
}
-static int execlists_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx)
+static struct intel_ring *
+execlists_context_pin(struct intel_engine_cs *engine,
+ struct i915_gem_context *ctx)
{
struct intel_context *ce = &ctx->engine[engine->id];
unsigned int flags;
lockdep_assert_held(&ctx->i915->drm.struct_mutex);
- if (ce->pin_count++)
- return 0;
+ if (likely(ce->pin_count++))
+ goto out;
GEM_BUG_ON(!ce->pin_count); /* no overflow please! */
if (!ce->state) {
goto unpin_vma;
}
- ret = intel_ring_pin(ce->ring, ctx->ggtt_offset_bias);
+ ret = intel_ring_pin(ce->ring, ctx->i915, ctx->ggtt_offset_bias);
if (ret)
goto unpin_map;
ce->state->obj->mm.dirty = true;
i915_gem_context_get(ctx);
- return 0;
+out:
+ return ce->ring;
unpin_map:
i915_gem_object_unpin_map(ce->state->obj);
__i915_vma_unpin(ce->state);
err:
ce->pin_count = 0;
- return ret;
+ return ERR_PTR(ret);
}
static void execlists_context_unpin(struct intel_engine_cs *engine,
*/
request->reserved_space += EXECLISTS_REQUEST_SIZE;
- GEM_BUG_ON(!ce->ring);
- request->ring = ce->ring;
-
if (i915.enable_guc_submission) {
/*
* Check that the GuC has space for the request before
return ret;
}
-static u32 port_seqno(struct execlist_port *port)
-{
- return port->request ? port->request->global_seqno : 0;
-}
-
static int gen8_init_common_ring(struct intel_engine_cs *engine)
{
struct drm_i915_private *dev_priv = engine->i915;
+ struct execlist_port *port = engine->execlist_port;
+ unsigned int n;
+ bool submit;
int ret;
ret = intel_mocs_init_engine(engine);
/* After a GPU reset, we may have requests to replay */
clear_bit(ENGINE_IRQ_EXECLIST, &engine->irq_posted);
- if (!i915.enable_guc_submission && !execlists_elsp_idle(engine)) {
- DRM_DEBUG_DRIVER("Restarting %s from requests [0x%x, 0x%x]\n",
- engine->name,
- port_seqno(&engine->execlist_port[0]),
- port_seqno(&engine->execlist_port[1]));
- engine->execlist_port[0].count = 0;
- engine->execlist_port[1].count = 0;
- execlists_submit_ports(engine);
+
+ submit = false;
+ for (n = 0; n < ARRAY_SIZE(engine->execlist_port); n++) {
+ if (!port_isset(&port[n]))
+ break;
+
+ DRM_DEBUG_DRIVER("Restarting %s:%d from 0x%x\n",
+ engine->name, n,
+ port_request(&port[n])->global_seqno);
+
+ /* Discard the current inflight count */
+ port_set(&port[n], port_request(&port[n]));
+ submit = true;
}
+ if (submit && !i915.enable_guc_submission)
+ execlists_submit_ports(engine);
+
return 0;
}
intel_ring_update_space(request->ring);
/* Catch up with any missed context-switch interrupts */
- if (request->ctx != port[0].request->ctx) {
- i915_gem_request_put(port[0].request);
+ if (request->ctx != port_request(port)->ctx) {
+ i915_gem_request_put(port_request(port));
port[0] = port[1];
memset(&port[1], 0, sizeof(port[1]));
}
- GEM_BUG_ON(request->ctx != port[0].request->ctx);
+ GEM_BUG_ON(request->ctx != port_request(port)->ctx);
/* Reset WaIdleLiteRestore:bdw,skl as well */
request->tail =
return 0;
}
-/**
- * intel_lr_context_size() - return the size of the context for an engine
- * @engine: which engine to find the context size for
- *
- * Each engine may require a different amount of space for a context image,
- * so when allocating (or copying) an image, this function can be used to
- * find the right size for the specific engine.
- *
- * Return: size (in bytes) of an engine-specific context image
- *
- * Note: this size includes the HWSP, which is part of the context image
- * in LRC mode, but does not include the "shared data page" used with
- * GuC submission. The caller should account for this if using the GuC.
- */
-uint32_t intel_lr_context_size(struct intel_engine_cs *engine)
-{
- int ret = 0;
-
- WARN_ON(INTEL_GEN(engine->i915) < 8);
-
- switch (engine->id) {
- case RCS:
- if (INTEL_GEN(engine->i915) >= 9)
- ret = GEN9_LR_CONTEXT_RENDER_SIZE;
- else
- ret = GEN8_LR_CONTEXT_RENDER_SIZE;
- break;
- case VCS:
- case BCS:
- case VECS:
- case VCS2:
- ret = GEN8_LR_CONTEXT_OTHER_SIZE;
- break;
- }
-
- return ret;
-}
-
static int execlists_context_deferred_alloc(struct i915_gem_context *ctx,
struct intel_engine_cs *engine)
{
WARN_ON(ce->state);
- context_size = round_up(intel_lr_context_size(engine),
- I915_GTT_PAGE_SIZE);
+ context_size = round_up(engine->context_size, I915_GTT_PAGE_SIZE);
/* One extra page as the sharing data between driver and GuC */
context_size += PAGE_SIZE * LRC_PPHWSP_PN;
ce->ring = ring;
ce->state = vma;
- ce->initialised = engine->init_context == NULL;
+ ce->initialised |= engine->init_context == NULL;
return 0;
ce->state->obj->mm.dirty = true;
i915_gem_object_unpin_map(ce->state->obj);
- ce->ring->head = ce->ring->tail = 0;
- intel_ring_update_space(ce->ring);
+ intel_ring_reset(ce->ring, 0);
}
}
}
struct drm_i915_private;
struct i915_gem_context;
-uint32_t intel_lr_context_size(struct intel_engine_cs *engine);
-
void intel_lr_context_resume(struct drm_i915_private *dev_priv);
uint64_t intel_lr_context_descriptor(struct i915_gem_context *ctx,
struct intel_engine_cs *engine);
struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
struct intel_panel *panel = &connector->panel;
enum pipe pipe = intel_get_pipe_from_connector(connector);
- enum transcoder cpu_transcoder =
- intel_pipe_to_cpu_transcoder(dev_priv, pipe);
+ enum transcoder cpu_transcoder;
u32 cpu_ctl2, pch_ctl1, pch_ctl2;
+ if (!WARN_ON_ONCE(pipe == INVALID_PIPE))
+ cpu_transcoder = intel_pipe_to_cpu_transcoder(dev_priv, pipe);
+ else
+ cpu_transcoder = TRANSCODER_EDP;
+
cpu_ctl2 = I915_READ(BLC_PWM_CPU_CTL2);
if (cpu_ctl2 & BLM_PWM_ENABLE) {
DRM_DEBUG_KMS("cpu backlight already enabled\n");
enum pipe pipe = intel_get_pipe_from_connector(connector);
u32 ctl, ctl2, freq;
+ if (WARN_ON_ONCE(pipe == INVALID_PIPE))
+ pipe = PIPE_A;
+
ctl2 = I915_READ(BLC_PWM_CTL2);
if (ctl2 & BLM_PWM_ENABLE) {
DRM_DEBUG_KMS("backlight already enabled\n");
enum pipe pipe = intel_get_pipe_from_connector(connector);
u32 pwm_ctl, val;
+ if (WARN_ON_ONCE(pipe == INVALID_PIPE))
+ pipe = PIPE_A;
+
/* Controller 1 uses the utility pin. */
if (panel->backlight.controller == 1) {
val = I915_READ(UTIL_PIN_CTL);
if (!panel->backlight.present)
return;
- DRM_DEBUG_KMS("pipe %c\n", pipe_name(pipe));
+ if (!WARN_ON_ONCE(pipe == INVALID_PIPE))
+ DRM_DEBUG_KMS("pipe %c\n", pipe_name(pipe));
mutex_lock(&dev_priv->backlight_lock);
struct intel_crtc *crtc = intel_get_crtc_for_pipe(dev_priv, PIPE_A);
struct intel_crtc_state *pipe_config;
struct drm_atomic_state *state;
+ struct drm_modeset_acquire_ctx ctx;
int ret = 0;
- drm_modeset_lock_all(dev);
+ drm_modeset_acquire_init(&ctx, 0);
+
state = drm_atomic_state_alloc(dev);
if (!state) {
ret = -ENOMEM;
goto unlock;
}
- state->acquire_ctx = crtc->base.dev->mode_config.acquire_ctx;
+ state->acquire_ctx = &ctx;
+
+retry:
pipe_config = intel_atomic_get_crtc_state(state, crtc);
if (IS_ERR(pipe_config)) {
ret = PTR_ERR(pipe_config);
ret = drm_atomic_commit(state);
put_state:
+ if (ret == -EDEADLK) {
+ drm_atomic_state_clear(state);
+ drm_modeset_backoff(&ctx);
+ goto retry;
+ }
+
drm_atomic_state_put(state);
unlock:
WARN(ret, "Toggling workaround to %i returns %i\n", enable, ret);
- drm_modeset_unlock_all(dev);
+ drm_modeset_drop_locks(&ctx);
+ drm_modeset_acquire_fini(&ctx);
}
static int ivb_pipe_crc_ctl_reg(struct drm_i915_private *dev_priv,
return -E2BIG;
}
- tmpbuf = kmalloc(len + 1, GFP_KERNEL);
- if (!tmpbuf)
- return -ENOMEM;
-
- if (copy_from_user(tmpbuf, ubuf, len)) {
- ret = -EFAULT;
- goto out;
- }
- tmpbuf[len] = '\0';
+ tmpbuf = memdup_user_nul(ubuf, len);
+ if (IS_ERR(tmpbuf))
+ return PTR_ERR(tmpbuf);
ret = display_crc_ctl_parse(dev_priv, tmpbuf, len);
-out:
kfree(tmpbuf);
if (ret < 0)
return ret;
return was_enabled;
}
+/**
+ * intel_set_memory_cxsr - Configure CxSR state
+ * @dev_priv: i915 device
+ * @enable: Allow vs. disallow CxSR
+ *
+ * Allow or disallow the system to enter a special CxSR
+ * (C-state self refresh) state. What typically happens in CxSR mode
+ * is that several display FIFOs may get combined into a single larger
+ * FIFO for a particular plane (so called max FIFO mode) to allow the
+ * system to defer memory fetches longer, and the memory will enter
+ * self refresh.
+ *
+ * Note that enabling CxSR does not guarantee that the system enter
+ * this special mode, nor does it guarantee that the system stays
+ * in that mode once entered. So this just allows/disallows the system
+ * to autonomously utilize the CxSR mode. Other factors such as core
+ * C-states will affect when/if the system actually enters/exits the
+ * CxSR mode.
+ *
+ * Note that on VLV/CHV this actually only controls the max FIFO mode,
+ * and the system is free to enter/exit memory self refresh at any time
+ * even when the use of CxSR has been disallowed.
+ *
+ * While the system is actually in the CxSR/max FIFO mode, some plane
+ * control registers will not get latched on vblank. Thus in order to
+ * guarantee the system will respond to changes in the plane registers
+ * we must always disallow CxSR prior to making changes to those registers.
+ * Unfortunately the system will re-evaluate the CxSR conditions at
+ * frame start which happens after vblank start (which is when the plane
+ * registers would get latched), so we can't proceed with the plane update
+ * during the same frame where we disallowed CxSR.
+ *
+ * Certain platforms also have a deeper HPLL SR mode. Fortunately the
+ * HPLL SR mode depends on CxSR itself, so we don't have to hand hold
+ * the hardware w.r.t. HPLL SR when writing to plane registers.
+ * Disallowing just CxSR is sufficient.
+ */
bool intel_set_memory_cxsr(struct drm_i915_private *dev_priv, bool enable)
{
bool ret;
mutex_lock(&dev_priv->wm.wm_mutex);
ret = _intel_set_memory_cxsr(dev_priv, enable);
- dev_priv->wm.vlv.cxsr = enable;
+ if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
+ dev_priv->wm.vlv.cxsr = enable;
+ else if (IS_G4X(dev_priv))
+ dev_priv->wm.g4x.cxsr = enable;
mutex_unlock(&dev_priv->wm.wm_mutex);
return ret;
fifo_state->plane[PLANE_SPRITE0] = sprite1_start - sprite0_start;
fifo_state->plane[PLANE_SPRITE1] = 511 - sprite1_start;
fifo_state->plane[PLANE_CURSOR] = 63;
-
- DRM_DEBUG_KMS("Pipe %c FIFO size: %d/%d/%d/%d\n",
- pipe_name(pipe),
- fifo_state->plane[PLANE_PRIMARY],
- fifo_state->plane[PLANE_SPRITE0],
- fifo_state->plane[PLANE_SPRITE1],
- fifo_state->plane[PLANE_CURSOR]);
}
static int i9xx_get_fifo_size(struct drm_i915_private *dev_priv, int plane)
.guard_size = PINEVIEW_CURSOR_GUARD_WM,
.cacheline_size = PINEVIEW_FIFO_LINE_SIZE,
};
-static const struct intel_watermark_params g4x_wm_info = {
- .fifo_size = G4X_FIFO_SIZE,
- .max_wm = G4X_MAX_WM,
- .default_wm = G4X_MAX_WM,
- .guard_size = 2,
- .cacheline_size = G4X_FIFO_LINE_SIZE,
-};
-static const struct intel_watermark_params g4x_cursor_wm_info = {
- .fifo_size = I965_CURSOR_FIFO,
- .max_wm = I965_CURSOR_MAX_WM,
- .default_wm = I965_CURSOR_DFT_WM,
- .guard_size = 2,
- .cacheline_size = G4X_FIFO_LINE_SIZE,
-};
static const struct intel_watermark_params i965_cursor_wm_info = {
.fifo_size = I965_CURSOR_FIFO,
.max_wm = I965_CURSOR_MAX_WM,
.cacheline_size = I830_FIFO_LINE_SIZE,
};
+/**
+ * intel_wm_method1 - Method 1 / "small buffer" watermark formula
+ * @pixel_rate: Pipe pixel rate in kHz
+ * @cpp: Plane bytes per pixel
+ * @latency: Memory wakeup latency in 0.1us units
+ *
+ * Compute the watermark using the method 1 or "small buffer"
+ * formula. The caller may additonally add extra cachelines
+ * to account for TLB misses and clock crossings.
+ *
+ * This method is concerned with the short term drain rate
+ * of the FIFO, ie. it does not account for blanking periods
+ * which would effectively reduce the average drain rate across
+ * a longer period. The name "small" refers to the fact the
+ * FIFO is relatively small compared to the amount of data
+ * fetched.
+ *
+ * The FIFO level vs. time graph might look something like:
+ *
+ * |\ |\
+ * | \ | \
+ * __---__---__ (- plane active, _ blanking)
+ * -> time
+ *
+ * or perhaps like this:
+ *
+ * |\|\ |\|\
+ * __----__----__ (- plane active, _ blanking)
+ * -> time
+ *
+ * Returns:
+ * The watermark in bytes
+ */
+static unsigned int intel_wm_method1(unsigned int pixel_rate,
+ unsigned int cpp,
+ unsigned int latency)
+{
+ uint64_t ret;
+
+ ret = (uint64_t) pixel_rate * cpp * latency;
+ ret = DIV_ROUND_UP_ULL(ret, 10000);
+
+ return ret;
+}
+
+/**
+ * intel_wm_method2 - Method 2 / "large buffer" watermark formula
+ * @pixel_rate: Pipe pixel rate in kHz
+ * @htotal: Pipe horizontal total
+ * @width: Plane width in pixels
+ * @cpp: Plane bytes per pixel
+ * @latency: Memory wakeup latency in 0.1us units
+ *
+ * Compute the watermark using the method 2 or "large buffer"
+ * formula. The caller may additonally add extra cachelines
+ * to account for TLB misses and clock crossings.
+ *
+ * This method is concerned with the long term drain rate
+ * of the FIFO, ie. it does account for blanking periods
+ * which effectively reduce the average drain rate across
+ * a longer period. The name "large" refers to the fact the
+ * FIFO is relatively large compared to the amount of data
+ * fetched.
+ *
+ * The FIFO level vs. time graph might look something like:
+ *
+ * |\___ |\___
+ * | \___ | \___
+ * | \ | \
+ * __ --__--__--__--__--__--__ (- plane active, _ blanking)
+ * -> time
+ *
+ * Returns:
+ * The watermark in bytes
+ */
+static unsigned int intel_wm_method2(unsigned int pixel_rate,
+ unsigned int htotal,
+ unsigned int width,
+ unsigned int cpp,
+ unsigned int latency)
+{
+ unsigned int ret;
+
+ /*
+ * FIXME remove once all users are computing
+ * watermarks in the correct place.
+ */
+ if (WARN_ON_ONCE(htotal == 0))
+ htotal = 1;
+
+ ret = (latency * pixel_rate) / (htotal * 10000);
+ ret = (ret + 1) * width * cpp;
+
+ return ret;
+}
+
/**
* intel_calculate_wm - calculate watermark level
- * @clock_in_khz: pixel clock
+ * @pixel_rate: pixel clock
* @wm: chip FIFO params
* @cpp: bytes per pixel
* @latency_ns: memory latency for the platform
* past the watermark point. If the FIFO drains completely, a FIFO underrun
* will occur, and a display engine hang could result.
*/
-static unsigned long intel_calculate_wm(unsigned long clock_in_khz,
- const struct intel_watermark_params *wm,
- int fifo_size, int cpp,
- unsigned long latency_ns)
+static unsigned int intel_calculate_wm(int pixel_rate,
+ const struct intel_watermark_params *wm,
+ int fifo_size, int cpp,
+ unsigned int latency_ns)
{
- long entries_required, wm_size;
+ int entries, wm_size;
/*
* Note: we need to make sure we don't overflow for various clock &
* clocks go from a few thousand to several hundred thousand.
* latency is usually a few thousand
*/
- entries_required = ((clock_in_khz / 1000) * cpp * latency_ns) /
- 1000;
- entries_required = DIV_ROUND_UP(entries_required, wm->cacheline_size);
+ entries = intel_wm_method1(pixel_rate, cpp,
+ latency_ns / 100);
+ entries = DIV_ROUND_UP(entries, wm->cacheline_size) +
+ wm->guard_size;
+ DRM_DEBUG_KMS("FIFO entries required for mode: %d\n", entries);
- DRM_DEBUG_KMS("FIFO entries required for mode: %ld\n", entries_required);
-
- wm_size = fifo_size - (entries_required + wm->guard_size);
-
- DRM_DEBUG_KMS("FIFO watermark level: %ld\n", wm_size);
+ wm_size = fifo_size - entries;
+ DRM_DEBUG_KMS("FIFO watermark level: %d\n", wm_size);
/* Don't promote wm_size to unsigned... */
- if (wm_size > (long)wm->max_wm)
+ if (wm_size > wm->max_wm)
wm_size = wm->max_wm;
if (wm_size <= 0)
wm_size = wm->default_wm;
return wm_size;
}
+static bool is_disabling(int old, int new, int threshold)
+{
+ return old >= threshold && new < threshold;
+}
+
+static bool is_enabling(int old, int new, int threshold)
+{
+ return old < threshold && new >= threshold;
+}
+
+static int intel_wm_num_levels(struct drm_i915_private *dev_priv)
+{
+ return dev_priv->wm.max_level + 1;
+}
+
static bool intel_wm_plane_visible(const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
struct intel_crtc *crtc;
const struct cxsr_latency *latency;
u32 reg;
- unsigned long wm;
+ unsigned int wm;
latency = intel_get_cxsr_latency(IS_PINEVIEW_G(dev_priv),
dev_priv->is_ddr3,
/* cursor SR */
wm = intel_calculate_wm(clock, &pineview_cursor_wm,
pineview_display_wm.fifo_size,
- cpp, latency->cursor_sr);
+ 4, latency->cursor_sr);
reg = I915_READ(DSPFW3);
reg &= ~DSPFW_CURSOR_SR_MASK;
reg |= FW_WM(wm, CURSOR_SR);
/* cursor HPLL off SR */
wm = intel_calculate_wm(clock, &pineview_cursor_hplloff_wm,
pineview_display_hplloff_wm.fifo_size,
- cpp, latency->cursor_hpll_disable);
+ 4, latency->cursor_hpll_disable);
reg = I915_READ(DSPFW3);
reg &= ~DSPFW_HPLL_CURSOR_MASK;
reg |= FW_WM(wm, HPLL_CURSOR);
}
}
-static bool g4x_compute_wm0(struct drm_i915_private *dev_priv,
- int plane,
- const struct intel_watermark_params *display,
- int display_latency_ns,
- const struct intel_watermark_params *cursor,
- int cursor_latency_ns,
- int *plane_wm,
- int *cursor_wm)
-{
- struct intel_crtc *crtc;
- const struct drm_display_mode *adjusted_mode;
- const struct drm_framebuffer *fb;
- int htotal, hdisplay, clock, cpp;
- int line_time_us, line_count;
- int entries, tlb_miss;
-
- crtc = intel_get_crtc_for_plane(dev_priv, plane);
- if (!intel_crtc_active(crtc)) {
- *cursor_wm = cursor->guard_size;
- *plane_wm = display->guard_size;
- return false;
- }
-
- adjusted_mode = &crtc->config->base.adjusted_mode;
- fb = crtc->base.primary->state->fb;
- clock = adjusted_mode->crtc_clock;
- htotal = adjusted_mode->crtc_htotal;
- hdisplay = crtc->config->pipe_src_w;
- cpp = fb->format->cpp[0];
-
- /* Use the small buffer method to calculate plane watermark */
- entries = ((clock * cpp / 1000) * display_latency_ns) / 1000;
- tlb_miss = display->fifo_size*display->cacheline_size - hdisplay * 8;
- if (tlb_miss > 0)
- entries += tlb_miss;
- entries = DIV_ROUND_UP(entries, display->cacheline_size);
- *plane_wm = entries + display->guard_size;
- if (*plane_wm > (int)display->max_wm)
- *plane_wm = display->max_wm;
-
- /* Use the large buffer method to calculate cursor watermark */
- line_time_us = max(htotal * 1000 / clock, 1);
- line_count = (cursor_latency_ns / line_time_us + 1000) / 1000;
- entries = line_count * crtc->base.cursor->state->crtc_w * cpp;
- tlb_miss = cursor->fifo_size*cursor->cacheline_size - hdisplay * 8;
- if (tlb_miss > 0)
- entries += tlb_miss;
- entries = DIV_ROUND_UP(entries, cursor->cacheline_size);
- *cursor_wm = entries + cursor->guard_size;
- if (*cursor_wm > (int)cursor->max_wm)
- *cursor_wm = (int)cursor->max_wm;
-
- return true;
-}
-
/*
- * Check the wm result.
- *
- * If any calculated watermark values is larger than the maximum value that
- * can be programmed into the associated watermark register, that watermark
- * must be disabled.
+ * Documentation says:
+ * "If the line size is small, the TLB fetches can get in the way of the
+ * data fetches, causing some lag in the pixel data return which is not
+ * accounted for in the above formulas. The following adjustment only
+ * needs to be applied if eight whole lines fit in the buffer at once.
+ * The WM is adjusted upwards by the difference between the FIFO size
+ * and the size of 8 whole lines. This adjustment is always performed
+ * in the actual pixel depth regardless of whether FBC is enabled or not."
*/
-static bool g4x_check_srwm(struct drm_i915_private *dev_priv,
- int display_wm, int cursor_wm,
- const struct intel_watermark_params *display,
- const struct intel_watermark_params *cursor)
+static int g4x_tlb_miss_wa(int fifo_size, int width, int cpp)
{
- DRM_DEBUG_KMS("SR watermark: display plane %d, cursor %d\n",
- display_wm, cursor_wm);
-
- if (display_wm > display->max_wm) {
- DRM_DEBUG_KMS("display watermark is too large(%d/%u), disabling\n",
- display_wm, display->max_wm);
- return false;
- }
-
- if (cursor_wm > cursor->max_wm) {
- DRM_DEBUG_KMS("cursor watermark is too large(%d/%u), disabling\n",
- cursor_wm, cursor->max_wm);
- return false;
- }
-
- if (!(display_wm || cursor_wm)) {
- DRM_DEBUG_KMS("SR latency is 0, disabling\n");
- return false;
- }
+ int tlb_miss = fifo_size * 64 - width * cpp * 8;
- return true;
+ return max(0, tlb_miss);
}
-static bool g4x_compute_srwm(struct drm_i915_private *dev_priv,
- int plane,
- int latency_ns,
- const struct intel_watermark_params *display,
- const struct intel_watermark_params *cursor,
- int *display_wm, int *cursor_wm)
+static void g4x_write_wm_values(struct drm_i915_private *dev_priv,
+ const struct g4x_wm_values *wm)
{
- struct intel_crtc *crtc;
- const struct drm_display_mode *adjusted_mode;
- const struct drm_framebuffer *fb;
- int hdisplay, htotal, cpp, clock;
- unsigned long line_time_us;
- int line_count, line_size;
- int small, large;
- int entries;
-
- if (!latency_ns) {
- *display_wm = *cursor_wm = 0;
- return false;
- }
-
- crtc = intel_get_crtc_for_plane(dev_priv, plane);
- adjusted_mode = &crtc->config->base.adjusted_mode;
- fb = crtc->base.primary->state->fb;
- clock = adjusted_mode->crtc_clock;
- htotal = adjusted_mode->crtc_htotal;
- hdisplay = crtc->config->pipe_src_w;
- cpp = fb->format->cpp[0];
-
- line_time_us = max(htotal * 1000 / clock, 1);
- line_count = (latency_ns / line_time_us + 1000) / 1000;
- line_size = hdisplay * cpp;
-
- /* Use the minimum of the small and large buffer method for primary */
- small = ((clock * cpp / 1000) * latency_ns) / 1000;
- large = line_count * line_size;
+ enum pipe pipe;
- entries = DIV_ROUND_UP(min(small, large), display->cacheline_size);
- *display_wm = entries + display->guard_size;
+ for_each_pipe(dev_priv, pipe)
+ trace_g4x_wm(intel_get_crtc_for_pipe(dev_priv, pipe), wm);
- /* calculate the self-refresh watermark for display cursor */
- entries = line_count * cpp * crtc->base.cursor->state->crtc_w;
- entries = DIV_ROUND_UP(entries, cursor->cacheline_size);
- *cursor_wm = entries + cursor->guard_size;
+ I915_WRITE(DSPFW1,
+ FW_WM(wm->sr.plane, SR) |
+ FW_WM(wm->pipe[PIPE_B].plane[PLANE_CURSOR], CURSORB) |
+ FW_WM(wm->pipe[PIPE_B].plane[PLANE_PRIMARY], PLANEB) |
+ FW_WM(wm->pipe[PIPE_A].plane[PLANE_PRIMARY], PLANEA));
+ I915_WRITE(DSPFW2,
+ (wm->fbc_en ? DSPFW_FBC_SR_EN : 0) |
+ FW_WM(wm->sr.fbc, FBC_SR) |
+ FW_WM(wm->hpll.fbc, FBC_HPLL_SR) |
+ FW_WM(wm->pipe[PIPE_B].plane[PLANE_SPRITE0], SPRITEB) |
+ FW_WM(wm->pipe[PIPE_A].plane[PLANE_CURSOR], CURSORA) |
+ FW_WM(wm->pipe[PIPE_A].plane[PLANE_SPRITE0], SPRITEA));
+ I915_WRITE(DSPFW3,
+ (wm->hpll_en ? DSPFW_HPLL_SR_EN : 0) |
+ FW_WM(wm->sr.cursor, CURSOR_SR) |
+ FW_WM(wm->hpll.cursor, HPLL_CURSOR) |
+ FW_WM(wm->hpll.plane, HPLL_SR));
- return g4x_check_srwm(dev_priv,
- *display_wm, *cursor_wm,
- display, cursor);
+ POSTING_READ(DSPFW1);
}
#define FW_WM_VLV(value, plane) \
FW_WM(wm->pipe[PIPE_A].plane[PLANE_PRIMARY] >> 8, PLANEA_HI));
}
- POSTING_READ(DSPFW1);
+ POSTING_READ(DSPFW1);
+}
+
+#undef FW_WM_VLV
+
+static void g4x_setup_wm_latency(struct drm_i915_private *dev_priv)
+{
+ /* all latencies in usec */
+ dev_priv->wm.pri_latency[G4X_WM_LEVEL_NORMAL] = 5;
+ dev_priv->wm.pri_latency[G4X_WM_LEVEL_SR] = 12;
+ dev_priv->wm.pri_latency[G4X_WM_LEVEL_HPLL] = 35;
+
+ dev_priv->wm.max_level = G4X_WM_LEVEL_HPLL;
+}
+
+static int g4x_plane_fifo_size(enum plane_id plane_id, int level)
+{
+ /*
+ * DSPCNTR[13] supposedly controls whether the
+ * primary plane can use the FIFO space otherwise
+ * reserved for the sprite plane. It's not 100% clear
+ * what the actual FIFO size is, but it looks like we
+ * can happily set both primary and sprite watermarks
+ * up to 127 cachelines. So that would seem to mean
+ * that either DSPCNTR[13] doesn't do anything, or that
+ * the total FIFO is >= 256 cachelines in size. Either
+ * way, we don't seem to have to worry about this
+ * repartitioning as the maximum watermark value the
+ * register can hold for each plane is lower than the
+ * minimum FIFO size.
+ */
+ switch (plane_id) {
+ case PLANE_CURSOR:
+ return 63;
+ case PLANE_PRIMARY:
+ return level == G4X_WM_LEVEL_NORMAL ? 127 : 511;
+ case PLANE_SPRITE0:
+ return level == G4X_WM_LEVEL_NORMAL ? 127 : 0;
+ default:
+ MISSING_CASE(plane_id);
+ return 0;
+ }
+}
+
+static int g4x_fbc_fifo_size(int level)
+{
+ switch (level) {
+ case G4X_WM_LEVEL_SR:
+ return 7;
+ case G4X_WM_LEVEL_HPLL:
+ return 15;
+ default:
+ MISSING_CASE(level);
+ return 0;
+ }
+}
+
+static uint16_t g4x_compute_wm(const struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state,
+ int level)
+{
+ struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ const struct drm_display_mode *adjusted_mode =
+ &crtc_state->base.adjusted_mode;
+ int clock, htotal, cpp, width, wm;
+ int latency = dev_priv->wm.pri_latency[level] * 10;
+
+ if (latency == 0)
+ return USHRT_MAX;
+
+ if (!intel_wm_plane_visible(crtc_state, plane_state))
+ return 0;
+
+ /*
+ * Not 100% sure which way ELK should go here as the
+ * spec only says CL/CTG should assume 32bpp and BW
+ * doesn't need to. But as these things followed the
+ * mobile vs. desktop lines on gen3 as well, let's
+ * assume ELK doesn't need this.
+ *
+ * The spec also fails to list such a restriction for
+ * the HPLL watermark, which seems a little strange.
+ * Let's use 32bpp for the HPLL watermark as well.
+ */
+ if (IS_GM45(dev_priv) && plane->id == PLANE_PRIMARY &&
+ level != G4X_WM_LEVEL_NORMAL)
+ cpp = 4;
+ else
+ cpp = plane_state->base.fb->format->cpp[0];
+
+ clock = adjusted_mode->crtc_clock;
+ htotal = adjusted_mode->crtc_htotal;
+
+ if (plane->id == PLANE_CURSOR)
+ width = plane_state->base.crtc_w;
+ else
+ width = drm_rect_width(&plane_state->base.dst);
+
+ if (plane->id == PLANE_CURSOR) {
+ wm = intel_wm_method2(clock, htotal, width, cpp, latency);
+ } else if (plane->id == PLANE_PRIMARY &&
+ level == G4X_WM_LEVEL_NORMAL) {
+ wm = intel_wm_method1(clock, cpp, latency);
+ } else {
+ int small, large;
+
+ small = intel_wm_method1(clock, cpp, latency);
+ large = intel_wm_method2(clock, htotal, width, cpp, latency);
+
+ wm = min(small, large);
+ }
+
+ wm += g4x_tlb_miss_wa(g4x_plane_fifo_size(plane->id, level),
+ width, cpp);
+
+ wm = DIV_ROUND_UP(wm, 64) + 2;
+
+ return min_t(int, wm, USHRT_MAX);
+}
+
+static bool g4x_raw_plane_wm_set(struct intel_crtc_state *crtc_state,
+ int level, enum plane_id plane_id, u16 value)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ bool dirty = false;
+
+ for (; level < intel_wm_num_levels(dev_priv); level++) {
+ struct g4x_pipe_wm *raw = &crtc_state->wm.g4x.raw[level];
+
+ dirty |= raw->plane[plane_id] != value;
+ raw->plane[plane_id] = value;
+ }
+
+ return dirty;
+}
+
+static bool g4x_raw_fbc_wm_set(struct intel_crtc_state *crtc_state,
+ int level, u16 value)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ bool dirty = false;
+
+ /* NORMAL level doesn't have an FBC watermark */
+ level = max(level, G4X_WM_LEVEL_SR);
+
+ for (; level < intel_wm_num_levels(dev_priv); level++) {
+ struct g4x_pipe_wm *raw = &crtc_state->wm.g4x.raw[level];
+
+ dirty |= raw->fbc != value;
+ raw->fbc = value;
+ }
+
+ return dirty;
+}
+
+static uint32_t ilk_compute_fbc_wm(const struct intel_crtc_state *cstate,
+ const struct intel_plane_state *pstate,
+ uint32_t pri_val);
+
+static bool g4x_raw_plane_wm_compute(struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state)
+{
+ struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
+ int num_levels = intel_wm_num_levels(to_i915(plane->base.dev));
+ enum plane_id plane_id = plane->id;
+ bool dirty = false;
+ int level;
+
+ if (!intel_wm_plane_visible(crtc_state, plane_state)) {
+ dirty |= g4x_raw_plane_wm_set(crtc_state, 0, plane_id, 0);
+ if (plane_id == PLANE_PRIMARY)
+ dirty |= g4x_raw_fbc_wm_set(crtc_state, 0, 0);
+ goto out;
+ }
+
+ for (level = 0; level < num_levels; level++) {
+ struct g4x_pipe_wm *raw = &crtc_state->wm.g4x.raw[level];
+ int wm, max_wm;
+
+ wm = g4x_compute_wm(crtc_state, plane_state, level);
+ max_wm = g4x_plane_fifo_size(plane_id, level);
+
+ if (wm > max_wm)
+ break;
+
+ dirty |= raw->plane[plane_id] != wm;
+ raw->plane[plane_id] = wm;
+
+ if (plane_id != PLANE_PRIMARY ||
+ level == G4X_WM_LEVEL_NORMAL)
+ continue;
+
+ wm = ilk_compute_fbc_wm(crtc_state, plane_state,
+ raw->plane[plane_id]);
+ max_wm = g4x_fbc_fifo_size(level);
+
+ /*
+ * FBC wm is not mandatory as we
+ * can always just disable its use.
+ */
+ if (wm > max_wm)
+ wm = USHRT_MAX;
+
+ dirty |= raw->fbc != wm;
+ raw->fbc = wm;
+ }
+
+ /* mark watermarks as invalid */
+ dirty |= g4x_raw_plane_wm_set(crtc_state, level, plane_id, USHRT_MAX);
+
+ if (plane_id == PLANE_PRIMARY)
+ dirty |= g4x_raw_fbc_wm_set(crtc_state, level, USHRT_MAX);
+
+ out:
+ if (dirty) {
+ DRM_DEBUG_KMS("%s watermarks: normal=%d, SR=%d, HPLL=%d\n",
+ plane->base.name,
+ crtc_state->wm.g4x.raw[G4X_WM_LEVEL_NORMAL].plane[plane_id],
+ crtc_state->wm.g4x.raw[G4X_WM_LEVEL_SR].plane[plane_id],
+ crtc_state->wm.g4x.raw[G4X_WM_LEVEL_HPLL].plane[plane_id]);
+
+ if (plane_id == PLANE_PRIMARY)
+ DRM_DEBUG_KMS("FBC watermarks: SR=%d, HPLL=%d\n",
+ crtc_state->wm.g4x.raw[G4X_WM_LEVEL_SR].fbc,
+ crtc_state->wm.g4x.raw[G4X_WM_LEVEL_HPLL].fbc);
+ }
+
+ return dirty;
+}
+
+static bool g4x_raw_plane_wm_is_valid(const struct intel_crtc_state *crtc_state,
+ enum plane_id plane_id, int level)
+{
+ const struct g4x_pipe_wm *raw = &crtc_state->wm.g4x.raw[level];
+
+ return raw->plane[plane_id] <= g4x_plane_fifo_size(plane_id, level);
+}
+
+static bool g4x_raw_crtc_wm_is_valid(const struct intel_crtc_state *crtc_state,
+ int level)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+
+ if (level > dev_priv->wm.max_level)
+ return false;
+
+ return g4x_raw_plane_wm_is_valid(crtc_state, PLANE_PRIMARY, level) &&
+ g4x_raw_plane_wm_is_valid(crtc_state, PLANE_SPRITE0, level) &&
+ g4x_raw_plane_wm_is_valid(crtc_state, PLANE_CURSOR, level);
+}
+
+/* mark all levels starting from 'level' as invalid */
+static void g4x_invalidate_wms(struct intel_crtc *crtc,
+ struct g4x_wm_state *wm_state, int level)
+{
+ if (level <= G4X_WM_LEVEL_NORMAL) {
+ enum plane_id plane_id;
+
+ for_each_plane_id_on_crtc(crtc, plane_id)
+ wm_state->wm.plane[plane_id] = USHRT_MAX;
+ }
+
+ if (level <= G4X_WM_LEVEL_SR) {
+ wm_state->cxsr = false;
+ wm_state->sr.cursor = USHRT_MAX;
+ wm_state->sr.plane = USHRT_MAX;
+ wm_state->sr.fbc = USHRT_MAX;
+ }
+
+ if (level <= G4X_WM_LEVEL_HPLL) {
+ wm_state->hpll_en = false;
+ wm_state->hpll.cursor = USHRT_MAX;
+ wm_state->hpll.plane = USHRT_MAX;
+ wm_state->hpll.fbc = USHRT_MAX;
+ }
+}
+
+static int g4x_compute_pipe_wm(struct intel_crtc_state *crtc_state)
+{
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+ struct intel_atomic_state *state =
+ to_intel_atomic_state(crtc_state->base.state);
+ struct g4x_wm_state *wm_state = &crtc_state->wm.g4x.optimal;
+ int num_active_planes = hweight32(crtc_state->active_planes &
+ ~BIT(PLANE_CURSOR));
+ const struct g4x_pipe_wm *raw;
+ struct intel_plane_state *plane_state;
+ struct intel_plane *plane;
+ enum plane_id plane_id;
+ int i, level;
+ unsigned int dirty = 0;
+
+ for_each_intel_plane_in_state(state, plane, plane_state, i) {
+ const struct intel_plane_state *old_plane_state =
+ to_intel_plane_state(plane->base.state);
+
+ if (plane_state->base.crtc != &crtc->base &&
+ old_plane_state->base.crtc != &crtc->base)
+ continue;
+
+ if (g4x_raw_plane_wm_compute(crtc_state, plane_state))
+ dirty |= BIT(plane->id);
+ }
+
+ if (!dirty)
+ return 0;
+
+ level = G4X_WM_LEVEL_NORMAL;
+ if (!g4x_raw_crtc_wm_is_valid(crtc_state, level))
+ goto out;
+
+ raw = &crtc_state->wm.g4x.raw[level];
+ for_each_plane_id_on_crtc(crtc, plane_id)
+ wm_state->wm.plane[plane_id] = raw->plane[plane_id];
+
+ level = G4X_WM_LEVEL_SR;
+
+ if (!g4x_raw_crtc_wm_is_valid(crtc_state, level))
+ goto out;
+
+ raw = &crtc_state->wm.g4x.raw[level];
+ wm_state->sr.plane = raw->plane[PLANE_PRIMARY];
+ wm_state->sr.cursor = raw->plane[PLANE_CURSOR];
+ wm_state->sr.fbc = raw->fbc;
+
+ wm_state->cxsr = num_active_planes == BIT(PLANE_PRIMARY);
+
+ level = G4X_WM_LEVEL_HPLL;
+
+ if (!g4x_raw_crtc_wm_is_valid(crtc_state, level))
+ goto out;
+
+ raw = &crtc_state->wm.g4x.raw[level];
+ wm_state->hpll.plane = raw->plane[PLANE_PRIMARY];
+ wm_state->hpll.cursor = raw->plane[PLANE_CURSOR];
+ wm_state->hpll.fbc = raw->fbc;
+
+ wm_state->hpll_en = wm_state->cxsr;
+
+ level++;
+
+ out:
+ if (level == G4X_WM_LEVEL_NORMAL)
+ return -EINVAL;
+
+ /* invalidate the higher levels */
+ g4x_invalidate_wms(crtc, wm_state, level);
+
+ /*
+ * Determine if the FBC watermark(s) can be used. IF
+ * this isn't the case we prefer to disable the FBC
+ ( watermark(s) rather than disable the SR/HPLL
+ * level(s) entirely.
+ */
+ wm_state->fbc_en = level > G4X_WM_LEVEL_NORMAL;
+
+ if (level >= G4X_WM_LEVEL_SR &&
+ wm_state->sr.fbc > g4x_fbc_fifo_size(G4X_WM_LEVEL_SR))
+ wm_state->fbc_en = false;
+ else if (level >= G4X_WM_LEVEL_HPLL &&
+ wm_state->hpll.fbc > g4x_fbc_fifo_size(G4X_WM_LEVEL_HPLL))
+ wm_state->fbc_en = false;
+
+ return 0;
+}
+
+static int g4x_compute_intermediate_wm(struct drm_device *dev,
+ struct intel_crtc *crtc,
+ struct intel_crtc_state *crtc_state)
+{
+ struct g4x_wm_state *intermediate = &crtc_state->wm.g4x.intermediate;
+ const struct g4x_wm_state *optimal = &crtc_state->wm.g4x.optimal;
+ const struct g4x_wm_state *active = &crtc->wm.active.g4x;
+ enum plane_id plane_id;
+
+ intermediate->cxsr = optimal->cxsr && active->cxsr &&
+ !crtc_state->disable_cxsr;
+ intermediate->hpll_en = optimal->hpll_en && active->hpll_en &&
+ !crtc_state->disable_cxsr;
+ intermediate->fbc_en = optimal->fbc_en && active->fbc_en;
+
+ for_each_plane_id_on_crtc(crtc, plane_id) {
+ intermediate->wm.plane[plane_id] =
+ max(optimal->wm.plane[plane_id],
+ active->wm.plane[plane_id]);
+
+ WARN_ON(intermediate->wm.plane[plane_id] >
+ g4x_plane_fifo_size(plane_id, G4X_WM_LEVEL_NORMAL));
+ }
+
+ intermediate->sr.plane = max(optimal->sr.plane,
+ active->sr.plane);
+ intermediate->sr.cursor = max(optimal->sr.cursor,
+ active->sr.cursor);
+ intermediate->sr.fbc = max(optimal->sr.fbc,
+ active->sr.fbc);
+
+ intermediate->hpll.plane = max(optimal->hpll.plane,
+ active->hpll.plane);
+ intermediate->hpll.cursor = max(optimal->hpll.cursor,
+ active->hpll.cursor);
+ intermediate->hpll.fbc = max(optimal->hpll.fbc,
+ active->hpll.fbc);
+
+ WARN_ON((intermediate->sr.plane >
+ g4x_plane_fifo_size(PLANE_PRIMARY, G4X_WM_LEVEL_SR) ||
+ intermediate->sr.cursor >
+ g4x_plane_fifo_size(PLANE_CURSOR, G4X_WM_LEVEL_SR)) &&
+ intermediate->cxsr);
+ WARN_ON((intermediate->sr.plane >
+ g4x_plane_fifo_size(PLANE_PRIMARY, G4X_WM_LEVEL_HPLL) ||
+ intermediate->sr.cursor >
+ g4x_plane_fifo_size(PLANE_CURSOR, G4X_WM_LEVEL_HPLL)) &&
+ intermediate->hpll_en);
+
+ WARN_ON(intermediate->sr.fbc > g4x_fbc_fifo_size(1) &&
+ intermediate->fbc_en && intermediate->cxsr);
+ WARN_ON(intermediate->hpll.fbc > g4x_fbc_fifo_size(2) &&
+ intermediate->fbc_en && intermediate->hpll_en);
+
+ /*
+ * If our intermediate WM are identical to the final WM, then we can
+ * omit the post-vblank programming; only update if it's different.
+ */
+ if (memcmp(intermediate, optimal, sizeof(*intermediate)) != 0)
+ crtc_state->wm.need_postvbl_update = true;
+
+ return 0;
+}
+
+static void g4x_merge_wm(struct drm_i915_private *dev_priv,
+ struct g4x_wm_values *wm)
+{
+ struct intel_crtc *crtc;
+ int num_active_crtcs = 0;
+
+ wm->cxsr = true;
+ wm->hpll_en = true;
+ wm->fbc_en = true;
+
+ for_each_intel_crtc(&dev_priv->drm, crtc) {
+ const struct g4x_wm_state *wm_state = &crtc->wm.active.g4x;
+
+ if (!crtc->active)
+ continue;
+
+ if (!wm_state->cxsr)
+ wm->cxsr = false;
+ if (!wm_state->hpll_en)
+ wm->hpll_en = false;
+ if (!wm_state->fbc_en)
+ wm->fbc_en = false;
+
+ num_active_crtcs++;
+ }
+
+ if (num_active_crtcs != 1) {
+ wm->cxsr = false;
+ wm->hpll_en = false;
+ wm->fbc_en = false;
+ }
+
+ for_each_intel_crtc(&dev_priv->drm, crtc) {
+ const struct g4x_wm_state *wm_state = &crtc->wm.active.g4x;
+ enum pipe pipe = crtc->pipe;
+
+ wm->pipe[pipe] = wm_state->wm;
+ if (crtc->active && wm->cxsr)
+ wm->sr = wm_state->sr;
+ if (crtc->active && wm->hpll_en)
+ wm->hpll = wm_state->hpll;
+ }
}
-#undef FW_WM_VLV
+static void g4x_program_watermarks(struct drm_i915_private *dev_priv)
+{
+ struct g4x_wm_values *old_wm = &dev_priv->wm.g4x;
+ struct g4x_wm_values new_wm = {};
+
+ g4x_merge_wm(dev_priv, &new_wm);
+
+ if (memcmp(old_wm, &new_wm, sizeof(new_wm)) == 0)
+ return;
+
+ if (is_disabling(old_wm->cxsr, new_wm.cxsr, true))
+ _intel_set_memory_cxsr(dev_priv, false);
+
+ g4x_write_wm_values(dev_priv, &new_wm);
+
+ if (is_enabling(old_wm->cxsr, new_wm.cxsr, true))
+ _intel_set_memory_cxsr(dev_priv, true);
+
+ *old_wm = new_wm;
+}
+
+static void g4x_initial_watermarks(struct intel_atomic_state *state,
+ struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
+
+ mutex_lock(&dev_priv->wm.wm_mutex);
+ crtc->wm.active.g4x = crtc_state->wm.g4x.intermediate;
+ g4x_program_watermarks(dev_priv);
+ mutex_unlock(&dev_priv->wm.wm_mutex);
+}
+
+static void g4x_optimize_watermarks(struct intel_atomic_state *state,
+ struct intel_crtc_state *crtc_state)
+{
+ struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
+ struct intel_crtc *intel_crtc = to_intel_crtc(crtc_state->base.crtc);
+
+ if (!crtc_state->wm.need_postvbl_update)
+ return;
+
+ mutex_lock(&dev_priv->wm.wm_mutex);
+ intel_crtc->wm.active.g4x = crtc_state->wm.g4x.optimal;
+ g4x_program_watermarks(dev_priv);
+ mutex_unlock(&dev_priv->wm.wm_mutex);
+}
/* latency must be in 0.1us units. */
static unsigned int vlv_wm_method2(unsigned int pixel_rate,
- unsigned int pipe_htotal,
- unsigned int horiz_pixels,
+ unsigned int htotal,
+ unsigned int width,
unsigned int cpp,
unsigned int latency)
{
unsigned int ret;
- ret = (latency * pixel_rate) / (pipe_htotal * 10000);
- ret = (ret + 1) * horiz_pixels * cpp;
+ ret = intel_wm_method2(pixel_rate, htotal,
+ width, cpp, latency);
ret = DIV_ROUND_UP(ret, 64);
return ret;
if (dev_priv->wm.pri_latency[level] == 0)
return USHRT_MAX;
- if (!plane_state->base.visible)
+ if (!intel_wm_plane_visible(crtc_state, plane_state))
return 0;
cpp = plane_state->base.fb->format->cpp[0];
clock = adjusted_mode->crtc_clock;
htotal = adjusted_mode->crtc_htotal;
width = crtc_state->pipe_src_w;
- if (WARN_ON(htotal == 0))
- htotal = 1;
- if (plane->base.type == DRM_PLANE_TYPE_CURSOR) {
+ if (plane->id == PLANE_CURSOR) {
/*
* FIXME the formula gives values that are
* too big for the cursor FIFO, and hence we
static int vlv_compute_fifo(struct intel_crtc_state *crtc_state)
{
struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
- const struct vlv_pipe_wm *raw =
+ const struct g4x_pipe_wm *raw =
&crtc_state->wm.vlv.raw[VLV_WM_LEVEL_PM2];
struct vlv_fifo_state *fifo_state = &crtc_state->wm.vlv.fifo_state;
unsigned int active_planes = crtc_state->active_planes & ~BIT(PLANE_CURSOR);
return 0;
}
-static int vlv_num_wm_levels(struct drm_i915_private *dev_priv)
-{
- return dev_priv->wm.max_level + 1;
-}
-
/* mark all levels starting from 'level' as invalid */
static void vlv_invalidate_wms(struct intel_crtc *crtc,
struct vlv_wm_state *wm_state, int level)
{
struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
- for (; level < vlv_num_wm_levels(dev_priv); level++) {
+ for (; level < intel_wm_num_levels(dev_priv); level++) {
enum plane_id plane_id;
for_each_plane_id_on_crtc(crtc, plane_id)
int level, enum plane_id plane_id, u16 value)
{
struct drm_i915_private *dev_priv = to_i915(crtc_state->base.crtc->dev);
- int num_levels = vlv_num_wm_levels(dev_priv);
+ int num_levels = intel_wm_num_levels(dev_priv);
bool dirty = false;
for (; level < num_levels; level++) {
- struct vlv_pipe_wm *raw = &crtc_state->wm.vlv.raw[level];
+ struct g4x_pipe_wm *raw = &crtc_state->wm.vlv.raw[level];
dirty |= raw->plane[plane_id] != value;
raw->plane[plane_id] = value;
return dirty;
}
-static bool vlv_plane_wm_compute(struct intel_crtc_state *crtc_state,
- const struct intel_plane_state *plane_state)
+static bool vlv_raw_plane_wm_compute(struct intel_crtc_state *crtc_state,
+ const struct intel_plane_state *plane_state)
{
struct intel_plane *plane = to_intel_plane(plane_state->base.plane);
enum plane_id plane_id = plane->id;
- int num_levels = vlv_num_wm_levels(to_i915(plane->base.dev));
+ int num_levels = intel_wm_num_levels(to_i915(plane->base.dev));
int level;
bool dirty = false;
- if (!plane_state->base.visible) {
+ if (!intel_wm_plane_visible(crtc_state, plane_state)) {
dirty |= vlv_raw_plane_wm_set(crtc_state, 0, plane_id, 0);
goto out;
}
for (level = 0; level < num_levels; level++) {
- struct vlv_pipe_wm *raw = &crtc_state->wm.vlv.raw[level];
+ struct g4x_pipe_wm *raw = &crtc_state->wm.vlv.raw[level];
int wm = vlv_compute_wm_level(crtc_state, plane_state, level);
int max_wm = plane_id == PLANE_CURSOR ? 63 : 511;
out:
if (dirty)
- DRM_DEBUG_KMS("%s wms: [0]=%d,[1]=%d,[2]=%d\n",
+ DRM_DEBUG_KMS("%s watermarks: PM2=%d, PM5=%d, DDR DVFS=%d\n",
plane->base.name,
crtc_state->wm.vlv.raw[VLV_WM_LEVEL_PM2].plane[plane_id],
crtc_state->wm.vlv.raw[VLV_WM_LEVEL_PM5].plane[plane_id],
return dirty;
}
-static bool vlv_plane_wm_is_valid(const struct intel_crtc_state *crtc_state,
- enum plane_id plane_id, int level)
+static bool vlv_raw_plane_wm_is_valid(const struct intel_crtc_state *crtc_state,
+ enum plane_id plane_id, int level)
{
- const struct vlv_pipe_wm *raw =
+ const struct g4x_pipe_wm *raw =
&crtc_state->wm.vlv.raw[level];
const struct vlv_fifo_state *fifo_state =
&crtc_state->wm.vlv.fifo_state;
return raw->plane[plane_id] <= fifo_state->plane[plane_id];
}
-static bool vlv_crtc_wm_is_valid(const struct intel_crtc_state *crtc_state, int level)
+static bool vlv_raw_crtc_wm_is_valid(const struct intel_crtc_state *crtc_state, int level)
{
- return vlv_plane_wm_is_valid(crtc_state, PLANE_PRIMARY, level) &&
- vlv_plane_wm_is_valid(crtc_state, PLANE_SPRITE0, level) &&
- vlv_plane_wm_is_valid(crtc_state, PLANE_SPRITE1, level) &&
- vlv_plane_wm_is_valid(crtc_state, PLANE_CURSOR, level);
+ return vlv_raw_plane_wm_is_valid(crtc_state, PLANE_PRIMARY, level) &&
+ vlv_raw_plane_wm_is_valid(crtc_state, PLANE_SPRITE0, level) &&
+ vlv_raw_plane_wm_is_valid(crtc_state, PLANE_SPRITE1, level) &&
+ vlv_raw_plane_wm_is_valid(crtc_state, PLANE_CURSOR, level);
}
static int vlv_compute_pipe_wm(struct intel_crtc_state *crtc_state)
old_plane_state->base.crtc != &crtc->base)
continue;
- if (vlv_plane_wm_compute(crtc_state, plane_state))
+ if (vlv_raw_plane_wm_compute(crtc_state, plane_state))
dirty |= BIT(plane->id);
}
}
/* initially allow all levels */
- wm_state->num_levels = vlv_num_wm_levels(dev_priv);
+ wm_state->num_levels = intel_wm_num_levels(dev_priv);
/*
* Note that enabling cxsr with no primary/sprite planes
* enabled can wedge the pipe. Hence we only allow cxsr
wm_state->cxsr = crtc->pipe != PIPE_C && num_active_planes == 1;
for (level = 0; level < wm_state->num_levels; level++) {
- const struct vlv_pipe_wm *raw = &crtc_state->wm.vlv.raw[level];
+ const struct g4x_pipe_wm *raw = &crtc_state->wm.vlv.raw[level];
const int sr_fifo_size = INTEL_INFO(dev_priv)->num_pipes * 512 - 1;
- if (!vlv_crtc_wm_is_valid(crtc_state, level))
+ if (!vlv_raw_crtc_wm_is_valid(crtc_state, level))
break;
for_each_plane_id_on_crtc(crtc, plane_id) {
}
}
-static bool is_disabling(int old, int new, int threshold)
-{
- return old >= threshold && new < threshold;
-}
-
-static bool is_enabling(int old, int new, int threshold)
-{
- return old < threshold && new >= threshold;
-}
-
static void vlv_program_watermarks(struct drm_i915_private *dev_priv)
{
struct vlv_wm_values *old_wm = &dev_priv->wm.vlv;
mutex_unlock(&dev_priv->wm.wm_mutex);
}
-#define single_plane_enabled(mask) is_power_of_2(mask)
-
-static void g4x_update_wm(struct intel_crtc *crtc)
-{
- struct drm_i915_private *dev_priv = to_i915(crtc->base.dev);
- static const int sr_latency_ns = 12000;
- int planea_wm, planeb_wm, cursora_wm, cursorb_wm;
- int plane_sr, cursor_sr;
- unsigned int enabled = 0;
- bool cxsr_enabled;
-
- if (g4x_compute_wm0(dev_priv, PIPE_A,
- &g4x_wm_info, pessimal_latency_ns,
- &g4x_cursor_wm_info, pessimal_latency_ns,
- &planea_wm, &cursora_wm))
- enabled |= 1 << PIPE_A;
-
- if (g4x_compute_wm0(dev_priv, PIPE_B,
- &g4x_wm_info, pessimal_latency_ns,
- &g4x_cursor_wm_info, pessimal_latency_ns,
- &planeb_wm, &cursorb_wm))
- enabled |= 1 << PIPE_B;
-
- if (single_plane_enabled(enabled) &&
- g4x_compute_srwm(dev_priv, ffs(enabled) - 1,
- sr_latency_ns,
- &g4x_wm_info,
- &g4x_cursor_wm_info,
- &plane_sr, &cursor_sr)) {
- cxsr_enabled = true;
- } else {
- cxsr_enabled = false;
- intel_set_memory_cxsr(dev_priv, false);
- plane_sr = cursor_sr = 0;
- }
-
- DRM_DEBUG_KMS("Setting FIFO watermarks - A: plane=%d, cursor=%d, "
- "B: plane=%d, cursor=%d, SR: plane=%d, cursor=%d\n",
- planea_wm, cursora_wm,
- planeb_wm, cursorb_wm,
- plane_sr, cursor_sr);
-
- I915_WRITE(DSPFW1,
- FW_WM(plane_sr, SR) |
- FW_WM(cursorb_wm, CURSORB) |
- FW_WM(planeb_wm, PLANEB) |
- FW_WM(planea_wm, PLANEA));
- I915_WRITE(DSPFW2,
- (I915_READ(DSPFW2) & ~DSPFW_CURSORA_MASK) |
- FW_WM(cursora_wm, CURSORA));
- /* HPLL off in SR has some issues on G4x... disable it */
- I915_WRITE(DSPFW3,
- (I915_READ(DSPFW3) & ~(DSPFW_HPLL_SR_EN | DSPFW_CURSOR_SR_MASK)) |
- FW_WM(cursor_sr, CURSOR_SR));
-
- if (cxsr_enabled)
- intel_set_memory_cxsr(dev_priv, true);
-}
-
static void i965_update_wm(struct intel_crtc *unused_crtc)
{
struct drm_i915_private *dev_priv = to_i915(unused_crtc->base.dev);
int htotal = adjusted_mode->crtc_htotal;
int hdisplay = crtc->config->pipe_src_w;
int cpp = fb->format->cpp[0];
- unsigned long line_time_us;
int entries;
- line_time_us = max(htotal * 1000 / clock, 1);
-
- /* Use ns/us then divide to preserve precision */
- entries = (((sr_latency_ns / line_time_us) + 1000) / 1000) *
- cpp * hdisplay;
+ entries = intel_wm_method2(clock, htotal,
+ hdisplay, cpp, sr_latency_ns / 100);
entries = DIV_ROUND_UP(entries, I915_FIFO_LINE_SIZE);
srwm = I965_FIFO_SIZE - entries;
if (srwm < 0)
DRM_DEBUG_KMS("self-refresh entries: %d, wm: %d\n",
entries, srwm);
- entries = (((sr_latency_ns / line_time_us) + 1000) / 1000) *
- cpp * crtc->base.cursor->state->crtc_w;
+ entries = intel_wm_method2(clock, htotal,
+ crtc->base.cursor->state->crtc_w, 4,
+ sr_latency_ns / 100);
entries = DIV_ROUND_UP(entries,
- i965_cursor_wm_info.cacheline_size);
- cursor_sr = i965_cursor_wm_info.fifo_size -
- (entries + i965_cursor_wm_info.guard_size);
+ i965_cursor_wm_info.cacheline_size) +
+ i965_cursor_wm_info.guard_size;
+ cursor_sr = i965_cursor_wm_info.fifo_size - entries;
if (cursor_sr > i965_cursor_wm_info.max_wm)
cursor_sr = i965_cursor_wm_info.max_wm;
int htotal = adjusted_mode->crtc_htotal;
int hdisplay = enabled->config->pipe_src_w;
int cpp;
- unsigned long line_time_us;
int entries;
if (IS_I915GM(dev_priv) || IS_I945GM(dev_priv))
else
cpp = fb->format->cpp[0];
- line_time_us = max(htotal * 1000 / clock, 1);
-
- /* Use ns/us then divide to preserve precision */
- entries = (((sr_latency_ns / line_time_us) + 1000) / 1000) *
- cpp * hdisplay;
+ entries = intel_wm_method2(clock, htotal, hdisplay, cpp,
+ sr_latency_ns / 100);
entries = DIV_ROUND_UP(entries, wm_info->cacheline_size);
DRM_DEBUG_KMS("self-refresh entries: %d\n", entries);
srwm = wm_info->fifo_size - entries;
}
/* latency must be in 0.1us units. */
-static uint32_t ilk_wm_method1(uint32_t pixel_rate, uint8_t cpp, uint32_t latency)
+static unsigned int ilk_wm_method1(unsigned int pixel_rate,
+ unsigned int cpp,
+ unsigned int latency)
{
- uint64_t ret;
-
- if (WARN(latency == 0, "Latency value missing\n"))
- return UINT_MAX;
+ unsigned int ret;
- ret = (uint64_t) pixel_rate * cpp * latency;
- ret = DIV_ROUND_UP_ULL(ret, 64 * 10000) + 2;
+ ret = intel_wm_method1(pixel_rate, cpp, latency);
+ ret = DIV_ROUND_UP(ret, 64) + 2;
return ret;
}
/* latency must be in 0.1us units. */
-static uint32_t ilk_wm_method2(uint32_t pixel_rate, uint32_t pipe_htotal,
- uint32_t horiz_pixels, uint8_t cpp,
- uint32_t latency)
+static unsigned int ilk_wm_method2(unsigned int pixel_rate,
+ unsigned int htotal,
+ unsigned int width,
+ unsigned int cpp,
+ unsigned int latency)
{
- uint32_t ret;
-
- if (WARN(latency == 0, "Latency value missing\n"))
- return UINT_MAX;
- if (WARN_ON(!pipe_htotal))
- return UINT_MAX;
+ unsigned int ret;
- ret = (latency * pixel_rate) / (pipe_htotal * 10000);
- ret = (ret + 1) * horiz_pixels * cpp;
+ ret = intel_wm_method2(pixel_rate, htotal,
+ width, cpp, latency);
ret = DIV_ROUND_UP(ret, 64) + 2;
+
return ret;
}
* Return value is provided in 16.16 fixed point form to retain fractional part.
* Caller should take care of dividing & rounding off the value.
*/
-static uint32_t
+static uint_fixed_16_16_t
skl_plane_downscale_amount(const struct intel_crtc_state *cstate,
const struct intel_plane_state *pstate)
{
struct intel_plane *plane = to_intel_plane(pstate->base.plane);
- uint32_t downscale_h, downscale_w;
uint32_t src_w, src_h, dst_w, dst_h;
+ uint_fixed_16_16_t fp_w_ratio, fp_h_ratio;
+ uint_fixed_16_16_t downscale_h, downscale_w;
if (WARN_ON(!intel_wm_plane_visible(cstate, pstate)))
- return DRM_PLANE_HELPER_NO_SCALING;
+ return u32_to_fixed_16_16(0);
/* n.b., src is 16.16 fixed point, dst is whole integer */
if (plane->id == PLANE_CURSOR) {
- src_w = pstate->base.src_w;
- src_h = pstate->base.src_h;
+ src_w = pstate->base.src_w >> 16;
+ src_h = pstate->base.src_h >> 16;
dst_w = pstate->base.crtc_w;
dst_h = pstate->base.crtc_h;
} else {
- src_w = drm_rect_width(&pstate->base.src);
- src_h = drm_rect_height(&pstate->base.src);
+ src_w = drm_rect_width(&pstate->base.src) >> 16;
+ src_h = drm_rect_height(&pstate->base.src) >> 16;
dst_w = drm_rect_width(&pstate->base.dst);
dst_h = drm_rect_height(&pstate->base.dst);
}
if (drm_rotation_90_or_270(pstate->base.rotation))
swap(dst_w, dst_h);
- downscale_h = max(src_h / dst_h, (uint32_t)DRM_PLANE_HELPER_NO_SCALING);
- downscale_w = max(src_w / dst_w, (uint32_t)DRM_PLANE_HELPER_NO_SCALING);
+ fp_w_ratio = fixed_16_16_div(src_w, dst_w);
+ fp_h_ratio = fixed_16_16_div(src_h, dst_h);
+ downscale_w = max_fixed_16_16(fp_w_ratio, u32_to_fixed_16_16(1));
+ downscale_h = max_fixed_16_16(fp_h_ratio, u32_to_fixed_16_16(1));
- /* Provide result in 16.16 fixed point */
- return (uint64_t)downscale_w * downscale_h >> 16;
+ return mul_fixed16(downscale_w, downscale_h);
}
static unsigned int
{
struct intel_plane *plane = to_intel_plane(pstate->plane);
struct intel_plane_state *intel_pstate = to_intel_plane_state(pstate);
- uint32_t down_scale_amount, data_rate;
+ uint32_t data_rate;
uint32_t width = 0, height = 0;
struct drm_framebuffer *fb;
u32 format;
+ uint_fixed_16_16_t down_scale_amount;
if (!intel_pstate->base.visible)
return 0;
down_scale_amount = skl_plane_downscale_amount(cstate, intel_pstate);
- return (uint64_t)data_rate * down_scale_amount >> 16;
+ return mul_round_up_u32_fixed16(data_rate, down_scale_amount);
}
/*
int num_active;
unsigned plane_data_rate[I915_MAX_PLANES] = {};
unsigned plane_y_data_rate[I915_MAX_PLANES] = {};
+ uint16_t total_min_blocks = 0;
/* Clear the partitioning for disabled planes. */
memset(ddb->plane[pipe], 0, sizeof(ddb->plane[pipe]));
skl_ddb_get_pipe_allocation_limits(dev, cstate, alloc, &num_active);
alloc_size = skl_ddb_entry_size(alloc);
- if (alloc_size == 0) {
- memset(ddb->plane[pipe], 0, sizeof(ddb->plane[pipe]));
+ if (alloc_size == 0)
return 0;
- }
skl_ddb_calc_min(cstate, num_active, minimum, y_minimum);
*/
for_each_plane_id_on_crtc(intel_crtc, plane_id) {
- alloc_size -= minimum[plane_id];
- alloc_size -= y_minimum[plane_id];
+ total_min_blocks += minimum[plane_id];
+ total_min_blocks += y_minimum[plane_id];
+ }
+
+ if (total_min_blocks > alloc_size) {
+ DRM_DEBUG_KMS("Requested display configuration exceeds system DDB limitations");
+ DRM_DEBUG_KMS("minimum required %d/%d\n", total_min_blocks,
+ alloc_size);
+ return -EINVAL;
}
+ alloc_size -= total_min_blocks;
ddb->plane[pipe][PLANE_CURSOR].start = alloc->end - minimum[PLANE_CURSOR];
ddb->plane[pipe][PLANE_CURSOR].end = alloc->end;
return FP_16_16_MAX;
wm_intermediate_val = latency * pixel_rate * cpp;
- ret = fixed_16_16_div_round_up_u64(wm_intermediate_val, 1000 * 512);
+ ret = fixed_16_16_div_u64(wm_intermediate_val, 1000 * 512);
return ret;
}
return ret;
}
-static uint32_t skl_adjusted_plane_pixel_rate(const struct intel_crtc_state *cstate,
- struct intel_plane_state *pstate)
+static uint_fixed_16_16_t
+intel_get_linetime_us(struct intel_crtc_state *cstate)
+{
+ uint32_t pixel_rate;
+ uint32_t crtc_htotal;
+ uint_fixed_16_16_t linetime_us;
+
+ if (!cstate->base.active)
+ return u32_to_fixed_16_16(0);
+
+ pixel_rate = cstate->pixel_rate;
+
+ if (WARN_ON(pixel_rate == 0))
+ return u32_to_fixed_16_16(0);
+
+ crtc_htotal = cstate->base.adjusted_mode.crtc_htotal;
+ linetime_us = fixed_16_16_div_u64(crtc_htotal * 1000, pixel_rate);
+
+ return linetime_us;
+}
+
+static uint32_t
+skl_adjusted_plane_pixel_rate(const struct intel_crtc_state *cstate,
+ const struct intel_plane_state *pstate)
{
uint64_t adjusted_pixel_rate;
- uint64_t downscale_amount;
- uint64_t pixel_rate;
+ uint_fixed_16_16_t downscale_amount;
/* Shouldn't reach here on disabled planes... */
if (WARN_ON(!intel_wm_plane_visible(cstate, pstate)))
adjusted_pixel_rate = cstate->pixel_rate;
downscale_amount = skl_plane_downscale_amount(cstate, pstate);
- pixel_rate = adjusted_pixel_rate * downscale_amount >> 16;
- WARN_ON(pixel_rate != clamp_t(uint32_t, pixel_rate, 0, ~0));
-
- return pixel_rate;
+ return mul_round_up_u32_fixed16(adjusted_pixel_rate,
+ downscale_amount);
}
static int skl_compute_plane_wm(const struct drm_i915_private *dev_priv,
struct intel_crtc_state *cstate,
- struct intel_plane_state *intel_pstate,
+ const struct intel_plane_state *intel_pstate,
uint16_t ddb_allocation,
int level,
uint16_t *out_blocks, /* out */
bool *enabled /* out */)
{
struct intel_plane *plane = to_intel_plane(intel_pstate->base.plane);
- struct drm_plane_state *pstate = &intel_pstate->base;
- struct drm_framebuffer *fb = pstate->fb;
+ const struct drm_plane_state *pstate = &intel_pstate->base;
+ const struct drm_framebuffer *fb = pstate->fb;
uint32_t latency = dev_priv->wm.skl_latency[level];
uint_fixed_16_16_t method1, method2;
uint_fixed_16_16_t plane_blocks_per_line;
if (y_tiled) {
interm_pbpl = DIV_ROUND_UP(plane_bytes_per_line *
y_min_scanlines, 512);
- plane_blocks_per_line =
- fixed_16_16_div_round_up(interm_pbpl, y_min_scanlines);
+ plane_blocks_per_line = fixed_16_16_div(interm_pbpl,
+ y_min_scanlines);
} else if (x_tiled) {
interm_pbpl = DIV_ROUND_UP(plane_bytes_per_line, 512);
plane_blocks_per_line = u32_to_fixed_16_16(interm_pbpl);
if (y_tiled) {
selected_result = max_fixed_16_16(method2, y_tile_minimum);
} else {
+ uint32_t linetime_us;
+
+ linetime_us = fixed_16_16_to_u32_round_up(
+ intel_get_linetime_us(cstate));
if ((cpp * cstate->base.adjusted_mode.crtc_htotal / 512 < 1) &&
(plane_bytes_per_line / 512 < 1))
selected_result = method2;
- else if ((ddb_allocation /
+ else if ((ddb_allocation && ddb_allocation /
fixed_16_16_to_u32_round_up(plane_blocks_per_line)) >= 1)
selected_result = min_fixed_16_16(method1, method2);
+ else if (latency >= linetime_us)
+ selected_result = min_fixed_16_16(method1, method2);
else
selected_result = method1;
}
res_blocks = fixed_16_16_to_u32_round_up(selected_result) + 1;
- res_lines = DIV_ROUND_UP(selected_result.val,
- plane_blocks_per_line.val);
+ res_lines = div_round_up_fixed16(selected_result,
+ plane_blocks_per_line);
if (level >= 1 && level <= 7) {
if (y_tiled) {
}
static int
-skl_compute_wm_level(const struct drm_i915_private *dev_priv,
- struct skl_ddb_allocation *ddb,
- struct intel_crtc_state *cstate,
- struct intel_plane *intel_plane,
- int level,
- struct skl_wm_level *result)
+skl_compute_wm_levels(const struct drm_i915_private *dev_priv,
+ struct skl_ddb_allocation *ddb,
+ struct intel_crtc_state *cstate,
+ const struct intel_plane_state *intel_pstate,
+ struct skl_plane_wm *wm)
{
- struct drm_atomic_state *state = cstate->base.state;
struct intel_crtc *intel_crtc = to_intel_crtc(cstate->base.crtc);
- struct drm_plane *plane = &intel_plane->base;
- struct intel_plane_state *intel_pstate = NULL;
+ struct drm_plane *plane = intel_pstate->base.plane;
+ struct intel_plane *intel_plane = to_intel_plane(plane);
uint16_t ddb_blocks;
enum pipe pipe = intel_crtc->pipe;
+ int level, max_level = ilk_wm_max_level(dev_priv);
int ret;
- if (state)
- intel_pstate =
- intel_atomic_get_existing_plane_state(state,
- intel_plane);
-
- /*
- * Note: If we start supporting multiple pending atomic commits against
- * the same planes/CRTC's in the future, plane->state will no longer be
- * the correct pre-state to use for the calculations here and we'll
- * need to change where we get the 'unchanged' plane data from.
- *
- * For now this is fine because we only allow one queued commit against
- * a CRTC. Even if the plane isn't modified by this transaction and we
- * don't have a plane lock, we still have the CRTC's lock, so we know
- * that no other transactions are racing with us to update it.
- */
- if (!intel_pstate)
- intel_pstate = to_intel_plane_state(plane->state);
-
- WARN_ON(!intel_pstate->base.fb);
+ if (WARN_ON(!intel_pstate->base.fb))
+ return -EINVAL;
ddb_blocks = skl_ddb_entry_size(&ddb->plane[pipe][intel_plane->id]);
- ret = skl_compute_plane_wm(dev_priv,
- cstate,
- intel_pstate,
- ddb_blocks,
- level,
- &result->plane_res_b,
- &result->plane_res_l,
- &result->plane_en);
- if (ret)
- return ret;
+ for (level = 0; level <= max_level; level++) {
+ struct skl_wm_level *result = &wm->wm[level];
+
+ ret = skl_compute_plane_wm(dev_priv,
+ cstate,
+ intel_pstate,
+ ddb_blocks,
+ level,
+ &result->plane_res_b,
+ &result->plane_res_l,
+ &result->plane_en);
+ if (ret)
+ return ret;
+ }
return 0;
}
{
struct drm_atomic_state *state = cstate->base.state;
struct drm_i915_private *dev_priv = to_i915(state->dev);
- uint32_t pixel_rate;
+ uint_fixed_16_16_t linetime_us;
uint32_t linetime_wm;
- if (!cstate->base.active)
- return 0;
-
- pixel_rate = cstate->pixel_rate;
+ linetime_us = intel_get_linetime_us(cstate);
- if (WARN_ON(pixel_rate == 0))
+ if (is_fixed16_zero(linetime_us))
return 0;
- linetime_wm = DIV_ROUND_UP(8 * cstate->base.adjusted_mode.crtc_htotal *
- 1000, pixel_rate);
+ linetime_wm = fixed_16_16_to_u32_round_up(mul_u32_fixed_16_16(8,
+ linetime_us));
/* Display WA #1135: bxt. */
if (IS_BROXTON(dev_priv) && dev_priv->ipc_enabled)
struct skl_pipe_wm *pipe_wm)
{
struct drm_device *dev = cstate->base.crtc->dev;
+ struct drm_crtc_state *crtc_state = &cstate->base;
const struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane;
+ struct drm_plane *plane;
+ const struct drm_plane_state *pstate;
struct skl_plane_wm *wm;
- int level, max_level = ilk_wm_max_level(dev_priv);
int ret;
/*
*/
memset(pipe_wm->planes, 0, sizeof(pipe_wm->planes));
- for_each_intel_plane_mask(&dev_priv->drm,
- intel_plane,
- cstate->base.plane_mask) {
- wm = &pipe_wm->planes[intel_plane->id];
+ drm_atomic_crtc_state_for_each_plane_state(plane, pstate, crtc_state) {
+ const struct intel_plane_state *intel_pstate =
+ to_intel_plane_state(pstate);
+ enum plane_id plane_id = to_intel_plane(plane)->id;
+
+ wm = &pipe_wm->planes[plane_id];
- for (level = 0; level <= max_level; level++) {
- ret = skl_compute_wm_level(dev_priv, ddb, cstate,
- intel_plane, level,
- &wm->wm[level]);
- if (ret)
- return ret;
- }
+ ret = skl_compute_wm_levels(dev_priv, ddb, cstate,
+ intel_pstate, wm);
+ if (ret)
+ return ret;
skl_compute_transition_wm(cstate, &wm->trans_wm);
}
pipe_wm->linetime = skl_compute_linetime_wm(cstate);
#define _FW_WM_VLV(value, plane) \
(((value) & DSPFW_ ## plane ## _MASK_VLV) >> DSPFW_ ## plane ## _SHIFT)
+static void g4x_read_wm_values(struct drm_i915_private *dev_priv,
+ struct g4x_wm_values *wm)
+{
+ uint32_t tmp;
+
+ tmp = I915_READ(DSPFW1);
+ wm->sr.plane = _FW_WM(tmp, SR);
+ wm->pipe[PIPE_B].plane[PLANE_CURSOR] = _FW_WM(tmp, CURSORB);
+ wm->pipe[PIPE_B].plane[PLANE_PRIMARY] = _FW_WM(tmp, PLANEB);
+ wm->pipe[PIPE_A].plane[PLANE_PRIMARY] = _FW_WM(tmp, PLANEA);
+
+ tmp = I915_READ(DSPFW2);
+ wm->fbc_en = tmp & DSPFW_FBC_SR_EN;
+ wm->sr.fbc = _FW_WM(tmp, FBC_SR);
+ wm->hpll.fbc = _FW_WM(tmp, FBC_HPLL_SR);
+ wm->pipe[PIPE_B].plane[PLANE_SPRITE0] = _FW_WM(tmp, SPRITEB);
+ wm->pipe[PIPE_A].plane[PLANE_CURSOR] = _FW_WM(tmp, CURSORA);
+ wm->pipe[PIPE_A].plane[PLANE_SPRITE0] = _FW_WM(tmp, SPRITEA);
+
+ tmp = I915_READ(DSPFW3);
+ wm->hpll_en = tmp & DSPFW_HPLL_SR_EN;
+ wm->sr.cursor = _FW_WM(tmp, CURSOR_SR);
+ wm->hpll.cursor = _FW_WM(tmp, HPLL_CURSOR);
+ wm->hpll.plane = _FW_WM(tmp, HPLL_SR);
+}
+
static void vlv_read_wm_values(struct drm_i915_private *dev_priv,
struct vlv_wm_values *wm)
{
#undef _FW_WM
#undef _FW_WM_VLV
+void g4x_wm_get_hw_state(struct drm_device *dev)
+{
+ struct drm_i915_private *dev_priv = to_i915(dev);
+ struct g4x_wm_values *wm = &dev_priv->wm.g4x;
+ struct intel_crtc *crtc;
+
+ g4x_read_wm_values(dev_priv, wm);
+
+ wm->cxsr = I915_READ(FW_BLC_SELF) & FW_BLC_SELF_EN;
+
+ for_each_intel_crtc(dev, crtc) {
+ struct intel_crtc_state *crtc_state =
+ to_intel_crtc_state(crtc->base.state);
+ struct g4x_wm_state *active = &crtc->wm.active.g4x;
+ struct g4x_pipe_wm *raw;
+ enum pipe pipe = crtc->pipe;
+ enum plane_id plane_id;
+ int level, max_level;
+
+ active->cxsr = wm->cxsr;
+ active->hpll_en = wm->hpll_en;
+ active->fbc_en = wm->fbc_en;
+
+ active->sr = wm->sr;
+ active->hpll = wm->hpll;
+
+ for_each_plane_id_on_crtc(crtc, plane_id) {
+ active->wm.plane[plane_id] =
+ wm->pipe[pipe].plane[plane_id];
+ }
+
+ if (wm->cxsr && wm->hpll_en)
+ max_level = G4X_WM_LEVEL_HPLL;
+ else if (wm->cxsr)
+ max_level = G4X_WM_LEVEL_SR;
+ else
+ max_level = G4X_WM_LEVEL_NORMAL;
+
+ level = G4X_WM_LEVEL_NORMAL;
+ raw = &crtc_state->wm.g4x.raw[level];
+ for_each_plane_id_on_crtc(crtc, plane_id)
+ raw->plane[plane_id] = active->wm.plane[plane_id];
+
+ if (++level > max_level)
+ goto out;
+
+ raw = &crtc_state->wm.g4x.raw[level];
+ raw->plane[PLANE_PRIMARY] = active->sr.plane;
+ raw->plane[PLANE_CURSOR] = active->sr.cursor;
+ raw->plane[PLANE_SPRITE0] = 0;
+ raw->fbc = active->sr.fbc;
+
+ if (++level > max_level)
+ goto out;
+
+ raw = &crtc_state->wm.g4x.raw[level];
+ raw->plane[PLANE_PRIMARY] = active->hpll.plane;
+ raw->plane[PLANE_CURSOR] = active->hpll.cursor;
+ raw->plane[PLANE_SPRITE0] = 0;
+ raw->fbc = active->hpll.fbc;
+
+ out:
+ for_each_plane_id_on_crtc(crtc, plane_id)
+ g4x_raw_plane_wm_set(crtc_state, level,
+ plane_id, USHRT_MAX);
+ g4x_raw_fbc_wm_set(crtc_state, level, USHRT_MAX);
+
+ crtc_state->wm.g4x.optimal = *active;
+ crtc_state->wm.g4x.intermediate = *active;
+
+ DRM_DEBUG_KMS("Initial watermarks: pipe %c, plane=%d, cursor=%d, sprite=%d\n",
+ pipe_name(pipe),
+ wm->pipe[pipe].plane[PLANE_PRIMARY],
+ wm->pipe[pipe].plane[PLANE_CURSOR],
+ wm->pipe[pipe].plane[PLANE_SPRITE0]);
+ }
+
+ DRM_DEBUG_KMS("Initial SR watermarks: plane=%d, cursor=%d fbc=%d\n",
+ wm->sr.plane, wm->sr.cursor, wm->sr.fbc);
+ DRM_DEBUG_KMS("Initial HPLL watermarks: plane=%d, SR cursor=%d fbc=%d\n",
+ wm->hpll.plane, wm->hpll.cursor, wm->hpll.fbc);
+ DRM_DEBUG_KMS("Initial SR=%s HPLL=%s FBC=%s\n",
+ yesno(wm->cxsr), yesno(wm->hpll_en), yesno(wm->fbc_en));
+}
+
+void g4x_wm_sanitize(struct drm_i915_private *dev_priv)
+{
+ struct intel_plane *plane;
+ struct intel_crtc *crtc;
+
+ mutex_lock(&dev_priv->wm.wm_mutex);
+
+ for_each_intel_plane(&dev_priv->drm, plane) {
+ struct intel_crtc *crtc =
+ intel_get_crtc_for_pipe(dev_priv, plane->pipe);
+ struct intel_crtc_state *crtc_state =
+ to_intel_crtc_state(crtc->base.state);
+ struct intel_plane_state *plane_state =
+ to_intel_plane_state(plane->base.state);
+ struct g4x_wm_state *wm_state = &crtc_state->wm.g4x.optimal;
+ enum plane_id plane_id = plane->id;
+ int level;
+
+ if (plane_state->base.visible)
+ continue;
+
+ for (level = 0; level < 3; level++) {
+ struct g4x_pipe_wm *raw =
+ &crtc_state->wm.g4x.raw[level];
+
+ raw->plane[plane_id] = 0;
+ wm_state->wm.plane[plane_id] = 0;
+ }
+
+ if (plane_id == PLANE_PRIMARY) {
+ for (level = 0; level < 3; level++) {
+ struct g4x_pipe_wm *raw =
+ &crtc_state->wm.g4x.raw[level];
+ raw->fbc = 0;
+ }
+
+ wm_state->sr.fbc = 0;
+ wm_state->hpll.fbc = 0;
+ wm_state->fbc_en = false;
+ }
+ }
+
+ for_each_intel_crtc(&dev_priv->drm, crtc) {
+ struct intel_crtc_state *crtc_state =
+ to_intel_crtc_state(crtc->base.state);
+
+ crtc_state->wm.g4x.intermediate =
+ crtc_state->wm.g4x.optimal;
+ crtc->wm.active.g4x = crtc_state->wm.g4x.optimal;
+ }
+
+ g4x_program_watermarks(dev_priv);
+
+ mutex_unlock(&dev_priv->wm.wm_mutex);
+}
+
void vlv_wm_get_hw_state(struct drm_device *dev)
{
struct drm_i915_private *dev_priv = to_i915(dev);
active->cxsr = wm->cxsr;
for (level = 0; level < active->num_levels; level++) {
- struct vlv_pipe_wm *raw =
+ struct g4x_pipe_wm *raw =
&crtc_state->wm.vlv.raw[level];
active->sr[level].plane = wm->sr.plane;
continue;
for (level = 0; level < wm_state->num_levels; level++) {
- struct vlv_pipe_wm *raw =
+ struct g4x_pipe_wm *raw =
&crtc_state->wm.vlv.raw[level];
raw->plane[plane_id] = 0;
dev_priv->display.initial_watermarks = vlv_initial_watermarks;
dev_priv->display.optimize_watermarks = vlv_optimize_watermarks;
dev_priv->display.atomic_update_watermarks = vlv_atomic_update_fifo;
+ } else if (IS_G4X(dev_priv)) {
+ g4x_setup_wm_latency(dev_priv);
+ dev_priv->display.compute_pipe_wm = g4x_compute_pipe_wm;
+ dev_priv->display.compute_intermediate_wm = g4x_compute_intermediate_wm;
+ dev_priv->display.initial_watermarks = g4x_initial_watermarks;
+ dev_priv->display.optimize_watermarks = g4x_optimize_watermarks;
} else if (IS_PINEVIEW(dev_priv)) {
if (!intel_get_cxsr_latency(IS_PINEVIEW_G(dev_priv),
dev_priv->is_ddr3,
dev_priv->display.update_wm = NULL;
} else
dev_priv->display.update_wm = pineview_update_wm;
- } else if (IS_G4X(dev_priv)) {
- dev_priv->display.update_wm = g4x_update_wm;
} else if (IS_GEN4(dev_priv)) {
dev_priv->display.update_wm = i965_update_wm;
} else if (IS_GEN3(dev_priv)) {
I915_WRITE_FW(GEN6_PCODE_DATA1, 0);
I915_WRITE_FW(GEN6_PCODE_MAILBOX, GEN6_PCODE_READY | mbox);
- if (intel_wait_for_register_fw(dev_priv,
- GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
- 500)) {
+ if (__intel_wait_for_register_fw(dev_priv,
+ GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
+ 500, 0, NULL)) {
DRM_ERROR("timeout waiting for pcode read (%d) to finish\n", mbox);
return -ETIMEDOUT;
}
I915_WRITE_FW(GEN6_PCODE_DATA1, 0);
I915_WRITE_FW(GEN6_PCODE_MAILBOX, GEN6_PCODE_READY | mbox);
- if (intel_wait_for_register_fw(dev_priv,
- GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
- 500)) {
+ if (__intel_wait_for_register_fw(dev_priv,
+ GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
+ 500, 0, NULL)) {
DRM_ERROR("timeout waiting for pcode write (%d) to finish\n", mbox);
return -ETIMEDOUT;
}
*/
#define LEGACY_REQUEST_SIZE 200
-static int __intel_ring_space(int head, int tail, int size)
+static unsigned int __intel_ring_space(unsigned int head,
+ unsigned int tail,
+ unsigned int size)
{
- int space = head - tail;
- if (space <= 0)
- space += size;
- return space - I915_RING_FREE_SPACE;
+ /*
+ * "If the Ring Buffer Head Pointer and the Tail Pointer are on the
+ * same cacheline, the Head Pointer must not be greater than the Tail
+ * Pointer."
+ */
+ GEM_BUG_ON(!is_power_of_2(size));
+ return (head - tail - CACHELINE_BYTES) & (size - 1);
}
-void intel_ring_update_space(struct intel_ring *ring)
+unsigned int intel_ring_update_space(struct intel_ring *ring)
{
- ring->space = __intel_ring_space(ring->head, ring->tail, ring->size);
+ unsigned int space;
+
+ space = __intel_ring_space(ring->head, ring->emit, ring->size);
+
+ ring->space = space;
+ return space;
}
static int
I915_WRITE_CTL(engine, RING_CTL_SIZE(ring->size) | RING_VALID);
/* If the head is still not zero, the ring is dead */
- if (intel_wait_for_register_fw(dev_priv, RING_CTL(engine->mmio_base),
- RING_VALID, RING_VALID,
- 50)) {
+ if (intel_wait_for_register(dev_priv, RING_CTL(engine->mmio_base),
+ RING_VALID, RING_VALID,
+ 50)) {
DRM_ERROR("%s initialization failed "
"ctl %08x (valid? %d) head %08x [%08x] tail %08x [%08x] start %08x [expected %08x]\n",
engine->name,
i915_gem_request_submit(request);
- assert_ring_tail_valid(request->ring, request->tail);
- I915_WRITE_TAIL(request->engine, request->tail);
+ I915_WRITE_TAIL(request->engine,
+ intel_ring_set_tail(request->ring, request->tail));
}
static void i9xx_emit_breadcrumb(struct drm_i915_gem_request *req, u32 *cs)
{
struct drm_i915_private *dev_priv = engine->i915;
+ GEM_BUG_ON(engine->id != RCS);
+
dev_priv->status_page_dmah =
drm_pci_alloc(&dev_priv->drm, PAGE_SIZE, PAGE_SIZE);
if (!dev_priv->status_page_dmah)
return 0;
}
-int intel_ring_pin(struct intel_ring *ring, unsigned int offset_bias)
+int intel_ring_pin(struct intel_ring *ring,
+ struct drm_i915_private *i915,
+ unsigned int offset_bias)
{
- unsigned int flags;
- enum i915_map_type map;
+ enum i915_map_type map = HAS_LLC(i915) ? I915_MAP_WB : I915_MAP_WC;
struct i915_vma *vma = ring->vma;
+ unsigned int flags;
void *addr;
int ret;
GEM_BUG_ON(ring->vaddr);
- map = HAS_LLC(ring->engine->i915) ? I915_MAP_WB : I915_MAP_WC;
flags = PIN_GLOBAL;
if (offset_bias)
return PTR_ERR(addr);
}
+void intel_ring_reset(struct intel_ring *ring, u32 tail)
+{
+ GEM_BUG_ON(!list_empty(&ring->request_list));
+ ring->tail = tail;
+ ring->head = tail;
+ ring->emit = tail;
+ intel_ring_update_space(ring);
+}
+
void intel_ring_unpin(struct intel_ring *ring)
{
GEM_BUG_ON(!ring->vma);
GEM_BUG_ON(!ring->vaddr);
+ /* Discard any unused bytes beyond that submitted to hw. */
+ intel_ring_reset(ring, ring->tail);
+
if (i915_vma_is_map_and_fenceable(ring->vma))
i915_vma_unpin_iomap(ring->vma);
else
obj = i915_gem_object_create_stolen(dev_priv, size);
if (!obj)
- obj = i915_gem_object_create(dev_priv, size);
+ obj = i915_gem_object_create_internal(dev_priv, size);
if (IS_ERR(obj))
return ERR_CAST(obj);
if (!ring)
return ERR_PTR(-ENOMEM);
- ring->engine = engine;
-
INIT_LIST_HEAD(&ring->request_list);
ring->size = size;
PIN_GLOBAL | PIN_HIGH);
}
-static int intel_ring_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx)
+static struct i915_vma *
+alloc_context_vma(struct intel_engine_cs *engine)
+{
+ struct drm_i915_private *i915 = engine->i915;
+ struct drm_i915_gem_object *obj;
+ struct i915_vma *vma;
+
+ obj = i915_gem_object_create(i915, engine->context_size);
+ if (IS_ERR(obj))
+ return ERR_CAST(obj);
+
+ /*
+ * Try to make the context utilize L3 as well as LLC.
+ *
+ * On VLV we don't have L3 controls in the PTEs so we
+ * shouldn't touch the cache level, especially as that
+ * would make the object snooped which might have a
+ * negative performance impact.
+ *
+ * Snooping is required on non-llc platforms in execlist
+ * mode, but since all GGTT accesses use PAT entry 0 we
+ * get snooping anyway regardless of cache_level.
+ *
+ * This is only applicable for Ivy Bridge devices since
+ * later platforms don't have L3 control bits in the PTE.
+ */
+ if (IS_IVYBRIDGE(i915)) {
+ /* Ignore any error, regard it as a simple optimisation */
+ i915_gem_object_set_cache_level(obj, I915_CACHE_L3_LLC);
+ }
+
+ vma = i915_vma_instance(obj, &i915->ggtt.base, NULL);
+ if (IS_ERR(vma))
+ i915_gem_object_put(obj);
+
+ return vma;
+}
+
+static struct intel_ring *
+intel_ring_context_pin(struct intel_engine_cs *engine,
+ struct i915_gem_context *ctx)
{
struct intel_context *ce = &ctx->engine[engine->id];
int ret;
lockdep_assert_held(&ctx->i915->drm.struct_mutex);
- if (ce->pin_count++)
- return 0;
+ if (likely(ce->pin_count++))
+ goto out;
GEM_BUG_ON(!ce->pin_count); /* no overflow please! */
+ if (!ce->state && engine->context_size) {
+ struct i915_vma *vma;
+
+ vma = alloc_context_vma(engine);
+ if (IS_ERR(vma)) {
+ ret = PTR_ERR(vma);
+ goto err;
+ }
+
+ ce->state = vma;
+ }
+
if (ce->state) {
ret = context_pin(ctx);
if (ret)
- goto error;
+ goto err;
ce->state->obj->mm.dirty = true;
}
ce->initialised = true;
i915_gem_context_get(ctx);
- return 0;
-error:
+out:
+ /* One ringbuffer to rule them all */
+ return engine->buffer;
+
+err:
ce->pin_count = 0;
- return ret;
+ return ERR_PTR(ret);
}
static void intel_ring_context_unpin(struct intel_engine_cs *engine,
static int intel_init_ring_buffer(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv = engine->i915;
struct intel_ring *ring;
- int ret;
-
- WARN_ON(engine->buffer);
+ int err;
intel_engine_setup_common(engine);
- ret = intel_engine_init_common(engine);
- if (ret)
- goto error;
+ err = intel_engine_init_common(engine);
+ if (err)
+ goto err;
+
+ if (HWS_NEEDS_PHYSICAL(engine->i915))
+ err = init_phys_status_page(engine);
+ else
+ err = init_status_page(engine);
+ if (err)
+ goto err;
ring = intel_engine_create_ring(engine, 32 * PAGE_SIZE);
if (IS_ERR(ring)) {
- ret = PTR_ERR(ring);
- goto error;
- }
-
- if (HWS_NEEDS_PHYSICAL(dev_priv)) {
- WARN_ON(engine->id != RCS);
- ret = init_phys_status_page(engine);
- if (ret)
- goto error;
- } else {
- ret = init_status_page(engine);
- if (ret)
- goto error;
+ err = PTR_ERR(ring);
+ goto err_hws;
}
/* Ring wraparound at offset 0 sometimes hangs. No idea why. */
- ret = intel_ring_pin(ring, I915_GTT_PAGE_SIZE);
- if (ret) {
- intel_ring_free(ring);
- goto error;
- }
+ err = intel_ring_pin(ring, engine->i915, I915_GTT_PAGE_SIZE);
+ if (err)
+ goto err_ring;
+
+ GEM_BUG_ON(engine->buffer);
engine->buffer = ring;
return 0;
-error:
- intel_engine_cleanup(engine);
- return ret;
+err_ring:
+ intel_ring_free(ring);
+err_hws:
+ if (HWS_NEEDS_PHYSICAL(engine->i915))
+ cleanup_phys_status_page(engine);
+ else
+ cleanup_status_page(engine);
+err:
+ intel_engine_cleanup_common(engine);
+ return err;
}
void intel_engine_cleanup(struct intel_engine_cs *engine)
{
- struct drm_i915_private *dev_priv;
-
- dev_priv = engine->i915;
+ struct drm_i915_private *dev_priv = engine->i915;
- if (engine->buffer) {
- WARN_ON(INTEL_GEN(dev_priv) > 2 &&
- (I915_READ_MODE(engine) & MODE_IDLE) == 0);
+ WARN_ON(INTEL_GEN(dev_priv) > 2 &&
+ (I915_READ_MODE(engine) & MODE_IDLE) == 0);
- intel_ring_unpin(engine->buffer);
- intel_ring_free(engine->buffer);
- engine->buffer = NULL;
- }
+ intel_ring_unpin(engine->buffer);
+ intel_ring_free(engine->buffer);
if (engine->cleanup)
engine->cleanup(engine);
- if (HWS_NEEDS_PHYSICAL(dev_priv)) {
- WARN_ON(engine->id != RCS);
+ if (HWS_NEEDS_PHYSICAL(dev_priv))
cleanup_phys_status_page(engine);
- } else {
+ else
cleanup_status_page(engine);
- }
intel_engine_cleanup_common(engine);
- engine->i915 = NULL;
dev_priv->engine[engine->id] = NULL;
kfree(engine);
}
struct intel_engine_cs *engine;
enum intel_engine_id id;
+ /* Restart from the beginning of the rings for convenience */
for_each_engine(engine, dev_priv, id)
- engine->buffer->head = engine->buffer->tail;
+ intel_ring_reset(engine->buffer, 0);
}
static int ring_request_alloc(struct drm_i915_gem_request *request)
*/
request->reserved_space += LEGACY_REQUEST_SIZE;
- GEM_BUG_ON(!request->engine->buffer);
- request->ring = request->engine->buffer;
-
cs = intel_ring_begin(request, 0);
if (IS_ERR(cs))
return PTR_ERR(cs);
return 0;
}
-static int wait_for_space(struct drm_i915_gem_request *req, int bytes)
+static noinline int wait_for_space(struct drm_i915_gem_request *req,
+ unsigned int bytes)
{
struct intel_ring *ring = req->ring;
struct drm_i915_gem_request *target;
lockdep_assert_held(&req->i915->drm.struct_mutex);
- intel_ring_update_space(ring);
- if (ring->space >= bytes)
+ if (intel_ring_update_space(ring) >= bytes)
return 0;
/*
GEM_BUG_ON(!req->reserved_space);
list_for_each_entry(target, &ring->request_list, ring_link) {
- unsigned space;
-
/* Would completion of this request free enough space? */
- space = __intel_ring_space(target->postfix, ring->tail,
- ring->size);
- if (space >= bytes)
+ if (bytes <= __intel_ring_space(target->postfix,
+ ring->emit, ring->size))
break;
}
return 0;
}
-u32 *intel_ring_begin(struct drm_i915_gem_request *req, int num_dwords)
+u32 *intel_ring_begin(struct drm_i915_gem_request *req,
+ unsigned int num_dwords)
{
struct intel_ring *ring = req->ring;
- int remain_actual = ring->size - ring->tail;
- int remain_usable = ring->effective_size - ring->tail;
- int bytes = num_dwords * sizeof(u32);
- int total_bytes, wait_bytes;
- bool need_wrap = false;
+ const unsigned int remain_usable = ring->effective_size - ring->emit;
+ const unsigned int bytes = num_dwords * sizeof(u32);
+ unsigned int need_wrap = 0;
+ unsigned int total_bytes;
u32 *cs;
total_bytes = bytes + req->reserved_space;
+ GEM_BUG_ON(total_bytes > ring->effective_size);
- if (unlikely(bytes > remain_usable)) {
- /*
- * Not enough space for the basic request. So need to flush
- * out the remainder and then wait for base + reserved.
- */
- wait_bytes = remain_actual + total_bytes;
- need_wrap = true;
- } else if (unlikely(total_bytes > remain_usable)) {
- /*
- * The base request will fit but the reserved space
- * falls off the end. So we don't need an immediate wrap
- * and only need to effectively wait for the reserved
- * size space from the start of ringbuffer.
- */
- wait_bytes = remain_actual + req->reserved_space;
- } else {
- /* No wrapping required, just waiting. */
- wait_bytes = total_bytes;
+ if (unlikely(total_bytes > remain_usable)) {
+ const int remain_actual = ring->size - ring->emit;
+
+ if (bytes > remain_usable) {
+ /*
+ * Not enough space for the basic request. So need to
+ * flush out the remainder and then wait for
+ * base + reserved.
+ */
+ total_bytes += remain_actual;
+ need_wrap = remain_actual | 1;
+ } else {
+ /*
+ * The base request will fit but the reserved space
+ * falls off the end. So we don't need an immediate
+ * wrap and only need to effectively wait for the
+ * reserved size from the start of ringbuffer.
+ */
+ total_bytes = req->reserved_space + remain_actual;
+ }
}
- if (wait_bytes > ring->space) {
- int ret = wait_for_space(req, wait_bytes);
+ if (unlikely(total_bytes > ring->space)) {
+ int ret = wait_for_space(req, total_bytes);
if (unlikely(ret))
return ERR_PTR(ret);
}
if (unlikely(need_wrap)) {
- GEM_BUG_ON(remain_actual > ring->space);
- GEM_BUG_ON(ring->tail + remain_actual > ring->size);
+ need_wrap &= ~1;
+ GEM_BUG_ON(need_wrap > ring->space);
+ GEM_BUG_ON(ring->emit + need_wrap > ring->size);
/* Fill the tail with MI_NOOP */
- memset(ring->vaddr + ring->tail, 0, remain_actual);
- ring->tail = 0;
- ring->space -= remain_actual;
+ memset(ring->vaddr + ring->emit, 0, need_wrap);
+ ring->emit = 0;
+ ring->space -= need_wrap;
}
- GEM_BUG_ON(ring->tail > ring->size - bytes);
- cs = ring->vaddr + ring->tail;
- ring->tail += bytes;
+ GEM_BUG_ON(ring->emit > ring->size - bytes);
+ GEM_BUG_ON(ring->space < bytes);
+ cs = ring->vaddr + ring->emit;
+ GEM_DEBUG_EXEC(memset(cs, POISON_INUSE, bytes));
+ ring->emit += bytes;
ring->space -= bytes;
- GEM_BUG_ON(ring->space < 0);
return cs;
}
int intel_ring_cacheline_align(struct drm_i915_gem_request *req)
{
int num_dwords =
- (req->ring->tail & (CACHELINE_BYTES - 1)) / sizeof(uint32_t);
+ (req->ring->emit & (CACHELINE_BYTES - 1)) / sizeof(uint32_t);
u32 *cs;
if (num_dwords == 0)
I915_WRITE64_FW(GEN6_BSD_RNCID, 0x0);
/* Wait for the ring not to be idle, i.e. for it to wake up. */
- if (intel_wait_for_register_fw(dev_priv,
- GEN6_BSD_SLEEP_PSMI_CONTROL,
- GEN6_BSD_SLEEP_INDICATOR,
- 0,
- 50))
+ if (__intel_wait_for_register_fw(dev_priv,
+ GEN6_BSD_SLEEP_PSMI_CONTROL,
+ GEN6_BSD_SLEEP_INDICATOR,
+ 0,
+ 1000, 0, NULL))
DRM_ERROR("timed out waiting for the BSD ring to wake up\n");
/* Now that the ring is fully powered up, update the tail */
return intel_init_ring_buffer(engine);
}
-/**
- * Initialize the second BSD ring (eg. Broadwell GT3, Skylake GT3)
- */
-int intel_init_bsd2_ring_buffer(struct intel_engine_cs *engine)
-{
- struct drm_i915_private *dev_priv = engine->i915;
-
- intel_ring_default_vfuncs(dev_priv, engine);
-
- engine->emit_flush = gen6_bsd_ring_flush;
-
- return intel_init_ring_buffer(engine);
-}
-
int intel_init_blt_ring_buffer(struct intel_engine_cs *engine)
{
struct drm_i915_private *dev_priv = engine->i915;
#define CACHELINE_BYTES 64
#define CACHELINE_DWORDS (CACHELINE_BYTES / sizeof(uint32_t))
-/*
- * Gen2 BSpec "1. Programming Environment" / 1.4.4.6 "Ring Buffer Use"
- * Gen3 BSpec "vol1c Memory Interface Functions" / 2.3.4.5 "Ring Buffer Use"
- * Gen4+ BSpec "vol1c Memory Interface and Command Stream" / 5.3.4.5 "Ring Buffer Use"
- *
- * "If the Ring Buffer Head Pointer and the Tail Pointer are on the same
- * cacheline, the Head Pointer must not be greater than the Tail
- * Pointer."
- */
-#define I915_RING_FREE_SPACE 64
-
struct intel_hw_status_page {
struct i915_vma *vma;
u32 *page_addr;
struct i915_vma *vma;
void *vaddr;
- struct intel_engine_cs *engine;
-
struct list_head request_list;
u32 head;
u32 tail;
+ u32 emit;
- int space;
- int size;
- int effective_size;
+ u32 space;
+ u32 size;
+ u32 effective_size;
};
struct i915_gem_context;
VECS
};
+struct i915_priolist {
+ struct rb_node node;
+ struct list_head requests;
+ int priority;
+};
+
+#define INTEL_ENGINE_CS_MAX_NAME 8
+
struct intel_engine_cs {
struct drm_i915_private *i915;
- const char *name;
+ char name[INTEL_ENGINE_CS_MAX_NAME];
enum intel_engine_id id;
- unsigned int exec_id;
+ unsigned int uabi_id;
unsigned int hw_id;
unsigned int guc_id;
- u32 mmio_base;
+
+ u8 class;
+ u8 instance;
+ u32 context_size;
+ u32 mmio_base;
unsigned int irq_shift;
+
struct intel_ring *buffer;
struct intel_timeline *timeline;
void (*set_default_submission)(struct intel_engine_cs *engine);
- int (*context_pin)(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx);
+ struct intel_ring *(*context_pin)(struct intel_engine_cs *engine,
+ struct i915_gem_context *ctx);
void (*context_unpin)(struct intel_engine_cs *engine,
struct i915_gem_context *ctx);
int (*request_alloc)(struct drm_i915_gem_request *req);
/* Execlists */
struct tasklet_struct irq_tasklet;
+ struct i915_priolist default_priolist;
+ bool no_priolist;
struct execlist_port {
- struct drm_i915_gem_request *request;
- unsigned int count;
+ struct drm_i915_gem_request *request_count;
+#define EXECLIST_COUNT_BITS 2
+#define port_request(p) ptr_mask_bits((p)->request_count, EXECLIST_COUNT_BITS)
+#define port_count(p) ptr_unmask_bits((p)->request_count, EXECLIST_COUNT_BITS)
+#define port_pack(rq, count) ptr_pack_bits(rq, count, EXECLIST_COUNT_BITS)
+#define port_unpack(p, count) ptr_unpack_bits((p)->request_count, count, EXECLIST_COUNT_BITS)
+#define port_set(p, packed) ((p)->request_count = (packed))
+#define port_isset(p) ((p)->request_count)
+#define port_index(p, e) ((p) - (e)->execlist_port)
GEM_DEBUG_DECL(u32 context_id);
} execlist_port[2];
struct rb_root execlist_queue;
struct intel_ring *
intel_engine_create_ring(struct intel_engine_cs *engine, int size);
-int intel_ring_pin(struct intel_ring *ring, unsigned int offset_bias);
+int intel_ring_pin(struct intel_ring *ring,
+ struct drm_i915_private *i915,
+ unsigned int offset_bias);
+void intel_ring_reset(struct intel_ring *ring, u32 tail);
+unsigned int intel_ring_update_space(struct intel_ring *ring);
void intel_ring_unpin(struct intel_ring *ring);
void intel_ring_free(struct intel_ring *ring);
int __must_check intel_ring_cacheline_align(struct drm_i915_gem_request *req);
-u32 __must_check *intel_ring_begin(struct drm_i915_gem_request *req, int n);
+u32 __must_check *intel_ring_begin(struct drm_i915_gem_request *req,
+ unsigned int n);
static inline void
intel_ring_advance(struct drm_i915_gem_request *req, u32 *cs)
* reserved for the command packet (i.e. the value passed to
* intel_ring_begin()).
*/
- GEM_BUG_ON((req->ring->vaddr + req->ring->tail) != cs);
+ GEM_BUG_ON((req->ring->vaddr + req->ring->emit) != cs);
}
static inline u32
*/
GEM_BUG_ON(!IS_ALIGNED(tail, 8));
GEM_BUG_ON(tail >= ring->size);
+
+ /*
+ * "Ring Buffer Use"
+ * Gen2 BSpec "1. Programming Environment" / 1.4.4.6
+ * Gen3 BSpec "1c Memory Interface Functions" / 2.3.4.5
+ * Gen4+ BSpec "1c Memory Interface and Command Stream" / 5.3.4.5
+ * "If the Ring Buffer Head Pointer and the Tail Pointer are on the
+ * same cacheline, the Head Pointer must not be greater than the Tail
+ * Pointer."
+ *
+ * We use ring->head as the last known location of the actual RING_HEAD,
+ * it may have advanced but in the worst case it is equally the same
+ * as ring->head and so we should never program RING_TAIL to advance
+ * into the same cacheline as ring->head.
+ */
+#define cacheline(a) round_down(a, CACHELINE_BYTES)
+ GEM_BUG_ON(cacheline(tail) == cacheline(ring->head) &&
+ tail < ring->head);
+#undef cacheline
}
-void intel_ring_update_space(struct intel_ring *ring);
+static inline unsigned int
+intel_ring_set_tail(struct intel_ring *ring, unsigned int tail)
+{
+ /* Whilst writes to the tail are strictly order, there is no
+ * serialisation between readers and the writers. The tail may be
+ * read by i915_gem_request_retire() just as it is being updated
+ * by execlists, as although the breadcrumb is complete, the context
+ * switch hasn't been seen.
+ */
+ assert_ring_tail_valid(ring, tail);
+ ring->tail = tail;
+ return tail;
+}
void intel_engine_init_global_seqno(struct intel_engine_cs *engine, u32 seqno);
int intel_init_render_ring_buffer(struct intel_engine_cs *engine);
int intel_init_bsd_ring_buffer(struct intel_engine_cs *engine);
-int intel_init_bsd2_ring_buffer(struct intel_engine_cs *engine);
int intel_init_blt_ring_buffer(struct intel_engine_cs *engine);
int intel_init_vebox_ring_buffer(struct intel_engine_cs *engine);
struct intel_wait *wait);
void intel_engine_remove_wait(struct intel_engine_cs *engine,
struct intel_wait *wait);
-void intel_engine_enable_signaling(struct drm_i915_gem_request *request);
+void intel_engine_enable_signaling(struct drm_i915_gem_request *request,
+ bool wakeup);
void intel_engine_cancel_signaling(struct drm_i915_gem_request *request);
static inline bool intel_engine_has_waiter(const struct intel_engine_cs *engine)
bool intel_engine_is_idle(struct intel_engine_cs *engine);
bool intel_engines_are_idle(struct drm_i915_private *dev_priv);
+void intel_engines_mark_idle(struct drm_i915_private *i915);
void intel_engines_reset_default_submission(struct drm_i915_private *i915);
#endif /* _INTEL_RINGBUFFER_H_ */
BUILD_BUG_ON(sizeof(enhancements) != 2);
- enhancements.response = 0;
- intel_sdvo_get_value(intel_sdvo,
- SDVO_CMD_GET_SUPPORTED_ENHANCEMENTS,
- &enhancements, sizeof(enhancements));
- if (enhancements.response == 0) {
+ if (!intel_sdvo_get_value(intel_sdvo,
+ SDVO_CMD_GET_SUPPORTED_ENHANCEMENTS,
+ &enhancements, sizeof(enhancements)) ||
+ enhancements.response == 0) {
DRM_DEBUG_KMS("No enhancement is supported\n");
return true;
}
}
static void
-skl_update_plane(struct drm_plane *drm_plane,
+skl_update_plane(struct intel_plane *plane,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = drm_plane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(drm_plane);
- struct drm_framebuffer *fb = plane_state->base.fb;
- enum plane_id plane_id = intel_plane->id;
- enum pipe pipe = intel_plane->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum plane_id plane_id = plane->id;
+ enum pipe pipe = plane->pipe;
u32 plane_ctl = plane_state->ctl;
const struct drm_intel_sprite_colorkey *key = &plane_state->ckey;
u32 surf_addr = plane_state->main.offset;
}
static void
-skl_disable_plane(struct drm_plane *dplane, struct drm_crtc *crtc)
+skl_disable_plane(struct intel_plane *plane, struct intel_crtc *crtc)
{
- struct drm_device *dev = dplane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(dplane);
- enum plane_id plane_id = intel_plane->id;
- enum pipe pipe = intel_plane->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ enum plane_id plane_id = plane->id;
+ enum pipe pipe = plane->pipe;
unsigned long irqflags;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
}
static void
-chv_update_csc(struct intel_plane *intel_plane, uint32_t format)
+chv_update_csc(struct intel_plane *plane, uint32_t format)
{
- struct drm_i915_private *dev_priv = to_i915(intel_plane->base.dev);
- enum plane_id plane_id = intel_plane->id;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ enum plane_id plane_id = plane->id;
/* Seems RGB data bypasses the CSC always */
if (!format_is_yuv(format))
}
static void
-vlv_update_plane(struct drm_plane *dplane,
+vlv_update_plane(struct intel_plane *plane,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = dplane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(dplane);
- struct drm_framebuffer *fb = plane_state->base.fb;
- enum pipe pipe = intel_plane->pipe;
- enum plane_id plane_id = intel_plane->id;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum pipe pipe = plane->pipe;
+ enum plane_id plane_id = plane->id;
u32 sprctl = plane_state->ctl;
u32 sprsurf_offset = plane_state->main.offset;
u32 linear_offset;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
if (IS_CHERRYVIEW(dev_priv) && pipe == PIPE_B)
- chv_update_csc(intel_plane, fb->format->format);
+ chv_update_csc(plane, fb->format->format);
if (key->flags) {
I915_WRITE_FW(SPKEYMINVAL(pipe, plane_id), key->min_value);
}
static void
-vlv_disable_plane(struct drm_plane *dplane, struct drm_crtc *crtc)
+vlv_disable_plane(struct intel_plane *plane, struct intel_crtc *crtc)
{
- struct drm_device *dev = dplane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(dplane);
- enum pipe pipe = intel_plane->pipe;
- enum plane_id plane_id = intel_plane->id;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ enum pipe pipe = plane->pipe;
+ enum plane_id plane_id = plane->id;
unsigned long irqflags;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
}
static void
-ivb_update_plane(struct drm_plane *plane,
+ivb_update_plane(struct intel_plane *plane,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = plane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(plane);
- struct drm_framebuffer *fb = plane_state->base.fb;
- enum pipe pipe = intel_plane->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum pipe pipe = plane->pipe;
u32 sprctl = plane_state->ctl, sprscale = 0;
u32 sprsurf_offset = plane_state->main.offset;
u32 linear_offset;
I915_WRITE_FW(SPRLINOFF(pipe), linear_offset);
I915_WRITE_FW(SPRSIZE(pipe), (crtc_h << 16) | crtc_w);
- if (intel_plane->can_scale)
+ if (plane->can_scale)
I915_WRITE_FW(SPRSCALE(pipe), sprscale);
I915_WRITE_FW(SPRCTL(pipe), sprctl);
I915_WRITE_FW(SPRSURF(pipe),
}
static void
-ivb_disable_plane(struct drm_plane *plane, struct drm_crtc *crtc)
+ivb_disable_plane(struct intel_plane *plane, struct intel_crtc *crtc)
{
- struct drm_device *dev = plane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(plane);
- int pipe = intel_plane->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ enum pipe pipe = plane->pipe;
unsigned long irqflags;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
I915_WRITE_FW(SPRCTL(pipe), 0);
/* Can't leave the scaler enabled... */
- if (intel_plane->can_scale)
+ if (plane->can_scale)
I915_WRITE_FW(SPRSCALE(pipe), 0);
I915_WRITE_FW(SPRSURF(pipe), 0);
spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
}
-static u32 ilk_sprite_ctl(const struct intel_crtc_state *crtc_state,
+static u32 g4x_sprite_ctl(const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
struct drm_i915_private *dev_priv =
}
static void
-ilk_update_plane(struct drm_plane *plane,
+g4x_update_plane(struct intel_plane *plane,
const struct intel_crtc_state *crtc_state,
const struct intel_plane_state *plane_state)
{
- struct drm_device *dev = plane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(plane);
- struct drm_framebuffer *fb = plane_state->base.fb;
- int pipe = intel_plane->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ const struct drm_framebuffer *fb = plane_state->base.fb;
+ enum pipe pipe = plane->pipe;
u32 dvscntr = plane_state->ctl, dvsscale = 0;
u32 dvssurf_offset = plane_state->main.offset;
u32 linear_offset;
}
static void
-ilk_disable_plane(struct drm_plane *plane, struct drm_crtc *crtc)
+g4x_disable_plane(struct intel_plane *plane, struct intel_crtc *crtc)
{
- struct drm_device *dev = plane->dev;
- struct drm_i915_private *dev_priv = to_i915(dev);
- struct intel_plane *intel_plane = to_intel_plane(plane);
- int pipe = intel_plane->pipe;
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ enum pipe pipe = plane->pipe;
unsigned long irqflags;
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
}
static int
-intel_check_sprite_plane(struct drm_plane *plane,
+intel_check_sprite_plane(struct intel_plane *plane,
struct intel_crtc_state *crtc_state,
struct intel_plane_state *state)
{
- struct drm_i915_private *dev_priv = to_i915(plane->dev);
- struct drm_crtc *crtc = state->base.crtc;
- struct intel_crtc *intel_crtc = to_intel_crtc(crtc);
- struct intel_plane *intel_plane = to_intel_plane(plane);
+ struct drm_i915_private *dev_priv = to_i915(plane->base.dev);
+ struct intel_crtc *crtc = to_intel_crtc(crtc_state->base.crtc);
struct drm_framebuffer *fb = state->base.fb;
int crtc_x, crtc_y;
unsigned int crtc_w, crtc_h;
}
/* Don't modify another pipe's plane */
- if (intel_plane->pipe != intel_crtc->pipe) {
+ if (plane->pipe != crtc->pipe) {
DRM_DEBUG_KMS("Wrong plane <-> crtc mapping\n");
return -EINVAL;
}
if (state->ckey.flags == I915_SET_COLORKEY_NONE) {
can_scale = 1;
min_scale = 1;
- max_scale = skl_max_scale(intel_crtc, crtc_state);
+ max_scale = skl_max_scale(crtc, crtc_state);
} else {
can_scale = 0;
min_scale = DRM_PLANE_HELPER_NO_SCALING;
max_scale = DRM_PLANE_HELPER_NO_SCALING;
}
} else {
- can_scale = intel_plane->can_scale;
- max_scale = intel_plane->max_downscale << 16;
- min_scale = intel_plane->can_scale ? 1 : (1 << 16);
+ can_scale = plane->can_scale;
+ max_scale = plane->max_downscale << 16;
+ min_scale = plane->can_scale ? 1 : (1 << 16);
}
/*
if (ret)
return ret;
- state->ctl = ilk_sprite_ctl(crtc_state, state);
+ state->ctl = g4x_sprite_ctl(crtc_state, state);
}
return 0;
return ret;
}
-static const uint32_t ilk_plane_formats[] = {
+static const uint32_t g4x_plane_formats[] = {
DRM_FORMAT_XRGB8888,
DRM_FORMAT_YUYV,
DRM_FORMAT_YVYU,
intel_plane->can_scale = true;
intel_plane->max_downscale = 16;
- intel_plane->update_plane = ilk_update_plane;
- intel_plane->disable_plane = ilk_disable_plane;
+ intel_plane->update_plane = g4x_update_plane;
+ intel_plane->disable_plane = g4x_disable_plane;
if (IS_GEN6(dev_priv)) {
plane_formats = snb_plane_formats;
num_plane_formats = ARRAY_SIZE(snb_plane_formats);
} else {
- plane_formats = ilk_plane_formats;
- num_plane_formats = ARRAY_SIZE(ilk_plane_formats);
+ plane_formats = g4x_plane_formats;
+ num_plane_formats = ARRAY_SIZE(g4x_plane_formats);
}
}
struct intel_encoder base;
int type;
- const char *tv_format;
- int margin[4];
- u32 save_TV_H_CTL_1;
- u32 save_TV_H_CTL_2;
- u32 save_TV_H_CTL_3;
- u32 save_TV_V_CTL_1;
- u32 save_TV_V_CTL_2;
- u32 save_TV_V_CTL_3;
- u32 save_TV_V_CTL_4;
- u32 save_TV_V_CTL_5;
- u32 save_TV_V_CTL_6;
- u32 save_TV_V_CTL_7;
- u32 save_TV_SC_CTL_1, save_TV_SC_CTL_2, save_TV_SC_CTL_3;
-
- u32 save_TV_CSC_Y;
- u32 save_TV_CSC_Y2;
- u32 save_TV_CSC_U;
- u32 save_TV_CSC_U2;
- u32 save_TV_CSC_V;
- u32 save_TV_CSC_V2;
- u32 save_TV_CLR_KNOBS;
- u32 save_TV_CLR_LEVEL;
- u32 save_TV_WIN_POS;
- u32 save_TV_WIN_SIZE;
- u32 save_TV_FILTER_CTL_1;
- u32 save_TV_FILTER_CTL_2;
- u32 save_TV_FILTER_CTL_3;
-
- u32 save_TV_H_LUMA[60];
- u32 save_TV_H_CHROMA[60];
- u32 save_TV_V_LUMA[43];
- u32 save_TV_V_CHROMA[43];
-
- u32 save_TV_DAC;
- u32 save_TV_CTL;
};
struct video_levels {
I915_WRITE(TV_CTL, I915_READ(TV_CTL) & ~TV_ENC_ENABLE);
}
-static const struct tv_mode *
-intel_tv_mode_lookup(const char *tv_format)
+static const struct tv_mode *intel_tv_mode_find(struct drm_connector_state *conn_state)
{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(tv_modes); i++) {
- const struct tv_mode *tv_mode = &tv_modes[i];
+ int format = conn_state->tv.mode;
- if (!strcmp(tv_format, tv_mode->name))
- return tv_mode;
- }
- return NULL;
-}
-
-static const struct tv_mode *
-intel_tv_mode_find(struct intel_tv *intel_tv)
-{
- return intel_tv_mode_lookup(intel_tv->tv_format);
+ return &tv_modes[format];
}
static enum drm_mode_status
intel_tv_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
- struct intel_tv *intel_tv = intel_attached_tv(connector);
- const struct tv_mode *tv_mode = intel_tv_mode_find(intel_tv);
+ const struct tv_mode *tv_mode = intel_tv_mode_find(connector->state);
int max_dotclk = to_i915(connector->dev)->max_dotclk_freq;
if (mode->clock > max_dotclk)
struct intel_crtc_state *pipe_config,
struct drm_connector_state *conn_state)
{
- struct intel_tv *intel_tv = enc_to_tv(encoder);
- const struct tv_mode *tv_mode = intel_tv_mode_find(intel_tv);
+ const struct tv_mode *tv_mode = intel_tv_mode_find(conn_state);
if (!tv_mode)
return false;
struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
struct intel_crtc *intel_crtc = to_intel_crtc(encoder->base.crtc);
struct intel_tv *intel_tv = enc_to_tv(encoder);
- const struct tv_mode *tv_mode = intel_tv_mode_find(intel_tv);
+ const struct tv_mode *tv_mode = intel_tv_mode_find(conn_state);
u32 tv_ctl;
u32 scctl1, scctl2, scctl3;
int i, j;
else
ysize = 2*tv_mode->nbr_end + 1;
- xpos += intel_tv->margin[TV_MARGIN_LEFT];
- ypos += intel_tv->margin[TV_MARGIN_TOP];
- xsize -= (intel_tv->margin[TV_MARGIN_LEFT] +
- intel_tv->margin[TV_MARGIN_RIGHT]);
- ysize -= (intel_tv->margin[TV_MARGIN_TOP] +
- intel_tv->margin[TV_MARGIN_BOTTOM]);
+ xpos += conn_state->tv.margins.left;
+ ypos += conn_state->tv.margins.top;
+ xsize -= (conn_state->tv.margins.left +
+ conn_state->tv.margins.right);
+ ysize -= (conn_state->tv.margins.top +
+ conn_state->tv.margins.bottom);
I915_WRITE(TV_WIN_POS, (xpos<<16)|ypos);
I915_WRITE(TV_WIN_SIZE, (xsize<<16)|ysize);
static void intel_tv_find_better_format(struct drm_connector *connector)
{
struct intel_tv *intel_tv = intel_attached_tv(connector);
- const struct tv_mode *tv_mode = intel_tv_mode_find(intel_tv);
+ const struct tv_mode *tv_mode = intel_tv_mode_find(connector->state);
int i;
if ((intel_tv->type == DRM_MODE_CONNECTOR_Component) ==
break;
}
- intel_tv->tv_format = tv_mode->name;
- drm_object_property_set_value(&connector->base,
- connector->dev->mode_config.tv_mode_property, i);
+ connector->state->tv.mode = i;
}
/**
connector_status_connected;
} else
status = connector_status_unknown;
- } else
- return connector->status;
- if (status != connector_status_connected)
- return status;
-
- intel_tv->type = type;
- intel_tv_find_better_format(connector);
+ if (status == connector_status_connected) {
+ intel_tv->type = type;
+ intel_tv_find_better_format(connector);
+ }
- return connector_status_connected;
+ return status;
+ } else
+ return connector->status;
}
static const struct input_res {
* Chose preferred mode according to line number of TV format
*/
static void
-intel_tv_chose_preferred_modes(struct drm_connector *connector,
+intel_tv_choose_preferred_modes(const struct tv_mode *tv_mode,
struct drm_display_mode *mode_ptr)
{
- struct intel_tv *intel_tv = intel_attached_tv(connector);
- const struct tv_mode *tv_mode = intel_tv_mode_find(intel_tv);
-
if (tv_mode->nbr_end < 480 && mode_ptr->vdisplay == 480)
mode_ptr->type |= DRM_MODE_TYPE_PREFERRED;
else if (tv_mode->nbr_end > 480) {
intel_tv_get_modes(struct drm_connector *connector)
{
struct drm_display_mode *mode_ptr;
- struct intel_tv *intel_tv = intel_attached_tv(connector);
- const struct tv_mode *tv_mode = intel_tv_mode_find(intel_tv);
+ const struct tv_mode *tv_mode = intel_tv_mode_find(connector->state);
int j, count = 0;
u64 tmp;
mode_ptr->clock = (int) tmp;
mode_ptr->type = DRM_MODE_TYPE_DRIVER;
- intel_tv_chose_preferred_modes(connector, mode_ptr);
+ intel_tv_choose_preferred_modes(tv_mode, mode_ptr);
drm_mode_probed_add(connector, mode_ptr);
count++;
}
kfree(connector);
}
-
-static int
-intel_tv_set_property(struct drm_connector *connector, struct drm_property *property,
- uint64_t val)
-{
- struct drm_device *dev = connector->dev;
- struct intel_tv *intel_tv = intel_attached_tv(connector);
- struct drm_crtc *crtc = intel_tv->base.base.crtc;
- int ret = 0;
- bool changed = false;
-
- ret = drm_object_property_set_value(&connector->base, property, val);
- if (ret < 0)
- goto out;
-
- if (property == dev->mode_config.tv_left_margin_property &&
- intel_tv->margin[TV_MARGIN_LEFT] != val) {
- intel_tv->margin[TV_MARGIN_LEFT] = val;
- changed = true;
- } else if (property == dev->mode_config.tv_right_margin_property &&
- intel_tv->margin[TV_MARGIN_RIGHT] != val) {
- intel_tv->margin[TV_MARGIN_RIGHT] = val;
- changed = true;
- } else if (property == dev->mode_config.tv_top_margin_property &&
- intel_tv->margin[TV_MARGIN_TOP] != val) {
- intel_tv->margin[TV_MARGIN_TOP] = val;
- changed = true;
- } else if (property == dev->mode_config.tv_bottom_margin_property &&
- intel_tv->margin[TV_MARGIN_BOTTOM] != val) {
- intel_tv->margin[TV_MARGIN_BOTTOM] = val;
- changed = true;
- } else if (property == dev->mode_config.tv_mode_property) {
- if (val >= ARRAY_SIZE(tv_modes)) {
- ret = -EINVAL;
- goto out;
- }
- if (!strcmp(intel_tv->tv_format, tv_modes[val].name))
- goto out;
-
- intel_tv->tv_format = tv_modes[val].name;
- changed = true;
- } else {
- ret = -EINVAL;
- goto out;
- }
-
- if (changed && crtc)
- intel_crtc_restore_mode(crtc);
-out:
- return ret;
-}
-
static const struct drm_connector_funcs intel_tv_connector_funcs = {
.dpms = drm_atomic_helper_connector_dpms,
.late_register = intel_connector_register,
.early_unregister = intel_connector_unregister,
.destroy = intel_tv_destroy,
- .set_property = intel_tv_set_property,
- .atomic_get_property = intel_connector_atomic_get_property,
+ .set_property = drm_atomic_helper_connector_set_property,
.fill_modes = drm_helper_probe_single_connector_modes,
.atomic_destroy_state = drm_atomic_helper_connector_destroy_state,
.atomic_duplicate_state = drm_atomic_helper_connector_duplicate_state,
};
+static int intel_tv_atomic_check(struct drm_connector *connector,
+ struct drm_connector_state *new_state)
+{
+ struct drm_crtc_state *new_crtc_state;
+ struct drm_connector_state *old_state;
+
+ if (!new_state->crtc)
+ return 0;
+
+ old_state = drm_atomic_get_old_connector_state(new_state->state, connector);
+ new_crtc_state = drm_atomic_get_new_crtc_state(new_state->state, new_state->crtc);
+
+ if (old_state->tv.mode != new_state->tv.mode ||
+ old_state->tv.margins.left != new_state->tv.margins.left ||
+ old_state->tv.margins.right != new_state->tv.margins.right ||
+ old_state->tv.margins.top != new_state->tv.margins.top ||
+ old_state->tv.margins.bottom != new_state->tv.margins.bottom) {
+ /* Force a modeset. */
+
+ new_crtc_state->connectors_changed = true;
+ }
+
+ return 0;
+}
+
static const struct drm_connector_helper_funcs intel_tv_connector_helper_funcs = {
.detect_ctx = intel_tv_detect,
.mode_valid = intel_tv_mode_valid,
.get_modes = intel_tv_get_modes,
+ .atomic_check = intel_tv_atomic_check,
};
static const struct drm_encoder_funcs intel_tv_enc_funcs = {
u32 tv_dac_on, tv_dac_off, save_tv_dac;
const char *tv_format_names[ARRAY_SIZE(tv_modes)];
int i, initial_mode = 0;
+ struct drm_connector_state *state;
if ((I915_READ(TV_CTL) & TV_FUSE_STATE_MASK) == TV_FUSE_STATE_DISABLED)
return;
intel_encoder = &intel_tv->base;
connector = &intel_connector->base;
+ state = connector->state;
/* The documentation, for the older chipsets at least, recommend
* using a polling method rather than hotplug detection for TVs.
intel_tv->type = DRM_MODE_CONNECTOR_Unknown;
/* BIOS margin values */
- intel_tv->margin[TV_MARGIN_LEFT] = 54;
- intel_tv->margin[TV_MARGIN_TOP] = 36;
- intel_tv->margin[TV_MARGIN_RIGHT] = 46;
- intel_tv->margin[TV_MARGIN_BOTTOM] = 37;
+ state->tv.margins.left = 54;
+ state->tv.margins.top = 36;
+ state->tv.margins.right = 46;
+ state->tv.margins.bottom = 37;
- intel_tv->tv_format = tv_modes[initial_mode].name;
+ state->tv.mode = initial_mode;
drm_connector_helper_add(connector, &intel_tv_connector_helper_funcs);
connector->interlace_allowed = false;
tv_format_names);
drm_object_attach_property(&connector->base, dev->mode_config.tv_mode_property,
- initial_mode);
+ state->tv.mode);
drm_object_attach_property(&connector->base,
dev->mode_config.tv_left_margin_property,
- intel_tv->margin[TV_MARGIN_LEFT]);
+ state->tv.margins.left);
drm_object_attach_property(&connector->base,
dev->mode_config.tv_top_margin_property,
- intel_tv->margin[TV_MARGIN_TOP]);
+ state->tv.margins.top);
drm_object_attach_property(&connector->base,
dev->mode_config.tv_right_margin_property,
- intel_tv->margin[TV_MARGIN_RIGHT]);
+ state->tv.margins.right);
drm_object_attach_property(&connector->base,
dev->mode_config.tv_bottom_margin_property,
- intel_tv->margin[TV_MARGIN_BOTTOM]);
+ state->tv.margins.bottom);
}
i915.enable_guc_submission = HAS_GUC_SCHED(dev_priv);
}
+static void guc_write_irq_trigger(struct intel_guc *guc)
+{
+ struct drm_i915_private *dev_priv = guc_to_i915(guc);
+
+ I915_WRITE(GUC_SEND_INTERRUPT, GUC_SEND_TRIGGER);
+}
+
void intel_uc_init_early(struct drm_i915_private *dev_priv)
{
struct intel_guc *guc = &dev_priv->guc;
+ intel_guc_ct_init_early(&guc->ct);
+
mutex_init(&guc->send_mutex);
- guc->send = intel_guc_send_mmio;
+ guc->send = intel_guc_send_nop;
+ guc->notify = guc_write_irq_trigger;
}
static void fetch_uc_fw(struct drm_i915_private *dev_priv,
__intel_uc_fw_fini(&dev_priv->huc.fw);
}
+static inline i915_reg_t guc_send_reg(struct intel_guc *guc, u32 i)
+{
+ GEM_BUG_ON(!guc->send_regs.base);
+ GEM_BUG_ON(!guc->send_regs.count);
+ GEM_BUG_ON(i >= guc->send_regs.count);
+
+ return _MMIO(guc->send_regs.base + 4 * i);
+}
+
+static void guc_init_send_regs(struct intel_guc *guc)
+{
+ struct drm_i915_private *dev_priv = guc_to_i915(guc);
+ enum forcewake_domains fw_domains = 0;
+ unsigned int i;
+
+ guc->send_regs.base = i915_mmio_reg_offset(SOFT_SCRATCH(0));
+ guc->send_regs.count = SOFT_SCRATCH_COUNT - 1;
+
+ for (i = 0; i < guc->send_regs.count; i++) {
+ fw_domains |= intel_uncore_forcewake_for_reg(dev_priv,
+ guc_send_reg(guc, i),
+ FW_REG_READ | FW_REG_WRITE);
+ }
+ guc->send_regs.fw_domains = fw_domains;
+}
+
+static void guc_capture_load_err_log(struct intel_guc *guc)
+{
+ if (!guc->log.vma || i915.guc_log_level < 0)
+ return;
+
+ if (!guc->load_err_log)
+ guc->load_err_log = i915_gem_object_get(guc->log.vma->obj);
+
+ return;
+}
+
+static void guc_free_load_err_log(struct intel_guc *guc)
+{
+ if (guc->load_err_log)
+ i915_gem_object_put(guc->load_err_log);
+}
+
+static int guc_enable_communication(struct intel_guc *guc)
+{
+ struct drm_i915_private *dev_priv = guc_to_i915(guc);
+
+ guc_init_send_regs(guc);
+
+ if (HAS_GUC_CT(dev_priv))
+ return intel_guc_enable_ct(guc);
+
+ guc->send = intel_guc_send_mmio;
+ return 0;
+}
+
+static void guc_disable_communication(struct intel_guc *guc)
+{
+ struct drm_i915_private *dev_priv = guc_to_i915(guc);
+
+ if (HAS_GUC_CT(dev_priv))
+ intel_guc_disable_ct(guc);
+
+ guc->send = intel_guc_send_nop;
+}
+
int intel_uc_init_hw(struct drm_i915_private *dev_priv)
{
+ struct intel_guc *guc = &dev_priv->guc;
int ret, attempts;
if (!i915.enable_guc_loading)
return 0;
+ guc_disable_communication(guc);
gen9_reset_guc_interrupts(dev_priv);
/* We need to notify the guc whenever we change the GGTT */
goto err_guc;
}
+ /* init WOPCM */
+ I915_WRITE(GUC_WOPCM_SIZE, intel_guc_wopcm_size(dev_priv));
+ I915_WRITE(DMA_GUC_WOPCM_OFFSET,
+ GUC_WOPCM_OFFSET_VALUE | HUC_LOADING_AGENT_GUC);
+
/* WaEnableuKernelHeaderValidFix:skl */
/* WaEnableGuCBootHashCheckNotSet:skl,bxt,kbl */
if (IS_GEN9(dev_priv))
/* Did we succeded or run out of retries? */
if (ret)
- goto err_submission;
+ goto err_log_capture;
+
+ ret = guc_enable_communication(guc);
+ if (ret)
+ goto err_log_capture;
intel_guc_auth_huc(dev_priv);
if (i915.enable_guc_submission) {
* marks the GPU as wedged until reset).
*/
err_interrupts:
+ guc_disable_communication(guc);
gen9_disable_guc_interrupts(dev_priv);
+err_log_capture:
+ guc_capture_load_err_log(guc);
err_submission:
if (i915.enable_guc_submission)
i915_guc_submission_fini(dev_priv);
if (!i915.enable_guc_loading)
return;
- if (i915.enable_guc_submission) {
+ guc_free_load_err_log(&dev_priv->guc);
+
+ if (i915.enable_guc_submission)
i915_guc_submission_disable(dev_priv);
+
+ guc_disable_communication(&dev_priv->guc);
+
+ if (i915.enable_guc_submission) {
gen9_disable_guc_interrupts(dev_priv);
i915_guc_submission_fini(dev_priv);
}
+
i915_ggtt_disable_guc(dev_priv);
}
-/*
- * Read GuC command/status register (SOFT_SCRATCH_0)
- * Return true if it contains a response rather than a command
- */
-static bool guc_recv(struct intel_guc *guc, u32 *status)
+int intel_guc_send_nop(struct intel_guc *guc, const u32 *action, u32 len)
{
- struct drm_i915_private *dev_priv = guc_to_i915(guc);
-
- u32 val = I915_READ(SOFT_SCRATCH(0));
- *status = val;
- return INTEL_GUC_RECV_IS_RESPONSE(val);
+ WARN(1, "Unexpected send: action=%#x\n", *action);
+ return -ENODEV;
}
/*
int i;
int ret;
- if (WARN_ON(len < 1 || len > 15))
- return -EINVAL;
+ GEM_BUG_ON(!len);
+ GEM_BUG_ON(len > guc->send_regs.count);
- mutex_lock(&guc->send_mutex);
- intel_uncore_forcewake_get(dev_priv, FORCEWAKE_BLITTER);
+ /* If CT is available, we expect to use MMIO only during init/fini */
+ GEM_BUG_ON(HAS_GUC_CT(dev_priv) &&
+ *action != INTEL_GUC_ACTION_REGISTER_COMMAND_TRANSPORT_BUFFER &&
+ *action != INTEL_GUC_ACTION_DEREGISTER_COMMAND_TRANSPORT_BUFFER);
- dev_priv->guc.action_count += 1;
- dev_priv->guc.action_cmd = action[0];
+ mutex_lock(&guc->send_mutex);
+ intel_uncore_forcewake_get(dev_priv, guc->send_regs.fw_domains);
for (i = 0; i < len; i++)
- I915_WRITE(SOFT_SCRATCH(i), action[i]);
+ I915_WRITE(guc_send_reg(guc, i), action[i]);
- POSTING_READ(SOFT_SCRATCH(i - 1));
+ POSTING_READ(guc_send_reg(guc, i - 1));
- I915_WRITE(GUC_SEND_INTERRUPT, GUC_SEND_TRIGGER);
+ intel_guc_notify(guc);
/*
- * Fast commands should complete in less than 10us, so sample quickly
- * up to that length of time, then switch to a slower sleep-wait loop.
- * No inte_guc_send command should ever take longer than 10ms.
+ * No GuC command should ever take longer than 10ms.
+ * Fast commands should still complete in 10us.
*/
- ret = wait_for_us(guc_recv(guc, &status), 10);
- if (ret)
- ret = wait_for(guc_recv(guc, &status), 10);
+ ret = __intel_wait_for_register_fw(dev_priv,
+ guc_send_reg(guc, 0),
+ INTEL_GUC_RECV_MASK,
+ INTEL_GUC_RECV_MASK,
+ 10, 10, &status);
if (status != INTEL_GUC_STATUS_SUCCESS) {
/*
* Either the GuC explicitly returned an error (which
DRM_WARN("INTEL_GUC_SEND: Action 0x%X failed;"
" ret=%d status=0x%08X response=0x%08X\n",
action[0], ret, status, I915_READ(SOFT_SCRATCH(15)));
-
- dev_priv->guc.action_fail += 1;
- dev_priv->guc.action_err = ret;
}
- dev_priv->guc.action_status = status;
- intel_uncore_forcewake_put(dev_priv, FORCEWAKE_BLITTER);
+ intel_uncore_forcewake_put(dev_priv, guc->send_regs.fw_domains);
mutex_unlock(&guc->send_mutex);
return ret;
#include "intel_guc_fwif.h"
#include "i915_guc_reg.h"
#include "intel_ringbuffer.h"
-
+#include "intel_guc_ct.h"
#include "i915_vma.h"
struct drm_i915_gem_request;
* available in the work queue (note, the queue is shared,
* not per-engine). It is OK for this to be nonzero, but
* it should not be huge!
- * q_fail: failed to enqueue a work item. This should never happen,
- * because we check for space beforehand.
- * b_fail: failed to ring the doorbell. This should never happen, unless
- * somehow the hardware misbehaves, or maybe if the GuC firmware
- * crashes? We probably need to reset the GPU to recover.
- * retcode: errno from last guc_submit()
*/
struct i915_guc_client {
struct i915_vma *vma;
uint32_t wq_tail;
uint32_t wq_rsvd;
uint32_t no_wq_space;
- uint32_t b_fail;
- int retcode;
/* Per-engine counts of GuC submissions */
uint64_t submissions[I915_NUM_ENGINES];
struct intel_guc {
struct intel_uc_fw fw;
struct intel_guc_log log;
+ struct intel_guc_ct ct;
+
+ /* Log snapshot if GuC errors during load */
+ struct drm_i915_gem_object *load_err_log;
/* intel_guc_recv interrupt related state */
bool interrupts_enabled;
DECLARE_BITMAP(doorbell_bitmap, GUC_NUM_DOORBELLS);
uint32_t db_cacheline; /* Cyclic counter mod pagesize */
- /* Action status & statistics */
- uint64_t action_count; /* Total commands issued */
- uint32_t action_cmd; /* Last command word */
- uint32_t action_status; /* Last return status */
- uint32_t action_fail; /* Total number of failures */
- int32_t action_err; /* Last error code */
-
- uint64_t submissions[I915_NUM_ENGINES];
- uint32_t last_seqno[I915_NUM_ENGINES];
+ /* GuC's FW specific registers used in MMIO send */
+ struct {
+ u32 base;
+ unsigned int count;
+ enum forcewake_domains fw_domains;
+ } send_regs;
/* To serialize the intel_guc_send actions */
struct mutex send_mutex;
/* GuC's FW specific send function */
int (*send)(struct intel_guc *guc, const u32 *data, u32 len);
+
+ /* GuC's FW specific notify function */
+ void (*notify)(struct intel_guc *guc);
};
struct intel_huc {
int intel_uc_init_hw(struct drm_i915_private *dev_priv);
void intel_uc_fini_hw(struct drm_i915_private *dev_priv);
int intel_guc_sample_forcewake(struct intel_guc *guc);
+int intel_guc_send_nop(struct intel_guc *guc, const u32 *action, u32 len);
int intel_guc_send_mmio(struct intel_guc *guc, const u32 *action, u32 len);
+
static inline int intel_guc_send(struct intel_guc *guc, const u32 *action, u32 len)
{
return guc->send(guc, action, len);
}
+static inline void intel_guc_notify(struct intel_guc *guc)
+{
+ guc->notify(guc);
+}
+
/* intel_guc_loader.c */
int intel_guc_select_fw(struct intel_guc *guc);
int intel_guc_init_hw(struct intel_guc *guc);
/* intel_huc.c */
void intel_huc_select_fw(struct intel_huc *huc);
-int intel_huc_init_hw(struct intel_huc *huc);
+void intel_huc_init_hw(struct intel_huc *huc);
void intel_guc_auth_huc(struct drm_i915_private *dev_priv);
#endif
#include <linux/pm_runtime.h>
#define FORCEWAKE_ACK_TIMEOUT_MS 50
+#define GT_FIFO_TIMEOUT_MS 10
#define __raw_posting_read(dev_priv__, reg__) (void)__raw_i915_read32((dev_priv__), (reg__))
__gen6_gt_wait_for_thread_c0(dev_priv);
}
-static void gen6_gt_check_fifodbg(struct drm_i915_private *dev_priv)
-{
- u32 gtfifodbg;
-
- gtfifodbg = __raw_i915_read32(dev_priv, GTFIFODBG);
- if (WARN(gtfifodbg, "GT wake FIFO error 0x%x\n", gtfifodbg))
- __raw_i915_write32(dev_priv, GTFIFODBG, gtfifodbg);
-}
-
-static void fw_domains_put_with_fifo(struct drm_i915_private *dev_priv,
- enum forcewake_domains fw_domains)
-{
- fw_domains_put(dev_priv, fw_domains);
- gen6_gt_check_fifodbg(dev_priv);
-}
-
static inline u32 fifo_free_entries(struct drm_i915_private *dev_priv)
{
u32 count = __raw_i915_read32(dev_priv, GTFIFOCTL);
return count & GT_FIFO_FREE_ENTRIES_MASK;
}
-static int __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
+static void __gen6_gt_wait_for_fifo(struct drm_i915_private *dev_priv)
{
- int ret = 0;
+ u32 n;
/* On VLV, FIFO will be shared by both SW and HW.
* So, we need to read the FREE_ENTRIES everytime */
if (IS_VALLEYVIEW(dev_priv))
- dev_priv->uncore.fifo_count = fifo_free_entries(dev_priv);
-
- if (dev_priv->uncore.fifo_count < GT_FIFO_NUM_RESERVED_ENTRIES) {
- int loop = 500;
- u32 fifo = fifo_free_entries(dev_priv);
-
- while (fifo <= GT_FIFO_NUM_RESERVED_ENTRIES && loop--) {
- udelay(10);
- fifo = fifo_free_entries(dev_priv);
+ n = fifo_free_entries(dev_priv);
+ else
+ n = dev_priv->uncore.fifo_count;
+
+ if (n <= GT_FIFO_NUM_RESERVED_ENTRIES) {
+ if (wait_for_atomic((n = fifo_free_entries(dev_priv)) >
+ GT_FIFO_NUM_RESERVED_ENTRIES,
+ GT_FIFO_TIMEOUT_MS)) {
+ DRM_DEBUG("GT_FIFO timeout, entries: %u\n", n);
+ return;
}
- if (WARN_ON(loop < 0 && fifo <= GT_FIFO_NUM_RESERVED_ENTRIES))
- ++ret;
- dev_priv->uncore.fifo_count = fifo;
}
- dev_priv->uncore.fifo_count--;
- return ret;
+ dev_priv->uncore.fifo_count = n - 1;
}
static enum hrtimer_restart
assert_rpm_device_not_suspended(dev_priv);
+ if (xchg(&domain->active, false))
+ return HRTIMER_RESTART;
+
spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
if (WARN_ON(domain->wake_count == 0))
domain->wake_count++;
active_domains = 0;
for_each_fw_domain(domain, dev_priv, tmp) {
+ smp_store_mb(domain->active, false);
if (hrtimer_cancel(&domain->timer) == 0)
continue;
return true;
}
+static bool
+gen6_check_for_fifo_debug(struct drm_i915_private *dev_priv)
+{
+ u32 fifodbg;
+
+ fifodbg = __raw_i915_read32(dev_priv, GTFIFODBG);
+
+ if (unlikely(fifodbg)) {
+ DRM_DEBUG_DRIVER("GTFIFODBG = 0x08%x\n", fifodbg);
+ __raw_i915_write32(dev_priv, GTFIFODBG, fifodbg);
+ }
+
+ return fifodbg;
+}
+
static bool
check_for_unclaimed_mmio(struct drm_i915_private *dev_priv)
{
+ bool ret = false;
+
if (HAS_FPGA_DBG_UNCLAIMED(dev_priv))
- return fpga_check_for_unclaimed_mmio(dev_priv);
+ ret |= fpga_check_for_unclaimed_mmio(dev_priv);
if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
- return vlv_check_for_unclaimed_mmio(dev_priv);
+ ret |= vlv_check_for_unclaimed_mmio(dev_priv);
- return false;
+ if (IS_GEN6(dev_priv) || IS_GEN7(dev_priv))
+ ret |= gen6_check_for_fifo_debug(dev_priv);
+
+ return ret;
}
static void __intel_uncore_early_sanitize(struct drm_i915_private *dev_priv,
if (check_for_unclaimed_mmio(dev_priv))
DRM_DEBUG("unclaimed mmio detected on uncore init, clearing\n");
- /* clear out old GT FIFO errors */
- if (IS_GEN6(dev_priv) || IS_GEN7(dev_priv))
- __raw_i915_write32(dev_priv, GTFIFODBG,
- __raw_i915_read32(dev_priv, GTFIFODBG));
-
/* WaDisableShadowRegForCpd:chv */
if (IS_CHERRYVIEW(dev_priv)) {
__raw_i915_write32(dev_priv, GTFIFOCTL,
fw_domains &= dev_priv->uncore.fw_domains;
- for_each_fw_domain_masked(domain, fw_domains, dev_priv, tmp)
- if (domain->wake_count++)
+ for_each_fw_domain_masked(domain, fw_domains, dev_priv, tmp) {
+ if (domain->wake_count++) {
fw_domains &= ~domain->mask;
+ domain->active = true;
+ }
+ }
if (fw_domains)
dev_priv->uncore.funcs.force_wake_get(dev_priv, fw_domains);
if (WARN_ON(domain->wake_count == 0))
continue;
- if (--domain->wake_count)
+ if (--domain->wake_count) {
+ domain->active = true;
continue;
+ }
fw_domain_arm_timer(domain);
}
__unclaimed_reg_debug(dev_priv, reg, read, before);
}
+enum decoupled_power_domain {
+ GEN9_DECOUPLED_PD_BLITTER = 0,
+ GEN9_DECOUPLED_PD_RENDER,
+ GEN9_DECOUPLED_PD_MEDIA,
+ GEN9_DECOUPLED_PD_ALL
+};
+
+enum decoupled_ops {
+ GEN9_DECOUPLED_OP_WRITE = 0,
+ GEN9_DECOUPLED_OP_READ
+};
+
static const enum decoupled_power_domain fw2dpd_domain[] = {
GEN9_DECOUPLED_PD_RENDER,
GEN9_DECOUPLED_PD_BLITTER,
#define __gen6_write(x) \
static void \
gen6_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { \
- u32 __fifo_ret = 0; \
GEN6_WRITE_HEADER; \
- if (NEEDS_FORCE_WAKE(offset)) { \
- __fifo_ret = __gen6_gt_wait_for_fifo(dev_priv); \
- } \
+ if (NEEDS_FORCE_WAKE(offset)) \
+ __gen6_gt_wait_for_fifo(dev_priv); \
__raw_i915_write##x(dev_priv, reg, val); \
- if (unlikely(__fifo_ret)) { \
- gen6_gt_check_fifodbg(dev_priv); \
- } \
GEN6_WRITE_FOOTER; \
}
#undef GEN6_WRITE_FOOTER
#undef GEN6_WRITE_HEADER
-#define ASSIGN_WRITE_MMIO_VFUNCS(x) \
+#define ASSIGN_WRITE_MMIO_VFUNCS(i915, x) \
do { \
- dev_priv->uncore.funcs.mmio_writeb = x##_write8; \
- dev_priv->uncore.funcs.mmio_writew = x##_write16; \
- dev_priv->uncore.funcs.mmio_writel = x##_write32; \
+ (i915)->uncore.funcs.mmio_writeb = x##_write8; \
+ (i915)->uncore.funcs.mmio_writew = x##_write16; \
+ (i915)->uncore.funcs.mmio_writel = x##_write32; \
} while (0)
-#define ASSIGN_READ_MMIO_VFUNCS(x) \
+#define ASSIGN_READ_MMIO_VFUNCS(i915, x) \
do { \
- dev_priv->uncore.funcs.mmio_readb = x##_read8; \
- dev_priv->uncore.funcs.mmio_readw = x##_read16; \
- dev_priv->uncore.funcs.mmio_readl = x##_read32; \
- dev_priv->uncore.funcs.mmio_readq = x##_read64; \
+ (i915)->uncore.funcs.mmio_readb = x##_read8; \
+ (i915)->uncore.funcs.mmio_readw = x##_read16; \
+ (i915)->uncore.funcs.mmio_readl = x##_read32; \
+ (i915)->uncore.funcs.mmio_readq = x##_read64; \
} while (0)
FORCEWAKE_MEDIA_GEN9, FORCEWAKE_ACK_MEDIA_GEN9);
} else if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv)) {
dev_priv->uncore.funcs.force_wake_get = fw_domains_get;
- if (!IS_CHERRYVIEW(dev_priv))
- dev_priv->uncore.funcs.force_wake_put =
- fw_domains_put_with_fifo;
- else
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
+ dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
FORCEWAKE_VLV, FORCEWAKE_ACK_VLV);
fw_domain_init(dev_priv, FW_DOMAIN_ID_MEDIA,
} else if (IS_HASWELL(dev_priv) || IS_BROADWELL(dev_priv)) {
dev_priv->uncore.funcs.force_wake_get =
fw_domains_get_with_thread_status;
- if (IS_HASWELL(dev_priv))
- dev_priv->uncore.funcs.force_wake_put =
- fw_domains_put_with_fifo;
- else
- dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
+ dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
FORCEWAKE_MT, FORCEWAKE_ACK_HSW);
} else if (IS_IVYBRIDGE(dev_priv)) {
*/
dev_priv->uncore.funcs.force_wake_get =
fw_domains_get_with_thread_status;
- dev_priv->uncore.funcs.force_wake_put =
- fw_domains_put_with_fifo;
+ dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
/* We need to init first for ECOBUS access and then
* determine later if we want to reinit, in case of MT access is
spin_lock_irq(&dev_priv->uncore.lock);
fw_domains_get_with_thread_status(dev_priv, FORCEWAKE_RENDER);
ecobus = __raw_i915_read32(dev_priv, ECOBUS);
- fw_domains_put_with_fifo(dev_priv, FORCEWAKE_RENDER);
+ fw_domains_put(dev_priv, FORCEWAKE_RENDER);
spin_unlock_irq(&dev_priv->uncore.lock);
if (!(ecobus & FORCEWAKE_MT_ENABLE)) {
} else if (IS_GEN6(dev_priv)) {
dev_priv->uncore.funcs.force_wake_get =
fw_domains_get_with_thread_status;
- dev_priv->uncore.funcs.force_wake_put =
- fw_domains_put_with_fifo;
+ dev_priv->uncore.funcs.force_wake_put = fw_domains_put;
fw_domain_init(dev_priv, FW_DOMAIN_ID_RENDER,
FORCEWAKE, FORCEWAKE_ACK);
}
i915_pmic_bus_access_notifier;
if (IS_GEN(dev_priv, 2, 4) || intel_vgpu_active(dev_priv)) {
- ASSIGN_WRITE_MMIO_VFUNCS(gen2);
- ASSIGN_READ_MMIO_VFUNCS(gen2);
+ ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen2);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen2);
} else if (IS_GEN5(dev_priv)) {
- ASSIGN_WRITE_MMIO_VFUNCS(gen5);
- ASSIGN_READ_MMIO_VFUNCS(gen5);
+ ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen5);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen5);
} else if (IS_GEN(dev_priv, 6, 7)) {
- ASSIGN_WRITE_MMIO_VFUNCS(gen6);
+ ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen6);
if (IS_VALLEYVIEW(dev_priv)) {
ASSIGN_FW_DOMAINS_TABLE(__vlv_fw_ranges);
- ASSIGN_READ_MMIO_VFUNCS(fwtable);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, fwtable);
} else {
- ASSIGN_READ_MMIO_VFUNCS(gen6);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen6);
}
} else if (IS_GEN8(dev_priv)) {
if (IS_CHERRYVIEW(dev_priv)) {
ASSIGN_FW_DOMAINS_TABLE(__chv_fw_ranges);
- ASSIGN_WRITE_MMIO_VFUNCS(fwtable);
- ASSIGN_READ_MMIO_VFUNCS(fwtable);
+ ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, fwtable);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, fwtable);
} else {
- ASSIGN_WRITE_MMIO_VFUNCS(gen8);
- ASSIGN_READ_MMIO_VFUNCS(gen6);
+ ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, gen8);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, gen6);
}
} else {
ASSIGN_FW_DOMAINS_TABLE(__gen9_fw_ranges);
- ASSIGN_WRITE_MMIO_VFUNCS(fwtable);
- ASSIGN_READ_MMIO_VFUNCS(fwtable);
+ ASSIGN_WRITE_MMIO_VFUNCS(dev_priv, fwtable);
+ ASSIGN_READ_MMIO_VFUNCS(dev_priv, fwtable);
if (HAS_DECOUPLED_MMIO(dev_priv)) {
dev_priv->uncore.funcs.mmio_readl =
gen9_decoupled_read32;
i915_check_and_clear_faults(dev_priv);
}
-#undef ASSIGN_WRITE_MMIO_VFUNCS
-#undef ASSIGN_READ_MMIO_VFUNCS
void intel_uncore_fini(struct drm_i915_private *dev_priv)
{
return ret;
}
-static int i915_reset_complete(struct pci_dev *pdev)
+static void gen3_stop_rings(struct drm_i915_private *dev_priv)
+{
+ struct intel_engine_cs *engine;
+ enum intel_engine_id id;
+
+ for_each_engine(engine, dev_priv, id) {
+ const u32 base = engine->mmio_base;
+ const i915_reg_t mode = RING_MI_MODE(base);
+
+ I915_WRITE_FW(mode, _MASKED_BIT_ENABLE(STOP_RING));
+ if (intel_wait_for_register_fw(dev_priv,
+ mode,
+ MODE_IDLE,
+ MODE_IDLE,
+ 500))
+ DRM_DEBUG_DRIVER("%s: timed out on STOP_RING\n",
+ engine->name);
+
+ I915_WRITE_FW(RING_CTL(base), 0);
+ I915_WRITE_FW(RING_HEAD(base), 0);
+ I915_WRITE_FW(RING_TAIL(base), 0);
+
+ /* Check acts as a post */
+ if (I915_READ_FW(RING_HEAD(base)) != 0)
+ DRM_DEBUG_DRIVER("%s: ring head not parked\n",
+ engine->name);
+ }
+}
+
+static bool i915_reset_complete(struct pci_dev *pdev)
{
u8 gdrst;
+
pci_read_config_byte(pdev, I915_GDRST, &gdrst);
return (gdrst & GRDOM_RESET_STATUS) == 0;
}
/* assert reset for at least 20 usec */
pci_write_config_byte(pdev, I915_GDRST, GRDOM_RESET_ENABLE);
- udelay(20);
+ usleep_range(50, 200);
pci_write_config_byte(pdev, I915_GDRST, 0);
return wait_for(i915_reset_complete(pdev), 500);
}
-static int g4x_reset_complete(struct pci_dev *pdev)
+static bool g4x_reset_complete(struct pci_dev *pdev)
{
u8 gdrst;
+
pci_read_config_byte(pdev, I915_GDRST, &gdrst);
return (gdrst & GRDOM_RESET_ENABLE) == 0;
}
static int g33_do_reset(struct drm_i915_private *dev_priv, unsigned engine_mask)
{
struct pci_dev *pdev = dev_priv->drm.pdev;
+
+ /* Stop engines before we reset; see g4x_do_reset() below for why. */
+ gen3_stop_rings(dev_priv);
+
pci_write_config_byte(pdev, I915_GDRST, GRDOM_RESET_ENABLE);
return wait_for(g4x_reset_complete(pdev), 500);
}
struct pci_dev *pdev = dev_priv->drm.pdev;
int ret;
- pci_write_config_byte(pdev, I915_GDRST,
- GRDOM_RENDER | GRDOM_RESET_ENABLE);
- ret = wait_for(g4x_reset_complete(pdev), 500);
- if (ret)
- return ret;
-
/* WaVcpClkGateDisableForMediaReset:ctg,elk */
- I915_WRITE(VDECCLK_GATE_D, I915_READ(VDECCLK_GATE_D) | VCP_UNIT_CLOCK_GATE_DISABLE);
+ I915_WRITE(VDECCLK_GATE_D,
+ I915_READ(VDECCLK_GATE_D) | VCP_UNIT_CLOCK_GATE_DISABLE);
POSTING_READ(VDECCLK_GATE_D);
+ /* We stop engines, otherwise we might get failed reset and a
+ * dead gpu (on elk).
+ * WaMediaResetMainRingCleanup:ctg,elk (presumably)
+ */
+ gen3_stop_rings(dev_priv);
+
pci_write_config_byte(pdev, I915_GDRST,
GRDOM_MEDIA | GRDOM_RESET_ENABLE);
ret = wait_for(g4x_reset_complete(pdev), 500);
- if (ret)
- return ret;
+ if (ret) {
+ DRM_DEBUG_DRIVER("Wait for media reset failed\n");
+ goto out;
+ }
- /* WaVcpClkGateDisableForMediaReset:ctg,elk */
- I915_WRITE(VDECCLK_GATE_D, I915_READ(VDECCLK_GATE_D) & ~VCP_UNIT_CLOCK_GATE_DISABLE);
- POSTING_READ(VDECCLK_GATE_D);
+ pci_write_config_byte(pdev, I915_GDRST,
+ GRDOM_RENDER | GRDOM_RESET_ENABLE);
+ ret = wait_for(g4x_reset_complete(pdev), 500);
+ if (ret) {
+ DRM_DEBUG_DRIVER("Wait for render reset failed\n");
+ goto out;
+ }
+out:
pci_write_config_byte(pdev, I915_GDRST, 0);
- return 0;
+ I915_WRITE(VDECCLK_GATE_D,
+ I915_READ(VDECCLK_GATE_D) & ~VCP_UNIT_CLOCK_GATE_DISABLE);
+ POSTING_READ(VDECCLK_GATE_D);
+
+ return ret;
}
static int ironlake_do_reset(struct drm_i915_private *dev_priv,
{
int ret;
- I915_WRITE(ILK_GDSR,
- ILK_GRDOM_RENDER | ILK_GRDOM_RESET_ENABLE);
+ I915_WRITE(ILK_GDSR, ILK_GRDOM_RENDER | ILK_GRDOM_RESET_ENABLE);
ret = intel_wait_for_register(dev_priv,
ILK_GDSR, ILK_GRDOM_RESET_ENABLE, 0,
500);
- if (ret)
- return ret;
+ if (ret) {
+ DRM_DEBUG_DRIVER("Wait for render reset failed\n");
+ goto out;
+ }
- I915_WRITE(ILK_GDSR,
- ILK_GRDOM_MEDIA | ILK_GRDOM_RESET_ENABLE);
+ I915_WRITE(ILK_GDSR, ILK_GRDOM_MEDIA | ILK_GRDOM_RESET_ENABLE);
ret = intel_wait_for_register(dev_priv,
ILK_GDSR, ILK_GRDOM_RESET_ENABLE, 0,
500);
- if (ret)
- return ret;
+ if (ret) {
+ DRM_DEBUG_DRIVER("Wait for media reset failed\n");
+ goto out;
+ }
+out:
I915_WRITE(ILK_GDSR, 0);
-
- return 0;
+ POSTING_READ(ILK_GDSR);
+ return ret;
}
/* Reset the hardware domains (GENX_GRDOM_*) specified by mask */
static int gen6_hw_domain_reset(struct drm_i915_private *dev_priv,
u32 hw_domain_mask)
{
+ int err;
+
/* GEN6_GDRST is not in the gt power well, no need to check
* for fifo space for the write or forcewake the chip for
* the read
*/
__raw_i915_write32(dev_priv, GEN6_GDRST, hw_domain_mask);
- /* Spin waiting for the device to ack the reset requests */
- return intel_wait_for_register_fw(dev_priv,
+ /* Wait for the device to ack the reset requests */
+ err = intel_wait_for_register_fw(dev_priv,
GEN6_GDRST, hw_domain_mask, 0,
500);
+ if (err)
+ DRM_DEBUG_DRIVER("Wait for 0x%08x engines reset failed\n",
+ hw_domain_mask);
+
+ return err;
}
/**
}
/**
- * intel_wait_for_register_fw - wait until register matches expected state
+ * __intel_wait_for_register_fw - wait until register matches expected state
* @dev_priv: the i915 device
* @reg: the register to read
* @mask: mask to apply to register value
* @value: expected value
- * @timeout_ms: timeout in millisecond
+ * @fast_timeout_us: fast timeout in microsecond for atomic/tight wait
+ * @slow_timeout_ms: slow timeout in millisecond
+ * @out_value: optional placeholder to hold registry value
*
* This routine waits until the target register @reg contains the expected
* @value after applying the @mask, i.e. it waits until ::
*
* (I915_READ_FW(reg) & mask) == value
*
- * Otherwise, the wait will timeout after @timeout_ms milliseconds.
+ * Otherwise, the wait will timeout after @slow_timeout_ms milliseconds.
+ * For atomic context @slow_timeout_ms must be zero and @fast_timeout_us
+ * must be not larger than 20,0000 microseconds.
*
* Note that this routine assumes the caller holds forcewake asserted, it is
* not suitable for very long waits. See intel_wait_for_register() if you
*
* Returns 0 if the register matches the desired condition, or -ETIMEOUT.
*/
-int intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
- i915_reg_t reg,
- const u32 mask,
- const u32 value,
- const unsigned long timeout_ms)
-{
-#define done ((I915_READ_FW(reg) & mask) == value)
- int ret = wait_for_us(done, 2);
- if (ret)
- ret = wait_for(done, timeout_ms);
+int __intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
+ unsigned int fast_timeout_us,
+ unsigned int slow_timeout_ms,
+ u32 *out_value)
+{
+ u32 uninitialized_var(reg_value);
+#define done (((reg_value = I915_READ_FW(reg)) & mask) == value)
+ int ret;
+
+ /* Catch any overuse of this function */
+ might_sleep_if(slow_timeout_ms);
+ GEM_BUG_ON(fast_timeout_us > 20000);
+
+ ret = -ETIMEDOUT;
+ if (fast_timeout_us && fast_timeout_us <= 20000)
+ ret = _wait_for_atomic(done, fast_timeout_us, 0);
+ if (ret && slow_timeout_ms)
+ ret = wait_for(done, slow_timeout_ms);
+
+ if (out_value)
+ *out_value = reg_value;
+
return ret;
#undef done
}
*/
int intel_wait_for_register(struct drm_i915_private *dev_priv,
i915_reg_t reg,
- const u32 mask,
- const u32 value,
- const unsigned long timeout_ms)
+ u32 mask,
+ u32 value,
+ unsigned int timeout_ms)
{
-
unsigned fw =
intel_uncore_forcewake_for_reg(dev_priv, reg, FW_REG_READ);
int ret;
- intel_uncore_forcewake_get(dev_priv, fw);
- ret = wait_for_us((I915_READ_FW(reg) & mask) == value, 2);
- intel_uncore_forcewake_put(dev_priv, fw);
+ might_sleep();
+
+ spin_lock_irq(&dev_priv->uncore.lock);
+ intel_uncore_forcewake_get__locked(dev_priv, fw);
+
+ ret = __intel_wait_for_register_fw(dev_priv,
+ reg, mask, value,
+ 2, 0, NULL);
+
+ intel_uncore_forcewake_put__locked(dev_priv, fw);
+ spin_unlock_irq(&dev_priv->uncore.lock);
+
if (ret)
ret = wait_for((I915_READ_NOTRACE(reg) & mask) == value,
timeout_ms);
return ret;
}
-static int gen8_request_engine_reset(struct intel_engine_cs *engine)
+static int gen8_reset_engine_start(struct intel_engine_cs *engine)
{
struct drm_i915_private *dev_priv = engine->i915;
int ret;
return ret;
}
-static void gen8_unrequest_engine_reset(struct intel_engine_cs *engine)
+static void gen8_reset_engine_cancel(struct intel_engine_cs *engine)
{
struct drm_i915_private *dev_priv = engine->i915;
unsigned int tmp;
for_each_engine_masked(engine, dev_priv, engine_mask, tmp)
- if (gen8_request_engine_reset(engine))
+ if (gen8_reset_engine_start(engine))
goto not_ready;
return gen6_reset_engines(dev_priv, engine_mask);
not_ready:
for_each_engine_masked(engine, dev_priv, engine_mask, tmp)
- gen8_unrequest_engine_reset(engine);
+ gen8_reset_engine_cancel(engine);
return -EIO;
}
int intel_gpu_reset(struct drm_i915_private *dev_priv, unsigned engine_mask)
{
reset_func reset;
+ int retry;
int ret;
+ might_sleep();
+
reset = intel_get_gpu_reset(dev_priv);
if (reset == NULL)
return -ENODEV;
* request may be dropped and never completes (causing -EIO).
*/
intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
- ret = reset(dev_priv, engine_mask);
+ for (retry = 0; retry < 3; retry++) {
+ ret = reset(dev_priv, engine_mask);
+ if (ret != -ETIMEDOUT)
+ break;
+
+ cond_resched();
+ }
intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
return ret;
int intel_guc_reset(struct drm_i915_private *dev_priv)
{
int ret;
- unsigned long irqflags;
if (!HAS_GUC(dev_priv))
return -EINVAL;
intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
- spin_lock_irqsave(&dev_priv->uncore.lock, irqflags);
-
ret = gen6_hw_domain_reset(dev_priv, GEN9_GRDOM_GUC);
-
- spin_unlock_irqrestore(&dev_priv->uncore.lock, irqflags);
intel_uncore_forcewake_put(dev_priv, FORCEWAKE_ALL);
return ret;
}
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
+#include "selftests/mock_uncore.c"
#include "selftests/intel_uncore.c"
#endif
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __INTEL_UNCORE_H__
+#define __INTEL_UNCORE_H__
+
+struct drm_i915_private;
+
+enum forcewake_domain_id {
+ FW_DOMAIN_ID_RENDER = 0,
+ FW_DOMAIN_ID_BLITTER,
+ FW_DOMAIN_ID_MEDIA,
+
+ FW_DOMAIN_ID_COUNT
+};
+
+enum forcewake_domains {
+ FORCEWAKE_RENDER = BIT(FW_DOMAIN_ID_RENDER),
+ FORCEWAKE_BLITTER = BIT(FW_DOMAIN_ID_BLITTER),
+ FORCEWAKE_MEDIA = BIT(FW_DOMAIN_ID_MEDIA),
+ FORCEWAKE_ALL = (FORCEWAKE_RENDER |
+ FORCEWAKE_BLITTER |
+ FORCEWAKE_MEDIA)
+};
+
+struct intel_uncore_funcs {
+ void (*force_wake_get)(struct drm_i915_private *dev_priv,
+ enum forcewake_domains domains);
+ void (*force_wake_put)(struct drm_i915_private *dev_priv,
+ enum forcewake_domains domains);
+
+ uint8_t (*mmio_readb)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, bool trace);
+ uint16_t (*mmio_readw)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, bool trace);
+ uint32_t (*mmio_readl)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, bool trace);
+ uint64_t (*mmio_readq)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, bool trace);
+
+ void (*mmio_writeb)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, uint8_t val, bool trace);
+ void (*mmio_writew)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, uint16_t val, bool trace);
+ void (*mmio_writel)(struct drm_i915_private *dev_priv,
+ i915_reg_t r, uint32_t val, bool trace);
+};
+
+struct intel_forcewake_range {
+ u32 start;
+ u32 end;
+
+ enum forcewake_domains domains;
+};
+
+struct intel_uncore {
+ spinlock_t lock; /** lock is also taken in irq contexts. */
+
+ const struct intel_forcewake_range *fw_domains_table;
+ unsigned int fw_domains_table_entries;
+
+ struct notifier_block pmic_bus_access_nb;
+ struct intel_uncore_funcs funcs;
+
+ unsigned int fifo_count;
+
+ enum forcewake_domains fw_domains;
+ enum forcewake_domains fw_domains_active;
+
+ u32 fw_set;
+ u32 fw_clear;
+ u32 fw_reset;
+
+ struct intel_uncore_forcewake_domain {
+ enum forcewake_domain_id id;
+ enum forcewake_domains mask;
+ unsigned int wake_count;
+ bool active;
+ struct hrtimer timer;
+ i915_reg_t reg_set;
+ i915_reg_t reg_ack;
+ } fw_domain[FW_DOMAIN_ID_COUNT];
+
+ int unclaimed_mmio_check;
+};
+
+/* Iterate over initialised fw domains */
+#define for_each_fw_domain_masked(domain__, mask__, dev_priv__, tmp__) \
+ for (tmp__ = (mask__); \
+ tmp__ ? (domain__ = &(dev_priv__)->uncore.fw_domain[__mask_next_bit(tmp__)]), 1 : 0;)
+
+#define for_each_fw_domain(domain__, dev_priv__, tmp__) \
+ for_each_fw_domain_masked(domain__, (dev_priv__)->uncore.fw_domains, dev_priv__, tmp__)
+
+
+void intel_uncore_sanitize(struct drm_i915_private *dev_priv);
+void intel_uncore_init(struct drm_i915_private *dev_priv);
+bool intel_uncore_unclaimed_mmio(struct drm_i915_private *dev_priv);
+bool intel_uncore_arm_unclaimed_mmio_detection(struct drm_i915_private *dev_priv);
+void intel_uncore_fini(struct drm_i915_private *dev_priv);
+void intel_uncore_suspend(struct drm_i915_private *dev_priv);
+void intel_uncore_resume_early(struct drm_i915_private *dev_priv);
+
+u64 intel_uncore_edram_size(struct drm_i915_private *dev_priv);
+void assert_forcewakes_inactive(struct drm_i915_private *dev_priv);
+const char *intel_uncore_forcewake_domain_to_str(const enum forcewake_domain_id id);
+
+enum forcewake_domains
+intel_uncore_forcewake_for_reg(struct drm_i915_private *dev_priv,
+ i915_reg_t reg, unsigned int op);
+#define FW_REG_READ (1)
+#define FW_REG_WRITE (2)
+
+void intel_uncore_forcewake_get(struct drm_i915_private *dev_priv,
+ enum forcewake_domains domains);
+void intel_uncore_forcewake_put(struct drm_i915_private *dev_priv,
+ enum forcewake_domains domains);
+/* Like above but the caller must manage the uncore.lock itself.
+ * Must be used with I915_READ_FW and friends.
+ */
+void intel_uncore_forcewake_get__locked(struct drm_i915_private *dev_priv,
+ enum forcewake_domains domains);
+void intel_uncore_forcewake_put__locked(struct drm_i915_private *dev_priv,
+ enum forcewake_domains domains);
+
+int intel_wait_for_register(struct drm_i915_private *dev_priv,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
+ unsigned int timeout_ms);
+int __intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
+ unsigned int fast_timeout_us,
+ unsigned int slow_timeout_ms,
+ u32 *out_value);
+static inline
+int intel_wait_for_register_fw(struct drm_i915_private *dev_priv,
+ i915_reg_t reg,
+ u32 mask,
+ u32 value,
+ unsigned int timeout_ms)
+{
+ return __intel_wait_for_register_fw(dev_priv, reg, mask, value,
+ 2, timeout_ms, NULL);
+}
+
+#endif /* !__INTEL_UNCORE_H__ */
typeof(v) *map;
int err;
- /* XXX GTT write followed by WC write go missing */
- i915_gem_object_flush_gtt_write_domain(obj);
-
- err = i915_gem_object_set_to_gtt_domain(obj, true);
+ err = i915_gem_object_set_to_wc_domain(obj, true);
if (err)
return err;
typeof(v) map;
int err;
- /* XXX WC write followed by GTT write go missing */
- i915_gem_object_flush_gtt_write_domain(obj);
-
- err = i915_gem_object_set_to_gtt_domain(obj, false);
+ err = i915_gem_object_set_to_wc_domain(obj, false);
if (err)
return err;
static int igt_ctx_exec(void *arg)
{
struct drm_i915_private *i915 = arg;
- struct drm_i915_gem_object *obj;
+ struct drm_i915_gem_object *obj = NULL;
struct drm_file *file;
IGT_TIMEOUT(end_time);
LIST_HEAD(objects);
}
for_each_engine(engine, i915, id) {
- if (dw == 0) {
+ if (!obj) {
obj = create_test_object(ctx, file, &objects);
if (IS_ERR(obj)) {
err = PTR_ERR(obj);
goto out_unlock;
}
- if (++dw == max_dwords(obj))
+ if (++dw == max_dwords(obj)) {
+ obj = NULL;
dw = 0;
+ }
ndwords++;
}
ncontexts++;
return err;
}
+static int igt_dmabuf_export_kmap(void *arg)
+{
+ struct drm_i915_private *i915 = arg;
+ struct drm_i915_gem_object *obj;
+ struct dma_buf *dmabuf;
+ void *ptr;
+ int err;
+
+ obj = i915_gem_object_create(i915, 2*PAGE_SIZE);
+ if (IS_ERR(obj))
+ return PTR_ERR(obj);
+
+ dmabuf = i915_gem_prime_export(&i915->drm, &obj->base, 0);
+ i915_gem_object_put(obj);
+ if (IS_ERR(dmabuf)) {
+ err = PTR_ERR(dmabuf);
+ pr_err("i915_gem_prime_export failed with err=%d\n", err);
+ return err;
+ }
+
+ ptr = dma_buf_kmap(dmabuf, 0);
+ if (!ptr) {
+ pr_err("dma_buf_kmap failed\n");
+ err = -ENOMEM;
+ goto err;
+ }
+
+ if (memchr_inv(ptr, 0, PAGE_SIZE)) {
+ dma_buf_kunmap(dmabuf, 0, ptr);
+ pr_err("Exported page[0] not initialiased to zero!\n");
+ err = -EINVAL;
+ goto err;
+ }
+
+ memset(ptr, 0xc5, PAGE_SIZE);
+ dma_buf_kunmap(dmabuf, 0, ptr);
+
+ ptr = i915_gem_object_pin_map(obj, I915_MAP_WB);
+ if (IS_ERR(ptr)) {
+ err = PTR_ERR(ptr);
+ pr_err("i915_gem_object_pin_map failed with err=%d\n", err);
+ goto err;
+ }
+ memset(ptr + PAGE_SIZE, 0xaa, PAGE_SIZE);
+ i915_gem_object_unpin_map(obj);
+
+ ptr = dma_buf_kmap(dmabuf, 1);
+ if (!ptr) {
+ pr_err("dma_buf_kmap failed\n");
+ err = -ENOMEM;
+ goto err;
+ }
+
+ if (memchr_inv(ptr, 0xaa, PAGE_SIZE)) {
+ dma_buf_kunmap(dmabuf, 1, ptr);
+ pr_err("Exported page[1] not set to 0xaa!\n");
+ err = -EINVAL;
+ goto err;
+ }
+
+ memset(ptr, 0xc5, PAGE_SIZE);
+ dma_buf_kunmap(dmabuf, 1, ptr);
+
+ ptr = dma_buf_kmap(dmabuf, 0);
+ if (!ptr) {
+ pr_err("dma_buf_kmap failed\n");
+ err = -ENOMEM;
+ goto err;
+ }
+ if (memchr_inv(ptr, 0xc5, PAGE_SIZE)) {
+ dma_buf_kunmap(dmabuf, 0, ptr);
+ pr_err("Exported page[0] did not retain 0xc5!\n");
+ err = -EINVAL;
+ goto err;
+ }
+ dma_buf_kunmap(dmabuf, 0, ptr);
+
+ ptr = dma_buf_kmap(dmabuf, 2);
+ if (ptr) {
+ pr_err("Erroneously kmapped beyond the end of the object!\n");
+ dma_buf_kunmap(dmabuf, 2, ptr);
+ err = -EINVAL;
+ goto err;
+ }
+
+ ptr = dma_buf_kmap(dmabuf, -1);
+ if (ptr) {
+ pr_err("Erroneously kmapped before the start of the object!\n");
+ dma_buf_kunmap(dmabuf, -1, ptr);
+ err = -EINVAL;
+ goto err;
+ }
+
+ err = 0;
+err:
+ dma_buf_put(dmabuf);
+ return err;
+}
+
int i915_gem_dmabuf_mock_selftests(void)
{
static const struct i915_subtest tests[] = {
SUBTEST(igt_dmabuf_import),
SUBTEST(igt_dmabuf_import_ownership),
SUBTEST(igt_dmabuf_export_vmap),
+ SUBTEST(igt_dmabuf_export_kmap),
};
struct drm_i915_private *i915;
int err;
if (offset >= obj->base.size)
continue;
- i915_gem_object_flush_gtt_write_domain(obj);
+ flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);
p = i915_gem_object_get_page(obj, offset >> PAGE_SHIFT);
cpu = kmap(p) + offset_in_page(offset);
}
mutex_lock(&i915->drm.struct_mutex);
+ intel_runtime_pm_get(i915);
err = make_obj_busy(obj);
+ intel_runtime_pm_put(i915);
mutex_unlock(&i915->drm.struct_mutex);
if (err) {
pr_err("[loop %d] Failed to busy the object\n", loop);
if (err)
goto err;
- err = i915_gem_object_set_to_gtt_domain(obj, true);
+ err = i915_gem_object_set_to_wc_domain(obj, true);
if (err)
goto err;
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "../i915_selftest.h"
+#include "i915_random.h"
+
+#include "mock_gem_device.h"
+#include "mock_timeline.h"
+
+struct __igt_sync {
+ const char *name;
+ u32 seqno;
+ bool expected;
+ bool set;
+};
+
+static int __igt_sync(struct intel_timeline *tl,
+ u64 ctx,
+ const struct __igt_sync *p,
+ const char *name)
+{
+ int ret;
+
+ if (__intel_timeline_sync_is_later(tl, ctx, p->seqno) != p->expected) {
+ pr_err("%s: %s(ctx=%llu, seqno=%u) expected passed %s but failed\n",
+ name, p->name, ctx, p->seqno, yesno(p->expected));
+ return -EINVAL;
+ }
+
+ if (p->set) {
+ ret = __intel_timeline_sync_set(tl, ctx, p->seqno);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int igt_sync(void *arg)
+{
+ const struct __igt_sync pass[] = {
+ { "unset", 0, false, false },
+ { "new", 0, false, true },
+ { "0a", 0, true, true },
+ { "1a", 1, false, true },
+ { "1b", 1, true, true },
+ { "0b", 0, true, false },
+ { "2a", 2, false, true },
+ { "4", 4, false, true },
+ { "INT_MAX", INT_MAX, false, true },
+ { "INT_MAX-1", INT_MAX-1, true, false },
+ { "INT_MAX+1", (u32)INT_MAX+1, false, true },
+ { "INT_MAX", INT_MAX, true, false },
+ { "UINT_MAX", UINT_MAX, false, true },
+ { "wrap", 0, false, true },
+ { "unwrap", UINT_MAX, true, false },
+ {},
+ }, *p;
+ struct intel_timeline *tl;
+ int order, offset;
+ int ret;
+
+ tl = mock_timeline(0);
+ if (!tl)
+ return -ENOMEM;
+
+ for (p = pass; p->name; p++) {
+ for (order = 1; order < 64; order++) {
+ for (offset = -1; offset <= (order > 1); offset++) {
+ u64 ctx = BIT_ULL(order) + offset;
+
+ ret = __igt_sync(tl, ctx, p, "1");
+ if (ret)
+ goto out;
+ }
+ }
+ }
+ mock_timeline_destroy(tl);
+
+ tl = mock_timeline(0);
+ if (!tl)
+ return -ENOMEM;
+
+ for (order = 1; order < 64; order++) {
+ for (offset = -1; offset <= (order > 1); offset++) {
+ u64 ctx = BIT_ULL(order) + offset;
+
+ for (p = pass; p->name; p++) {
+ ret = __igt_sync(tl, ctx, p, "2");
+ if (ret)
+ goto out;
+ }
+ }
+ }
+
+out:
+ mock_timeline_destroy(tl);
+ return ret;
+}
+
+static unsigned int random_engine(struct rnd_state *rnd)
+{
+ return ((u64)prandom_u32_state(rnd) * I915_NUM_ENGINES) >> 32;
+}
+
+static int bench_sync(void *arg)
+{
+ struct rnd_state prng;
+ struct intel_timeline *tl;
+ unsigned long end_time, count;
+ u64 prng32_1M;
+ ktime_t kt;
+ int order, last_order;
+
+ tl = mock_timeline(0);
+ if (!tl)
+ return -ENOMEM;
+
+ /* Lookups from cache are very fast and so the random number generation
+ * and the loop itself becomes a significant factor in the per-iteration
+ * timings. We try to compensate the results by measuring the overhead
+ * of the prng and subtract it from the reported results.
+ */
+ prandom_seed_state(&prng, i915_selftest.random_seed);
+ count = 0;
+ kt = ktime_get();
+ end_time = jiffies + HZ/10;
+ do {
+ u32 x;
+
+ /* Make sure the compiler doesn't optimise away the prng call */
+ WRITE_ONCE(x, prandom_u32_state(&prng));
+
+ count++;
+ } while (!time_after(jiffies, end_time));
+ kt = ktime_sub(ktime_get(), kt);
+ pr_debug("%s: %lu random evaluations, %lluns/prng\n",
+ __func__, count, (long long)div64_ul(ktime_to_ns(kt), count));
+ prng32_1M = div64_ul(ktime_to_ns(kt) << 20, count);
+
+ /* Benchmark (only) setting random context ids */
+ prandom_seed_state(&prng, i915_selftest.random_seed);
+ count = 0;
+ kt = ktime_get();
+ end_time = jiffies + HZ/10;
+ do {
+ u64 id = i915_prandom_u64_state(&prng);
+
+ __intel_timeline_sync_set(tl, id, 0);
+ count++;
+ } while (!time_after(jiffies, end_time));
+ kt = ktime_sub(ktime_get(), kt);
+ kt = ktime_sub_ns(kt, (count * prng32_1M * 2) >> 20);
+ pr_info("%s: %lu random insertions, %lluns/insert\n",
+ __func__, count, (long long)div64_ul(ktime_to_ns(kt), count));
+
+ /* Benchmark looking up the exact same context ids as we just set */
+ prandom_seed_state(&prng, i915_selftest.random_seed);
+ end_time = count;
+ kt = ktime_get();
+ while (end_time--) {
+ u64 id = i915_prandom_u64_state(&prng);
+
+ if (!__intel_timeline_sync_is_later(tl, id, 0)) {
+ mock_timeline_destroy(tl);
+ pr_err("Lookup of %llu failed\n", id);
+ return -EINVAL;
+ }
+ }
+ kt = ktime_sub(ktime_get(), kt);
+ kt = ktime_sub_ns(kt, (count * prng32_1M * 2) >> 20);
+ pr_info("%s: %lu random lookups, %lluns/lookup\n",
+ __func__, count, (long long)div64_ul(ktime_to_ns(kt), count));
+
+ mock_timeline_destroy(tl);
+ cond_resched();
+
+ tl = mock_timeline(0);
+ if (!tl)
+ return -ENOMEM;
+
+ /* Benchmark setting the first N (in order) contexts */
+ count = 0;
+ kt = ktime_get();
+ end_time = jiffies + HZ/10;
+ do {
+ __intel_timeline_sync_set(tl, count++, 0);
+ } while (!time_after(jiffies, end_time));
+ kt = ktime_sub(ktime_get(), kt);
+ pr_info("%s: %lu in-order insertions, %lluns/insert\n",
+ __func__, count, (long long)div64_ul(ktime_to_ns(kt), count));
+
+ /* Benchmark looking up the exact same context ids as we just set */
+ end_time = count;
+ kt = ktime_get();
+ while (end_time--) {
+ if (!__intel_timeline_sync_is_later(tl, end_time, 0)) {
+ pr_err("Lookup of %lu failed\n", end_time);
+ mock_timeline_destroy(tl);
+ return -EINVAL;
+ }
+ }
+ kt = ktime_sub(ktime_get(), kt);
+ pr_info("%s: %lu in-order lookups, %lluns/lookup\n",
+ __func__, count, (long long)div64_ul(ktime_to_ns(kt), count));
+
+ mock_timeline_destroy(tl);
+ cond_resched();
+
+ tl = mock_timeline(0);
+ if (!tl)
+ return -ENOMEM;
+
+ /* Benchmark searching for a random context id and maybe changing it */
+ prandom_seed_state(&prng, i915_selftest.random_seed);
+ count = 0;
+ kt = ktime_get();
+ end_time = jiffies + HZ/10;
+ do {
+ u32 id = random_engine(&prng);
+ u32 seqno = prandom_u32_state(&prng);
+
+ if (!__intel_timeline_sync_is_later(tl, id, seqno))
+ __intel_timeline_sync_set(tl, id, seqno);
+
+ count++;
+ } while (!time_after(jiffies, end_time));
+ kt = ktime_sub(ktime_get(), kt);
+ kt = ktime_sub_ns(kt, (count * prng32_1M * 2) >> 20);
+ pr_info("%s: %lu repeated insert/lookups, %lluns/op\n",
+ __func__, count, (long long)div64_ul(ktime_to_ns(kt), count));
+ mock_timeline_destroy(tl);
+ cond_resched();
+
+ /* Benchmark searching for a known context id and changing the seqno */
+ for (last_order = 1, order = 1; order < 32;
+ ({ int tmp = last_order; last_order = order; order += tmp; })) {
+ unsigned int mask = BIT(order) - 1;
+
+ tl = mock_timeline(0);
+ if (!tl)
+ return -ENOMEM;
+
+ count = 0;
+ kt = ktime_get();
+ end_time = jiffies + HZ/10;
+ do {
+ /* Without assuming too many details of the underlying
+ * implementation, try to identify its phase-changes
+ * (if any)!
+ */
+ u64 id = (u64)(count & mask) << order;
+
+ __intel_timeline_sync_is_later(tl, id, 0);
+ __intel_timeline_sync_set(tl, id, 0);
+
+ count++;
+ } while (!time_after(jiffies, end_time));
+ kt = ktime_sub(ktime_get(), kt);
+ pr_info("%s: %lu cyclic/%d insert/lookups, %lluns/op\n",
+ __func__, count, order,
+ (long long)div64_ul(ktime_to_ns(kt), count));
+ mock_timeline_destroy(tl);
+ cond_resched();
+ }
+
+ return 0;
+}
+
+int i915_gem_timeline_mock_selftests(void)
+{
+ static const struct i915_subtest tests[] = {
+ SUBTEST(igt_sync),
+ SUBTEST(bench_sync),
+ };
+
+ return i915_subtests(tests, NULL);
+}
* Tests are executed in order by igt/drv_selftest
*/
selftest(sanitycheck, i915_mock_sanitycheck) /* keep first (igt selfcheck) */
+selftest(fence, i915_sw_fence_mock_selftests)
selftest(scatterlist, scatterlist_mock_selftests)
+selftest(syncmap, i915_syncmap_mock_selftests)
selftest(uncore, intel_uncore_mock_selftests)
selftest(breadcrumbs, intel_breadcrumbs_mock_selftests)
+selftest(timelines, i915_gem_timeline_mock_selftests)
selftest(requests, i915_gem_request_mock_selftests)
selftest(objects, i915_gem_object_mock_selftests)
selftest(dmabuf, i915_gem_dmabuf_mock_selftests)
#include "i915_random.h"
+u64 i915_prandom_u64_state(struct rnd_state *rnd)
+{
+ u64 x;
+
+ x = prandom_u32_state(rnd);
+ x <<= 32;
+ x |= prandom_u32_state(rnd);
+
+ return x;
+}
+
static inline u32 i915_prandom_u32_max_state(u32 ep_ro, struct rnd_state *state)
{
return upper_32_bits((u64)prandom_u32_state(state) * ep_ro);
#define I915_RND_SUBSTATE(name__, parent__) \
struct rnd_state name__ = I915_RND_STATE_INITIALIZER(prandom_u32_state(&(parent__)))
+u64 i915_prandom_u64_state(struct rnd_state *rnd);
+
unsigned int *i915_random_order(unsigned int count,
struct rnd_state *state);
void i915_random_reorder(unsigned int *order,
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include <linux/completion.h>
+#include <linux/delay.h>
+
+#include "../i915_selftest.h"
+
+static int __i915_sw_fence_call
+fence_notify(struct i915_sw_fence *fence, enum i915_sw_fence_notify state)
+{
+ switch (state) {
+ case FENCE_COMPLETE:
+ break;
+
+ case FENCE_FREE:
+ /* Leave the fence for the caller to free it after testing */
+ break;
+ }
+
+ return NOTIFY_DONE;
+}
+
+static struct i915_sw_fence *alloc_fence(void)
+{
+ struct i915_sw_fence *fence;
+
+ fence = kmalloc(sizeof(*fence), GFP_KERNEL);
+ if (!fence)
+ return NULL;
+
+ i915_sw_fence_init(fence, fence_notify);
+ return fence;
+}
+
+static void free_fence(struct i915_sw_fence *fence)
+{
+ i915_sw_fence_fini(fence);
+ kfree(fence);
+}
+
+static int __test_self(struct i915_sw_fence *fence)
+{
+ if (i915_sw_fence_done(fence))
+ return -EINVAL;
+
+ i915_sw_fence_commit(fence);
+ if (!i915_sw_fence_done(fence))
+ return -EINVAL;
+
+ i915_sw_fence_wait(fence);
+ if (!i915_sw_fence_done(fence))
+ return -EINVAL;
+
+ return 0;
+}
+
+static int test_self(void *arg)
+{
+ struct i915_sw_fence *fence;
+ int ret;
+
+ /* Test i915_sw_fence signaling and completion testing */
+ fence = alloc_fence();
+ if (!fence)
+ return -ENOMEM;
+
+ ret = __test_self(fence);
+
+ free_fence(fence);
+ return ret;
+}
+
+static int test_dag(void *arg)
+{
+ struct i915_sw_fence *A, *B, *C;
+ int ret = -EINVAL;
+
+ /* Test detection of cycles within the i915_sw_fence graphs */
+ if (!IS_ENABLED(CONFIG_DRM_I915_SW_FENCE_CHECK_DAG))
+ return 0;
+
+ A = alloc_fence();
+ if (!A)
+ return -ENOMEM;
+
+ if (i915_sw_fence_await_sw_fence_gfp(A, A, GFP_KERNEL) != -EINVAL) {
+ pr_err("recursive cycle not detected (AA)\n");
+ goto err_A;
+ }
+
+ B = alloc_fence();
+ if (!B) {
+ ret = -ENOMEM;
+ goto err_A;
+ }
+
+ i915_sw_fence_await_sw_fence_gfp(A, B, GFP_KERNEL);
+ if (i915_sw_fence_await_sw_fence_gfp(B, A, GFP_KERNEL) != -EINVAL) {
+ pr_err("single depth cycle not detected (BAB)\n");
+ goto err_B;
+ }
+
+ C = alloc_fence();
+ if (!C) {
+ ret = -ENOMEM;
+ goto err_B;
+ }
+
+ if (i915_sw_fence_await_sw_fence_gfp(B, C, GFP_KERNEL) == -EINVAL) {
+ pr_err("invalid cycle detected\n");
+ goto err_C;
+ }
+ if (i915_sw_fence_await_sw_fence_gfp(C, B, GFP_KERNEL) != -EINVAL) {
+ pr_err("single depth cycle not detected (CBC)\n");
+ goto err_C;
+ }
+ if (i915_sw_fence_await_sw_fence_gfp(C, A, GFP_KERNEL) != -EINVAL) {
+ pr_err("cycle not detected (BA, CB, AC)\n");
+ goto err_C;
+ }
+ if (i915_sw_fence_await_sw_fence_gfp(A, C, GFP_KERNEL) == -EINVAL) {
+ pr_err("invalid cycle detected\n");
+ goto err_C;
+ }
+
+ i915_sw_fence_commit(A);
+ i915_sw_fence_commit(B);
+ i915_sw_fence_commit(C);
+
+ ret = 0;
+ if (!i915_sw_fence_done(C)) {
+ pr_err("fence C not done\n");
+ ret = -EINVAL;
+ }
+ if (!i915_sw_fence_done(B)) {
+ pr_err("fence B not done\n");
+ ret = -EINVAL;
+ }
+ if (!i915_sw_fence_done(A)) {
+ pr_err("fence A not done\n");
+ ret = -EINVAL;
+ }
+err_C:
+ free_fence(C);
+err_B:
+ free_fence(B);
+err_A:
+ free_fence(A);
+ return ret;
+}
+
+static int test_AB(void *arg)
+{
+ struct i915_sw_fence *A, *B;
+ int ret;
+
+ /* Test i915_sw_fence (A) waiting on an event source (B) */
+ A = alloc_fence();
+ if (!A)
+ return -ENOMEM;
+ B = alloc_fence();
+ if (!B) {
+ ret = -ENOMEM;
+ goto err_A;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(A, B, GFP_KERNEL);
+ if (ret < 0)
+ goto err_B;
+ if (ret == 0) {
+ pr_err("Incorrectly reported fence A was complete before await\n");
+ ret = -EINVAL;
+ goto err_B;
+ }
+
+ ret = -EINVAL;
+ i915_sw_fence_commit(A);
+ if (i915_sw_fence_done(A))
+ goto err_B;
+
+ i915_sw_fence_commit(B);
+ if (!i915_sw_fence_done(B)) {
+ pr_err("Fence B is not done\n");
+ goto err_B;
+ }
+
+ if (!i915_sw_fence_done(A)) {
+ pr_err("Fence A is not done\n");
+ goto err_B;
+ }
+
+ ret = 0;
+err_B:
+ free_fence(B);
+err_A:
+ free_fence(A);
+ return ret;
+}
+
+static int test_ABC(void *arg)
+{
+ struct i915_sw_fence *A, *B, *C;
+ int ret;
+
+ /* Test a chain of fences, A waits on B who waits on C */
+ A = alloc_fence();
+ if (!A)
+ return -ENOMEM;
+
+ B = alloc_fence();
+ if (!B) {
+ ret = -ENOMEM;
+ goto err_A;
+ }
+
+ C = alloc_fence();
+ if (!C) {
+ ret = -ENOMEM;
+ goto err_B;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(A, B, GFP_KERNEL);
+ if (ret < 0)
+ goto err_C;
+ if (ret == 0) {
+ pr_err("Incorrectly reported fence B was complete before await\n");
+ goto err_C;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(B, C, GFP_KERNEL);
+ if (ret < 0)
+ goto err_C;
+ if (ret == 0) {
+ pr_err("Incorrectly reported fence C was complete before await\n");
+ goto err_C;
+ }
+
+ ret = -EINVAL;
+ i915_sw_fence_commit(A);
+ if (i915_sw_fence_done(A)) {
+ pr_err("Fence A completed early\n");
+ goto err_C;
+ }
+
+ i915_sw_fence_commit(B);
+ if (i915_sw_fence_done(B)) {
+ pr_err("Fence B completed early\n");
+ goto err_C;
+ }
+
+ if (i915_sw_fence_done(A)) {
+ pr_err("Fence A completed early (after signaling B)\n");
+ goto err_C;
+ }
+
+ i915_sw_fence_commit(C);
+
+ ret = 0;
+ if (!i915_sw_fence_done(C)) {
+ pr_err("Fence C not done\n");
+ ret = -EINVAL;
+ }
+ if (!i915_sw_fence_done(B)) {
+ pr_err("Fence B not done\n");
+ ret = -EINVAL;
+ }
+ if (!i915_sw_fence_done(A)) {
+ pr_err("Fence A not done\n");
+ ret = -EINVAL;
+ }
+err_C:
+ free_fence(C);
+err_B:
+ free_fence(B);
+err_A:
+ free_fence(A);
+ return ret;
+}
+
+static int test_AB_C(void *arg)
+{
+ struct i915_sw_fence *A, *B, *C;
+ int ret = -EINVAL;
+
+ /* Test multiple fences (AB) waiting on a single event (C) */
+ A = alloc_fence();
+ if (!A)
+ return -ENOMEM;
+
+ B = alloc_fence();
+ if (!B) {
+ ret = -ENOMEM;
+ goto err_A;
+ }
+
+ C = alloc_fence();
+ if (!C) {
+ ret = -ENOMEM;
+ goto err_B;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(A, C, GFP_KERNEL);
+ if (ret < 0)
+ goto err_C;
+ if (ret == 0) {
+ ret = -EINVAL;
+ goto err_C;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(B, C, GFP_KERNEL);
+ if (ret < 0)
+ goto err_C;
+ if (ret == 0) {
+ ret = -EINVAL;
+ goto err_C;
+ }
+
+ i915_sw_fence_commit(A);
+ i915_sw_fence_commit(B);
+
+ ret = 0;
+ if (i915_sw_fence_done(A)) {
+ pr_err("Fence A completed early\n");
+ ret = -EINVAL;
+ }
+
+ if (i915_sw_fence_done(B)) {
+ pr_err("Fence B completed early\n");
+ ret = -EINVAL;
+ }
+
+ i915_sw_fence_commit(C);
+ if (!i915_sw_fence_done(C)) {
+ pr_err("Fence C not done\n");
+ ret = -EINVAL;
+ }
+
+ if (!i915_sw_fence_done(B)) {
+ pr_err("Fence B not done\n");
+ ret = -EINVAL;
+ }
+
+ if (!i915_sw_fence_done(A)) {
+ pr_err("Fence A not done\n");
+ ret = -EINVAL;
+ }
+
+err_C:
+ free_fence(C);
+err_B:
+ free_fence(B);
+err_A:
+ free_fence(A);
+ return ret;
+}
+
+static int test_C_AB(void *arg)
+{
+ struct i915_sw_fence *A, *B, *C;
+ int ret;
+
+ /* Test multiple event sources (A,B) for a single fence (C) */
+ A = alloc_fence();
+ if (!A)
+ return -ENOMEM;
+
+ B = alloc_fence();
+ if (!B) {
+ ret = -ENOMEM;
+ goto err_A;
+ }
+
+ C = alloc_fence();
+ if (!C) {
+ ret = -ENOMEM;
+ goto err_B;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(C, A, GFP_KERNEL);
+ if (ret < 0)
+ goto err_C;
+ if (ret == 0) {
+ ret = -EINVAL;
+ goto err_C;
+ }
+
+ ret = i915_sw_fence_await_sw_fence_gfp(C, B, GFP_KERNEL);
+ if (ret < 0)
+ goto err_C;
+ if (ret == 0) {
+ ret = -EINVAL;
+ goto err_C;
+ }
+
+ ret = 0;
+ i915_sw_fence_commit(C);
+ if (i915_sw_fence_done(C))
+ ret = -EINVAL;
+
+ i915_sw_fence_commit(A);
+ i915_sw_fence_commit(B);
+
+ if (!i915_sw_fence_done(A)) {
+ pr_err("Fence A not done\n");
+ ret = -EINVAL;
+ }
+
+ if (!i915_sw_fence_done(B)) {
+ pr_err("Fence B not done\n");
+ ret = -EINVAL;
+ }
+
+ if (!i915_sw_fence_done(C)) {
+ pr_err("Fence C not done\n");
+ ret = -EINVAL;
+ }
+
+err_C:
+ free_fence(C);
+err_B:
+ free_fence(B);
+err_A:
+ free_fence(A);
+ return ret;
+}
+
+static int test_chain(void *arg)
+{
+ int nfences = 4096;
+ struct i915_sw_fence **fences;
+ int ret, i;
+
+ /* Test a long chain of fences */
+ fences = kmalloc_array(nfences, sizeof(*fences), GFP_KERNEL);
+ if (!fences)
+ return -ENOMEM;
+
+ for (i = 0; i < nfences; i++) {
+ fences[i] = alloc_fence();
+ if (!fences[i]) {
+ nfences = i;
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ if (i > 0) {
+ ret = i915_sw_fence_await_sw_fence_gfp(fences[i],
+ fences[i - 1],
+ GFP_KERNEL);
+ if (ret < 0) {
+ nfences = i + 1;
+ goto err;
+ }
+
+ i915_sw_fence_commit(fences[i]);
+ }
+ }
+
+ ret = 0;
+ for (i = nfences; --i; ) {
+ if (i915_sw_fence_done(fences[i])) {
+ if (ret == 0)
+ pr_err("Fence[%d] completed early\n", i);
+ ret = -EINVAL;
+ }
+ }
+ i915_sw_fence_commit(fences[0]);
+ for (i = 0; ret == 0 && i < nfences; i++) {
+ if (!i915_sw_fence_done(fences[i])) {
+ pr_err("Fence[%d] is not done\n", i);
+ ret = -EINVAL;
+ }
+ }
+
+err:
+ for (i = 0; i < nfences; i++)
+ free_fence(fences[i]);
+ kfree(fences);
+ return ret;
+}
+
+struct task_ipc {
+ struct work_struct work;
+ struct completion started;
+ struct i915_sw_fence *in, *out;
+ int value;
+};
+
+static void task_ipc(struct work_struct *work)
+{
+ struct task_ipc *ipc = container_of(work, typeof(*ipc), work);
+
+ complete(&ipc->started);
+
+ i915_sw_fence_wait(ipc->in);
+ smp_store_mb(ipc->value, 1);
+ i915_sw_fence_commit(ipc->out);
+}
+
+static int test_ipc(void *arg)
+{
+ struct task_ipc ipc;
+ int ret = 0;
+
+ /* Test use of i915_sw_fence as an interprocess signaling mechanism */
+ ipc.in = alloc_fence();
+ if (!ipc.in)
+ return -ENOMEM;
+ ipc.out = alloc_fence();
+ if (!ipc.out) {
+ ret = -ENOMEM;
+ goto err_in;
+ }
+
+ /* use a completion to avoid chicken-and-egg testing */
+ init_completion(&ipc.started);
+
+ ipc.value = 0;
+ INIT_WORK_ONSTACK(&ipc.work, task_ipc);
+ schedule_work(&ipc.work);
+
+ wait_for_completion(&ipc.started);
+
+ usleep_range(1000, 2000);
+ if (READ_ONCE(ipc.value)) {
+ pr_err("worker updated value before i915_sw_fence was signaled\n");
+ ret = -EINVAL;
+ }
+
+ i915_sw_fence_commit(ipc.in);
+ i915_sw_fence_wait(ipc.out);
+
+ if (!READ_ONCE(ipc.value)) {
+ pr_err("worker signaled i915_sw_fence before value was posted\n");
+ ret = -EINVAL;
+ }
+
+ flush_work(&ipc.work);
+ destroy_work_on_stack(&ipc.work);
+ free_fence(ipc.out);
+err_in:
+ free_fence(ipc.in);
+ return ret;
+}
+
+int i915_sw_fence_mock_selftests(void)
+{
+ static const struct i915_subtest tests[] = {
+ SUBTEST(test_self),
+ SUBTEST(test_dag),
+ SUBTEST(test_AB),
+ SUBTEST(test_ABC),
+ SUBTEST(test_AB_C),
+ SUBTEST(test_C_AB),
+ SUBTEST(test_chain),
+ SUBTEST(test_ipc),
+ };
+
+ return i915_subtests(tests, NULL);
+}
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "../i915_selftest.h"
+#include "i915_random.h"
+
+static char *
+__sync_print(struct i915_syncmap *p,
+ char *buf, unsigned long *sz,
+ unsigned int depth,
+ unsigned int last,
+ unsigned int idx)
+{
+ unsigned long len;
+ unsigned int i, X;
+
+ if (depth) {
+ unsigned int d;
+
+ for (d = 0; d < depth - 1; d++) {
+ if (last & BIT(depth - d - 1))
+ len = scnprintf(buf, *sz, "| ");
+ else
+ len = scnprintf(buf, *sz, " ");
+ buf += len;
+ *sz -= len;
+ }
+ len = scnprintf(buf, *sz, "%x-> ", idx);
+ buf += len;
+ *sz -= len;
+ }
+
+ /* We mark bits after the prefix as "X" */
+ len = scnprintf(buf, *sz, "0x%016llx", p->prefix << p->height << SHIFT);
+ buf += len;
+ *sz -= len;
+ X = (p->height + SHIFT) / 4;
+ scnprintf(buf - X, *sz + X, "%*s", X, "XXXXXXXXXXXXXXXXX");
+
+ if (!p->height) {
+ for_each_set_bit(i, (unsigned long *)&p->bitmap, KSYNCMAP) {
+ len = scnprintf(buf, *sz, " %x:%x,",
+ i, __sync_seqno(p)[i]);
+ buf += len;
+ *sz -= len;
+ }
+ buf -= 1;
+ *sz += 1;
+ }
+
+ len = scnprintf(buf, *sz, "\n");
+ buf += len;
+ *sz -= len;
+
+ if (p->height) {
+ for_each_set_bit(i, (unsigned long *)&p->bitmap, KSYNCMAP) {
+ buf = __sync_print(__sync_child(p)[i], buf, sz,
+ depth + 1,
+ last << 1 | !!(p->bitmap >> (i + 1)),
+ i);
+ }
+ }
+
+ return buf;
+}
+
+static bool
+i915_syncmap_print_to_buf(struct i915_syncmap *p, char *buf, unsigned long sz)
+{
+ if (!p)
+ return false;
+
+ while (p->parent)
+ p = p->parent;
+
+ __sync_print(p, buf, &sz, 0, 1, 0);
+ return true;
+}
+
+static int check_syncmap_free(struct i915_syncmap **sync)
+{
+ i915_syncmap_free(sync);
+ if (*sync) {
+ pr_err("sync not cleared after free\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int dump_syncmap(struct i915_syncmap *sync, int err)
+{
+ char *buf;
+
+ if (!err)
+ return check_syncmap_free(&sync);
+
+ buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!buf)
+ goto skip;
+
+ if (i915_syncmap_print_to_buf(sync, buf, PAGE_SIZE))
+ pr_err("%s", buf);
+
+ kfree(buf);
+
+skip:
+ i915_syncmap_free(&sync);
+ return err;
+}
+
+static int igt_syncmap_init(void *arg)
+{
+ struct i915_syncmap *sync = (void *)~0ul;
+
+ /*
+ * Cursory check that we can initialise a random pointer and transform
+ * it into the root pointer of a syncmap.
+ */
+
+ i915_syncmap_init(&sync);
+ return check_syncmap_free(&sync);
+}
+
+static int check_seqno(struct i915_syncmap *leaf, unsigned int idx, u32 seqno)
+{
+ if (leaf->height) {
+ pr_err("%s: not a leaf, height is %d\n",
+ __func__, leaf->height);
+ return -EINVAL;
+ }
+
+ if (__sync_seqno(leaf)[idx] != seqno) {
+ pr_err("%s: seqno[%d], found %x, expected %x\n",
+ __func__, idx, __sync_seqno(leaf)[idx], seqno);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int check_one(struct i915_syncmap **sync, u64 context, u32 seqno)
+{
+ int err;
+
+ err = i915_syncmap_set(sync, context, seqno);
+ if (err)
+ return err;
+
+ if ((*sync)->height) {
+ pr_err("Inserting first context=%llx did not return leaf (height=%d, prefix=%llx\n",
+ context, (*sync)->height, (*sync)->prefix);
+ return -EINVAL;
+ }
+
+ if ((*sync)->parent) {
+ pr_err("Inserting first context=%llx created branches!\n",
+ context);
+ return -EINVAL;
+ }
+
+ if (hweight32((*sync)->bitmap) != 1) {
+ pr_err("First bitmap does not contain a single entry, found %x (count=%d)!\n",
+ (*sync)->bitmap, hweight32((*sync)->bitmap));
+ return -EINVAL;
+ }
+
+ err = check_seqno((*sync), ilog2((*sync)->bitmap), seqno);
+ if (err)
+ return err;
+
+ if (!i915_syncmap_is_later(sync, context, seqno)) {
+ pr_err("Lookup of first context=%llx/seqno=%x failed!\n",
+ context, seqno);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int igt_syncmap_one(void *arg)
+{
+ I915_RND_STATE(prng);
+ IGT_TIMEOUT(end_time);
+ struct i915_syncmap *sync;
+ unsigned long max = 1;
+ int err;
+
+ /*
+ * Check that inserting a new id, creates a leaf and only that leaf.
+ */
+
+ i915_syncmap_init(&sync);
+
+ do {
+ u64 context = i915_prandom_u64_state(&prng);
+ unsigned long loop;
+
+ err = check_syncmap_free(&sync);
+ if (err)
+ goto out;
+
+ for (loop = 0; loop <= max; loop++) {
+ err = check_one(&sync, context,
+ prandom_u32_state(&prng));
+ if (err)
+ goto out;
+ }
+ max++;
+ } while (!__igt_timeout(end_time, NULL));
+ pr_debug("%s: Completed %lu single insertions\n",
+ __func__, max * (max - 1) / 2);
+out:
+ return dump_syncmap(sync, err);
+}
+
+static int check_leaf(struct i915_syncmap **sync, u64 context, u32 seqno)
+{
+ int err;
+
+ err = i915_syncmap_set(sync, context, seqno);
+ if (err)
+ return err;
+
+ if ((*sync)->height) {
+ pr_err("Inserting context=%llx did not return leaf (height=%d, prefix=%llx\n",
+ context, (*sync)->height, (*sync)->prefix);
+ return -EINVAL;
+ }
+
+ if (hweight32((*sync)->bitmap) != 1) {
+ pr_err("First entry into leaf (context=%llx) does not contain a single entry, found %x (count=%d)!\n",
+ context, (*sync)->bitmap, hweight32((*sync)->bitmap));
+ return -EINVAL;
+ }
+
+ err = check_seqno((*sync), ilog2((*sync)->bitmap), seqno);
+ if (err)
+ return err;
+
+ if (!i915_syncmap_is_later(sync, context, seqno)) {
+ pr_err("Lookup of first entry context=%llx/seqno=%x failed!\n",
+ context, seqno);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int igt_syncmap_join_above(void *arg)
+{
+ struct i915_syncmap *sync;
+ unsigned int pass, order;
+ int err;
+
+ i915_syncmap_init(&sync);
+
+ /*
+ * When we have a new id that doesn't fit inside the existing tree,
+ * we need to add a new layer above.
+ *
+ * 1: 0x00000001
+ * 2: 0x00000010
+ * 3: 0x00000100
+ * 4: 0x00001000
+ * ...
+ * Each pass the common prefix shrinks and we have to insert a join.
+ * Each join will only contain two branches, the latest of which
+ * is always a leaf.
+ *
+ * If we then reuse the same set of contexts, we expect to build an
+ * identical tree.
+ */
+ for (pass = 0; pass < 3; pass++) {
+ for (order = 0; order < 64; order += SHIFT) {
+ u64 context = BIT_ULL(order);
+ struct i915_syncmap *join;
+
+ err = check_leaf(&sync, context, 0);
+ if (err)
+ goto out;
+
+ join = sync->parent;
+ if (!join) /* very first insert will have no parents */
+ continue;
+
+ if (!join->height) {
+ pr_err("Parent with no height!\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (hweight32(join->bitmap) != 2) {
+ pr_err("Join does not have 2 children: %x (%d)\n",
+ join->bitmap, hweight32(join->bitmap));
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (__sync_child(join)[__sync_branch_idx(join, context)] != sync) {
+ pr_err("Leaf misplaced in parent!\n");
+ err = -EINVAL;
+ goto out;
+ }
+ }
+ }
+out:
+ return dump_syncmap(sync, err);
+}
+
+static int igt_syncmap_join_below(void *arg)
+{
+ struct i915_syncmap *sync;
+ unsigned int step, order, idx;
+ int err;
+
+ i915_syncmap_init(&sync);
+
+ /*
+ * Check that we can split a compacted branch by replacing it with
+ * a join.
+ */
+ for (step = 0; step < KSYNCMAP; step++) {
+ for (order = 64 - SHIFT; order > 0; order -= SHIFT) {
+ u64 context = step * BIT_ULL(order);
+
+ err = i915_syncmap_set(&sync, context, 0);
+ if (err)
+ goto out;
+
+ if (sync->height) {
+ pr_err("Inserting context=%llx (order=%d, step=%d) did not return leaf (height=%d, prefix=%llx\n",
+ context, order, step, sync->height, sync->prefix);
+ err = -EINVAL;
+ goto out;
+ }
+ }
+ }
+
+ for (step = 0; step < KSYNCMAP; step++) {
+ for (order = SHIFT; order < 64; order += SHIFT) {
+ u64 context = step * BIT_ULL(order);
+
+ if (!i915_syncmap_is_later(&sync, context, 0)) {
+ pr_err("1: context %llx (order=%d, step=%d) not found\n",
+ context, order, step);
+ err = -EINVAL;
+ goto out;
+ }
+
+ for (idx = 1; idx < KSYNCMAP; idx++) {
+ if (i915_syncmap_is_later(&sync, context + idx, 0)) {
+ pr_err("1: context %llx (order=%d, step=%d) should not exist\n",
+ context + idx, order, step);
+ err = -EINVAL;
+ goto out;
+ }
+ }
+ }
+ }
+
+ for (order = SHIFT; order < 64; order += SHIFT) {
+ for (step = 0; step < KSYNCMAP; step++) {
+ u64 context = step * BIT_ULL(order);
+
+ if (!i915_syncmap_is_later(&sync, context, 0)) {
+ pr_err("2: context %llx (order=%d, step=%d) not found\n",
+ context, order, step);
+ err = -EINVAL;
+ goto out;
+ }
+ }
+ }
+
+out:
+ return dump_syncmap(sync, err);
+}
+
+static int igt_syncmap_neighbours(void *arg)
+{
+ I915_RND_STATE(prng);
+ IGT_TIMEOUT(end_time);
+ struct i915_syncmap *sync;
+ int err;
+
+ /*
+ * Each leaf holds KSYNCMAP seqno. Check that when we create KSYNCMAP
+ * neighbouring ids, they all fit into the same leaf.
+ */
+
+ i915_syncmap_init(&sync);
+ do {
+ u64 context = i915_prandom_u64_state(&prng) & ~MASK;
+ unsigned int idx;
+
+ if (i915_syncmap_is_later(&sync, context, 0)) /* Skip repeats */
+ continue;
+
+ for (idx = 0; idx < KSYNCMAP; idx++) {
+ err = i915_syncmap_set(&sync, context + idx, 0);
+ if (err)
+ goto out;
+
+ if (sync->height) {
+ pr_err("Inserting context=%llx did not return leaf (height=%d, prefix=%llx\n",
+ context, sync->height, sync->prefix);
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (sync->bitmap != BIT(idx + 1) - 1) {
+ pr_err("Inserting neighbouring context=0x%llx+%d, did not fit into the same leaf bitmap=%x (%d), expected %lx (%d)\n",
+ context, idx,
+ sync->bitmap, hweight32(sync->bitmap),
+ BIT(idx + 1) - 1, idx + 1);
+ err = -EINVAL;
+ goto out;
+ }
+ }
+ } while (!__igt_timeout(end_time, NULL));
+out:
+ return dump_syncmap(sync, err);
+}
+
+static int igt_syncmap_compact(void *arg)
+{
+ struct i915_syncmap *sync;
+ unsigned int idx, order;
+ int err;
+
+ i915_syncmap_init(&sync);
+
+ /*
+ * The syncmap are "space efficient" compressed radix trees - any
+ * branch with only one child is skipped and replaced by the child.
+ *
+ * If we construct a tree with ids that are neighbouring at a non-zero
+ * height, we form a join but each child of that join is directly a
+ * leaf holding the single id.
+ */
+ for (order = SHIFT; order < 64; order += SHIFT) {
+ err = check_syncmap_free(&sync);
+ if (err)
+ goto out;
+
+ /* Create neighbours in the parent */
+ for (idx = 0; idx < KSYNCMAP; idx++) {
+ u64 context = idx * BIT_ULL(order) + idx;
+
+ err = i915_syncmap_set(&sync, context, 0);
+ if (err)
+ goto out;
+
+ if (sync->height) {
+ pr_err("Inserting context=%llx (order=%d, idx=%d) did not return leaf (height=%d, prefix=%llx\n",
+ context, order, idx,
+ sync->height, sync->prefix);
+ err = -EINVAL;
+ goto out;
+ }
+ }
+
+ sync = sync->parent;
+ if (sync->parent) {
+ pr_err("Parent (join) of last leaf was not the sync!\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (sync->height != order) {
+ pr_err("Join does not have the expected height, found %d, expected %d\n",
+ sync->height, order);
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (sync->bitmap != BIT(KSYNCMAP) - 1) {
+ pr_err("Join is not full!, found %x (%d) expected %lx (%d)\n",
+ sync->bitmap, hweight32(sync->bitmap),
+ BIT(KSYNCMAP) - 1, KSYNCMAP);
+ err = -EINVAL;
+ goto out;
+ }
+
+ /* Each of our children should be a leaf */
+ for (idx = 0; idx < KSYNCMAP; idx++) {
+ struct i915_syncmap *leaf = __sync_child(sync)[idx];
+
+ if (leaf->height) {
+ pr_err("Child %d is a not leaf!\n", idx);
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (leaf->parent != sync) {
+ pr_err("Child %d is not attached to us!\n",
+ idx);
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (!is_power_of_2(leaf->bitmap)) {
+ pr_err("Child %d holds more than one id, found %x (%d)\n",
+ idx, leaf->bitmap, hweight32(leaf->bitmap));
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (leaf->bitmap != BIT(idx)) {
+ pr_err("Child %d has wrong seqno idx, found %d, expected %d\n",
+ idx, ilog2(leaf->bitmap), idx);
+ err = -EINVAL;
+ goto out;
+ }
+ }
+ }
+out:
+ return dump_syncmap(sync, err);
+}
+
+static int igt_syncmap_random(void *arg)
+{
+ I915_RND_STATE(prng);
+ IGT_TIMEOUT(end_time);
+ struct i915_syncmap *sync;
+ unsigned long count, phase, i;
+ u32 seqno;
+ int err;
+
+ i915_syncmap_init(&sync);
+
+ /*
+ * Having tried to test the individual operations within i915_syncmap,
+ * run a smoketest exploring the entire u64 space with random
+ * insertions.
+ */
+
+ count = 0;
+ phase = jiffies + HZ/100 + 1;
+ do {
+ u64 context = i915_prandom_u64_state(&prng);
+
+ err = i915_syncmap_set(&sync, context, 0);
+ if (err)
+ goto out;
+
+ count++;
+ } while (!time_after(jiffies, phase));
+ seqno = 0;
+
+ phase = 0;
+ do {
+ I915_RND_STATE(ctx);
+ u32 last_seqno = seqno;
+ bool expect;
+
+ seqno = prandom_u32_state(&prng);
+ expect = seqno_later(last_seqno, seqno);
+
+ for (i = 0; i < count; i++) {
+ u64 context = i915_prandom_u64_state(&ctx);
+
+ if (i915_syncmap_is_later(&sync, context, seqno) != expect) {
+ pr_err("context=%llu, last=%u this=%u did not match expectation (%d)\n",
+ context, last_seqno, seqno, expect);
+ err = -EINVAL;
+ goto out;
+ }
+
+ err = i915_syncmap_set(&sync, context, seqno);
+ if (err)
+ goto out;
+ }
+
+ phase++;
+ } while (!__igt_timeout(end_time, NULL));
+ pr_debug("Completed %lu passes, each of %lu contexts\n", phase, count);
+out:
+ return dump_syncmap(sync, err);
+}
+
+int i915_syncmap_mock_selftests(void)
+{
+ static const struct i915_subtest tests[] = {
+ SUBTEST(igt_syncmap_init),
+ SUBTEST(igt_syncmap_one),
+ SUBTEST(igt_syncmap_join_above),
+ SUBTEST(igt_syncmap_join_below),
+ SUBTEST(igt_syncmap_neighbours),
+ SUBTEST(igt_syncmap_compact),
+ SUBTEST(igt_syncmap_random),
+ };
+
+ return i915_subtests(tests, NULL);
+}
spin_unlock(&engine->hw_lock);
}
-static int mock_context_pin(struct intel_engine_cs *engine,
- struct i915_gem_context *ctx)
+static struct intel_ring *
+mock_context_pin(struct intel_engine_cs *engine,
+ struct i915_gem_context *ctx)
{
i915_gem_context_get(ctx);
- return 0;
+ return engine->buffer;
}
static void mock_context_unpin(struct intel_engine_cs *engine,
INIT_LIST_HEAD(&mock->link);
mock->delay = 0;
- request->ring = request->engine->buffer;
return 0;
}
if (!ring)
return NULL;
- ring->engine = engine;
ring->size = sz;
ring->effective_size = sz;
ring->vaddr = (void *)(ring + 1);
/* minimal engine setup for requests */
engine->base.i915 = i915;
- engine->base.name = name;
+ snprintf(engine->base.name, sizeof(engine->base.name), "%s", name);
engine->base.id = id++;
engine->base.status_page.page_addr = (void *)(engine + 1);
#include "mock_gem_device.h"
#include "mock_gem_object.h"
#include "mock_gtt.h"
+#include "mock_uncore.h"
void mock_device_flush(struct drm_i915_private *i915)
{
destroy_workqueue(i915->wq);
+ kmem_cache_destroy(i915->priorities);
kmem_cache_destroy(i915->dependencies);
kmem_cache_destroy(i915->requests);
kmem_cache_destroy(i915->vmas);
goto err;
device_initialize(&pdev->dev);
+ pdev->class = PCI_BASE_CLASS_DISPLAY << 16;
pdev->dev.release = release_dev;
dev_set_name(&pdev->dev, "mock");
dma_coerce_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
mkwrite_device_info(i915)->gen = -1;
spin_lock_init(&i915->mm.object_stat_lock);
+ mock_uncore_init(i915);
init_waitqueue_head(&i915->gpu_error.wait_queue);
init_waitqueue_head(&i915->gpu_error.reset_queue);
if (!i915->dependencies)
goto err_requests;
+ i915->priorities = KMEM_CACHE(i915_priolist, SLAB_HWCACHE_ALIGN);
+ if (!i915->priorities)
+ goto err_dependencies;
+
mutex_lock(&i915->drm.struct_mutex);
INIT_LIST_HEAD(&i915->gt.timelines);
err = i915_gem_timeline_init__global(i915);
if (err) {
mutex_unlock(&i915->drm.struct_mutex);
- goto err_dependencies;
+ goto err_priorities;
}
mock_init_ggtt(i915);
err_engine:
for_each_engine(engine, i915, id)
mock_engine_free(engine);
+err_priorities:
+ kmem_cache_destroy(i915->priorities);
err_dependencies:
kmem_cache_destroy(i915->dependencies);
err_requests:
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "mock_timeline.h"
+
+struct intel_timeline *mock_timeline(u64 context)
+{
+ static struct lock_class_key class;
+ struct intel_timeline *tl;
+
+ tl = kzalloc(sizeof(*tl), GFP_KERNEL);
+ if (!tl)
+ return NULL;
+
+ __intel_timeline_init(tl, NULL, context, &class, "mock");
+
+ return tl;
+}
+
+void mock_timeline_destroy(struct intel_timeline *tl)
+{
+ __intel_timeline_fini(tl);
+ kfree(tl);
+}
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __MOCK_TIMELINE__
+#define __MOCK_TIMELINE__
+
+#include "../i915_gem_timeline.h"
+
+struct intel_timeline *mock_timeline(u64 context);
+void mock_timeline_destroy(struct intel_timeline *tl);
+
+#endif /* !__MOCK_TIMELINE__ */
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "mock_uncore.h"
+
+#define __nop_write(x) \
+static void \
+nop_write##x(struct drm_i915_private *dev_priv, i915_reg_t reg, u##x val, bool trace) { }
+__nop_write(8)
+__nop_write(16)
+__nop_write(32)
+
+#define __nop_read(x) \
+static u##x \
+nop_read##x(struct drm_i915_private *dev_priv, i915_reg_t reg, bool trace) { return 0; }
+__nop_read(8)
+__nop_read(16)
+__nop_read(32)
+__nop_read(64)
+
+void mock_uncore_init(struct drm_i915_private *i915)
+{
+ ASSIGN_WRITE_MMIO_VFUNCS(i915, nop);
+ ASSIGN_READ_MMIO_VFUNCS(i915, nop);
+}
--- /dev/null
+/*
+ * Copyright © 2017 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef __MOCK_UNCORE_H
+#define __MOCK_UNCORE_H
+
+void mock_uncore_init(struct drm_i915_private *i915);
+
+#endif /* !__MOCK_UNCORE_H */
pm_runtime_get_sync(drm->dev->dev);
drm_helper_hpd_irq_event(drm->dev);
+ /* enable polling for external displays */
+ drm_kms_helper_poll_enable(drm->dev);
pm_runtime_mark_last_busy(drm->dev->dev);
pm_runtime_put_sync(drm->dev->dev);
if (ret)
return ret;
- /* enable polling for external displays */
- if (!dev->mode_config.poll_enabled)
- drm_kms_helper_poll_enable(dev);
-
/* enable hotplug interrupts */
list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
struct nouveau_connector *conn = nouveau_connector(connector);
pm_runtime_allow(dev->dev);
pm_runtime_mark_last_busy(dev->dev);
pm_runtime_put(dev->dev);
+ } else {
+ /* enable polling for external displays */
+ drm_kms_helper_poll_enable(dev);
}
return 0;
ret = nouveau_do_resume(drm_dev, true);
- if (!drm_dev->mode_config.poll_enabled)
- drm_kms_helper_poll_enable(drm_dev);
-
/* do magic */
nvif_mask(&device->object, 0x088488, (1 << 25), (1 << 25));
vga_switcheroo_set_dynamic_switch(pdev, VGA_SWITCHEROO_ON);
case NVKM_MEM_TARGET_NCOH: target = 3; break;
default:
WARN_ON(1);
- return;
+ goto unlock;
}
nvkm_wr32(device, 0x002270, (nvkm_memory_addr(mem) >> 12) |
& 0x00100000),
msecs_to_jiffies(2000)) == 0)
nvkm_error(subdev, "runlist %d update timeout\n", runl);
+unlock:
mutex_unlock(&subdev->mutex);
}
ret = nvkm_firmware_get(subdev->device, f, &sig);
if (ret)
goto free_data;
+
img->sig = kmemdup(sig->data, sig->size, GFP_KERNEL);
if (!img->sig) {
ret = -ENOMEM;
img->ucode_data = ls_ucode_img_build(bl, code, data,
&img->ucode_desc);
if (IS_ERR(img->ucode_data)) {
+ kfree(img->sig);
ret = PTR_ERR(img->ucode_data);
- goto free_data;
+ goto free_sig;
}
img->ucode_size = img->ucode_desc.image_size;
#include <linux/platform_device.h>
#include <linux/of.h>
-#include <video/omap-panel-data.h>
-
#include "../dss/omapdss.h"
struct panel_drv_data {
struct device *dev;
struct videomode vm;
-
- bool invert_polarity;
};
static const struct videomode tvc_pal_vm = {
in->ops.atv->set_timings(in, &ddata->vm);
- if (!ddata->dev->of_node) {
- in->ops.atv->set_type(in, OMAP_DSS_VENC_TYPE_COMPOSITE);
-
- in->ops.atv->invert_vid_out_polarity(in,
- ddata->invert_polarity);
- }
-
r = in->ops.atv->enable(in);
if (r)
return r;
.get_timings = tvc_get_timings,
.check_timings = tvc_check_timings,
- .get_resolution = omapdss_default_get_resolution,
-
.get_wss = tvc_get_wss,
.set_wss = tvc_set_wss,
};
-static int tvc_probe_pdata(struct platform_device *pdev)
-{
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct connector_atv_platform_data *pdata;
- struct omap_dss_device *in, *dssdev;
-
- pdata = dev_get_platdata(&pdev->dev);
-
- in = omap_dss_find_output(pdata->source);
- if (in == NULL) {
- dev_err(&pdev->dev, "Failed to find video source\n");
- return -EPROBE_DEFER;
- }
-
- ddata->in = in;
-
- ddata->invert_polarity = pdata->invert_polarity;
-
- dssdev = &ddata->dssdev;
- dssdev->name = pdata->name;
-
- return 0;
-}
-
static int tvc_probe_of(struct platform_device *pdev)
{
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
platform_set_drvdata(pdev, ddata);
ddata->dev = &pdev->dev;
- if (dev_get_platdata(&pdev->dev)) {
- r = tvc_probe_pdata(pdev);
- if (r)
- return r;
- } else if (pdev->dev.of_node) {
- r = tvc_probe_of(pdev);
- if (r)
- return r;
- } else {
- return -ENODEV;
- }
+ r = tvc_probe_of(pdev);
+ if (r)
+ return r;
ddata->vm = tvc_pal_vm;
#include <linux/slab.h>
#include <drm/drm_edid.h>
-#include <video/omap-panel-data.h>
#include "../dss/omapdss.h"
.get_timings = dvic_get_timings,
.check_timings = dvic_check_timings,
- .get_resolution = omapdss_default_get_resolution,
-
.read_edid = dvic_read_edid,
.detect = dvic_detect,
};
#include <linux/of_gpio.h>
#include <drm/drm_edid.h>
-#include <video/omap-panel-data.h>
#include "../dss/omapdss.h"
.get_timings = hdmic_get_timings,
.check_timings = hdmic_check_timings,
- .get_resolution = omapdss_default_get_resolution,
-
.read_edid = hdmic_read_edid,
.detect = hdmic_detect,
.set_hdmi_mode = hdmic_set_hdmi_mode,
return in->ops.atv->check_timings(in, vm);
}
-static void opa362_set_type(struct omap_dss_device *dssdev,
- enum omap_dss_venc_type type)
-{
- /* we can only drive a COMPOSITE output */
- WARN_ON(type != OMAP_DSS_VENC_TYPE_COMPOSITE);
-
-}
-
static const struct omapdss_atv_ops opa362_atv_ops = {
.connect = opa362_connect,
.disconnect = opa362_disconnect,
.check_timings = opa362_check_timings,
.set_timings = opa362_set_timings,
.get_timings = opa362_get_timings,
-
- .set_type = opa362_set_type,
};
static int opa362_probe(struct platform_device *pdev)
struct omap_dss_device *in;
int pd_gpio;
- int data_lines;
struct videomode vm;
};
return 0;
in->ops.dpi->set_timings(in, &ddata->vm);
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
r = in->ops.dpi->enable(in);
if (r)
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
dssdev->output_type = OMAP_DISPLAY_TYPE_DVI;
dssdev->owner = THIS_MODULE;
- dssdev->phy.dpi.data_lines = ddata->data_lines;
dssdev->port_num = 1;
r = omapdss_register_output(dssdev);
#include <linux/platform_device.h>
#include <linux/slab.h>
#include <linux/of.h>
-#include <linux/of_gpio.h>
#include <linux/regulator/consumer.h>
#include <linux/backlight.h>
-#include <video/omap-panel-data.h>
#include <video/of_display_timing.h>
#include "../dss/omapdss.h"
struct omap_dss_device dssdev;
struct omap_dss_device *in;
- int data_lines;
-
struct videomode vm;
struct backlight_device *backlight;
- /* used for non-DT boot, to be removed */
- int backlight_gpio;
-
struct gpio_desc *enable_gpio;
struct regulator *vcc_supply;
};
if (omapdss_device_is_enabled(dssdev))
return 0;
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
in->ops.dpi->set_timings(in, &ddata->vm);
r = in->ops.dpi->enable(in);
gpiod_set_value_cansleep(ddata->enable_gpio, 1);
- if (gpio_is_valid(ddata->backlight_gpio))
- gpio_set_value_cansleep(ddata->backlight_gpio, 1);
-
if (ddata->backlight) {
ddata->backlight->props.power = FB_BLANK_UNBLANK;
backlight_update_status(ddata->backlight);
if (!omapdss_device_is_enabled(dssdev))
return;
- if (gpio_is_valid(ddata->backlight_gpio))
- gpio_set_value_cansleep(ddata->backlight_gpio, 0);
-
if (ddata->backlight) {
ddata->backlight->props.power = FB_BLANK_POWERDOWN;
backlight_update_status(ddata->backlight);
.set_timings = panel_dpi_set_timings,
.get_timings = panel_dpi_get_timings,
.check_timings = panel_dpi_check_timings,
-
- .get_resolution = omapdss_default_get_resolution,
};
-static int panel_dpi_probe_pdata(struct platform_device *pdev)
-{
- const struct panel_dpi_platform_data *pdata;
- struct panel_drv_data *ddata = platform_get_drvdata(pdev);
- struct omap_dss_device *dssdev, *in;
- int r;
-
- pdata = dev_get_platdata(&pdev->dev);
-
- in = omap_dss_find_output(pdata->source);
- if (in == NULL) {
- dev_err(&pdev->dev, "failed to find video source '%s'\n",
- pdata->source);
- return -EPROBE_DEFER;
- }
-
- ddata->in = in;
-
- ddata->data_lines = pdata->data_lines;
-
- videomode_from_timing(pdata->display_timing, &ddata->vm);
-
- dssdev = &ddata->dssdev;
- dssdev->name = pdata->name;
-
- r = devm_gpio_request_one(&pdev->dev, pdata->enable_gpio,
- GPIOF_OUT_INIT_LOW, "panel enable");
- if (r)
- goto err_gpio;
-
- ddata->enable_gpio = gpio_to_desc(pdata->enable_gpio);
-
- ddata->backlight_gpio = pdata->backlight_gpio;
-
- return 0;
-
-err_gpio:
- omap_dss_put_device(ddata->in);
- return r;
-}
-
static int panel_dpi_probe_of(struct platform_device *pdev)
{
struct panel_drv_data *ddata = platform_get_drvdata(pdev);
if (IS_ERR(ddata->vcc_supply))
return PTR_ERR(ddata->vcc_supply);
- ddata->backlight_gpio = -ENOENT;
-
bl_node = of_parse_phandle(node, "backlight", 0);
if (bl_node) {
ddata->backlight = of_find_backlight_by_node(bl_node);
platform_set_drvdata(pdev, ddata);
- if (dev_get_platdata(&pdev->dev)) {
- r = panel_dpi_probe_pdata(pdev);
- if (r)
- return r;
- } else if (pdev->dev.of_node) {
- r = panel_dpi_probe_of(pdev);
- if (r)
- return r;
- } else {
- return -ENODEV;
- }
-
- if (gpio_is_valid(ddata->backlight_gpio)) {
- r = devm_gpio_request_one(&pdev->dev, ddata->backlight_gpio,
- GPIOF_OUT_INIT_LOW, "panel backlight");
- if (r)
- goto err_gpio;
- }
+ r = panel_dpi_probe_of(pdev);
+ if (r)
+ return r;
dssdev = &ddata->dssdev;
dssdev->dev = &pdev->dev;
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
dssdev->owner = THIS_MODULE;
dssdev->panel.vm = ddata->vm;
- dssdev->phy.dpi.data_lines = ddata->data_lines;
r = omapdss_register_display(dssdev);
if (r) {
return 0;
err_reg:
-err_gpio:
omap_dss_put_device(ddata->in);
return r;
}
.update_status = dsicm_bl_update_status,
};
-static void dsicm_get_resolution(struct omap_dss_device *dssdev,
- u16 *xres, u16 *yres)
-{
- *xres = dssdev->panel.vm.hactive;
- *yres = dssdev->panel.vm.vactive;
-}
-
static ssize_t dsicm_num_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
.update = dsicm_update,
.sync = dsicm_sync,
- .get_resolution = dsicm_get_resolution,
- .get_recommended_bpp = omapdss_default_get_recommended_bpp,
-
.enable_te = dsicm_enable_te,
.get_te = dsicm_get_te,
struct spi_device *spi;
- int data_lines;
-
struct videomode vm;
struct gpio_desc *enable_gpio;
if (omapdss_device_is_enabled(dssdev))
return 0;
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
in->ops.dpi->set_timings(in, &ddata->vm);
r = in->ops.dpi->enable(in);
.set_timings = lb035q02_set_timings,
.get_timings = lb035q02_get_timings,
.check_timings = lb035q02_check_timings,
-
- .get_resolution = omapdss_default_get_resolution,
};
static int lb035q02_probe_of(struct spi_device *spi)
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
dssdev->owner = THIS_MODULE;
dssdev->panel.vm = ddata->vm;
- dssdev->phy.dpi.data_lines = ddata->data_lines;
r = omapdss_register_display(dssdev);
if (r) {
struct videomode vm;
- int data_lines;
-
int res_gpio;
int qvga_gpio;
if (omapdss_device_is_enabled(dssdev))
return 0;
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
in->ops.dpi->set_timings(in, &ddata->vm);
r = in->ops.dpi->enable(in);
.set_timings = nec_8048_set_timings,
.get_timings = nec_8048_get_timings,
.check_timings = nec_8048_check_timings,
-
- .get_resolution = omapdss_default_get_resolution,
};
static int nec_8048_probe_of(struct spi_device *spi)
struct omap_dss_device *in;
struct regulator *vcc;
- int data_lines;
-
struct videomode vm;
struct gpio_desc *resb_gpio; /* low = reset active min 20 us */
if (omapdss_device_is_enabled(dssdev))
return 0;
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
in->ops.dpi->set_timings(in, &ddata->vm);
if (ddata->vcc) {
.set_timings = sharp_ls_set_timings,
.get_timings = sharp_ls_get_timings,
.check_timings = sharp_ls_check_timings,
-
- .get_resolution = omapdss_default_get_resolution,
};
static int sharp_ls_get_gpio_of(struct device *dev, int index, int val,
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
dssdev->owner = THIS_MODULE;
dssdev->panel.vm = ddata->vm;
- dssdev->phy.dpi.data_lines = ddata->data_lines;
r = omapdss_register_display(dssdev);
if (r) {
#include <linux/of.h>
#include <linux/of_gpio.h>
-#include <video/omap-panel-data.h>
-
#include "../dss/omapdss.h"
#define MIPID_CMD_READ_DISP_ID 0x04
struct omap_dss_device *in;
int reset_gpio;
- int datapairs;
struct videomode vm;
in->ops.sdi->set_timings(in, &ddata->vm);
- if (ddata->datapairs > 0)
- in->ops.sdi->set_datapairs(in, ddata->datapairs);
-
r = in->ops.sdi->enable(in);
if (r) {
pr_err("%s sdi enable failed\n", __func__);
.set_timings = acx565akm_set_timings,
.get_timings = acx565akm_get_timings,
.check_timings = acx565akm_check_timings,
-
- .get_resolution = omapdss_default_get_resolution,
};
-static int acx565akm_probe_pdata(struct spi_device *spi)
-{
- const struct panel_acx565akm_platform_data *pdata;
- struct panel_drv_data *ddata = dev_get_drvdata(&spi->dev);
- struct omap_dss_device *dssdev, *in;
-
- pdata = dev_get_platdata(&spi->dev);
-
- ddata->reset_gpio = pdata->reset_gpio;
-
- in = omap_dss_find_output(pdata->source);
- if (in == NULL) {
- dev_err(&spi->dev, "failed to find video source '%s'\n",
- pdata->source);
- return -EPROBE_DEFER;
- }
- ddata->in = in;
-
- ddata->datapairs = pdata->datapairs;
-
- dssdev = &ddata->dssdev;
- dssdev->name = pdata->name;
-
- return 0;
-}
-
static int acx565akm_probe_of(struct spi_device *spi)
{
struct panel_drv_data *ddata = dev_get_drvdata(&spi->dev);
mutex_init(&ddata->mutex);
- if (dev_get_platdata(&spi->dev)) {
- r = acx565akm_probe_pdata(spi);
- if (r)
- return r;
- } else if (spi->dev.of_node) {
- r = acx565akm_probe_of(spi);
- if (r)
- return r;
- } else {
- dev_err(&spi->dev, "platform data missing!\n");
- return -ENODEV;
- }
+ r = acx565akm_probe_of(spi);
+ if (r)
+ return r;
if (gpio_is_valid(ddata->reset_gpio)) {
r = devm_gpio_request_one(&spi->dev, ddata->reset_gpio,
struct omap_dss_device dssdev;
struct omap_dss_device *in;
- int data_lines;
-
struct videomode vm;
struct spi_device *spi_dev;
if (omapdss_device_is_enabled(dssdev))
return 0;
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
in->ops.dpi->set_timings(in, &ddata->vm);
r = in->ops.dpi->enable(in);
dssdev->type = OMAP_DISPLAY_TYPE_DPI;
dssdev->owner = THIS_MODULE;
dssdev->panel.vm = ddata->vm;
- dssdev->phy.dpi.data_lines = ddata->data_lines;
r = omapdss_register_display(dssdev);
if (r) {
struct videomode vm;
- int data_lines;
-
struct spi_device *spi;
struct regulator *vcc_reg;
int nreset_gpio;
if (omapdss_device_is_enabled(dssdev))
return 0;
- if (ddata->data_lines)
- in->ops.dpi->set_data_lines(in, ddata->data_lines);
in->ops.dpi->set_timings(in, &ddata->vm);
r = in->ops.dpi->enable(in);
.set_mirror = tpo_td043_set_hmirror,
.get_mirror = tpo_td043_get_hmirror,
-
- .get_resolution = omapdss_default_get_resolution,
};
static int tpo_td043_probe_of(struct spi_device *spi)
help
DPI Interface. This is the Parallel Display Interface.
-config OMAP2_DSS_RFBI
- bool "RFBI support"
- depends on BROKEN
- default n
- help
- MIPI DBI support (RFBI, Remote Framebuffer Interface, in Texas
- Instrument's terminology).
-
- DBI is a bus between the host processor and a peripheral,
- such as a display or a framebuffer chip.
-
- See http://www.mipi.org/ for DBI specifications.
-
config OMAP2_DSS_VENC
bool "VENC support"
default y
omapdss-y := core.o dss.o dss_features.o dispc.o dispc_coefs.o \
pll.o video-pll.o
omapdss-$(CONFIG_OMAP2_DSS_DPI) += dpi.o
-omapdss-$(CONFIG_OMAP2_DSS_RFBI) += rfbi.o
omapdss-$(CONFIG_OMAP2_DSS_VENC) += venc.o
omapdss-$(CONFIG_OMAP2_DSS_SDI) += sdi.o
omapdss-$(CONFIG_OMAP2_DSS_DSI) += dsi.o
static struct {
struct platform_device *pdev;
-
- const char *default_display_name;
} core;
-static char *def_disp_name;
-module_param_named(def_disp, def_disp_name, charp, 0);
-MODULE_PARM_DESC(def_disp, "default display name");
-
-const char *omapdss_get_default_display_name(void)
-{
- return core.default_display_name;
-}
-EXPORT_SYMBOL(omapdss_get_default_display_name);
-
enum omapdss_version omapdss_get_version(void)
{
struct omap_dss_board_info *pdata = core.pdev->dev.platform_data;
}
EXPORT_SYMBOL(omapdss_get_version);
-struct platform_device *dss_get_core_pdev(void)
-{
- return core.pdev;
-}
-
int dss_dsi_enable_pads(int dsi_id, unsigned lane_mask)
{
struct omap_dss_board_info *board_data = core.pdev->dev.platform_data;
static int __init omap_dss_probe(struct platform_device *pdev)
{
- struct omap_dss_board_info *pdata = pdev->dev.platform_data;
int r;
core.pdev = pdev;
if (r)
goto err_debugfs;
- if (def_disp_name)
- core.default_display_name = def_disp_name;
- else if (pdata->default_display_name)
- core.default_display_name = pdata->default_display_name;
-
return 0;
err_debugfs:
#ifdef CONFIG_OMAP2_DSS_DSI
dsi_init_platform_driver,
#endif
-#ifdef CONFIG_OMAP2_DSS_DPI
- dpi_init_platform_driver,
-#endif
-#ifdef CONFIG_OMAP2_DSS_SDI
- sdi_init_platform_driver,
-#endif
-#ifdef CONFIG_OMAP2_DSS_RFBI
- rfbi_init_platform_driver,
-#endif
#ifdef CONFIG_OMAP2_DSS_VENC
venc_init_platform_driver,
#endif
#ifdef CONFIG_OMAP2_DSS_VENC
venc_uninit_platform_driver,
#endif
-#ifdef CONFIG_OMAP2_DSS_RFBI
- rfbi_uninit_platform_driver,
-#endif
-#ifdef CONFIG_OMAP2_DSS_SDI
- sdi_uninit_platform_driver,
-#endif
-#ifdef CONFIG_OMAP2_DSS_DPI
- dpi_uninit_platform_driver,
-#endif
#ifdef CONFIG_OMAP2_DSS_DSI
dsi_uninit_platform_driver,
#endif
#include <linux/regmap.h>
#include <linux/of.h>
#include <linux/component.h>
+#include <drm/drm_fourcc.h>
+#include <drm/drm_blend.h>
#include "omapdss.h"
#include "dss.h"
int (*calc_scaling) (unsigned long pclk, unsigned long lclk,
const struct videomode *vm,
u16 width, u16 height, u16 out_width, u16 out_height,
- enum omap_color_mode color_mode, bool *five_taps,
+ u32 fourcc, bool *five_taps,
int *x_predecim, int *y_predecim, int *decim_x, int *decim_y,
u16 pos_x, unsigned long *core_clk, bool mem_to_mem);
unsigned long (*calc_core_clk) (unsigned long pclk,
*/
DISPC_COLOR_COMPONENT_RGB_Y = 1 << 0,
/* used for UV component for
- * OMAP_DSS_COLOR_YUV2, OMAP_DSS_COLOR_UYVY, OMAP_DSS_COLOR_NV12
+ * DRM_FORMAT_YUYV, DRM_FORMAT_UYVY, DRM_FORMAT_NV12
* color formats on OMAP4
*/
DISPC_COLOR_COMPONENT_UV = 1 << 1,
dispc_write_reg(DISPC_OVL_ROW_INC(plane), inc);
}
-static void dispc_ovl_set_color_mode(enum omap_plane_id plane,
- enum omap_color_mode color_mode)
+static void dispc_ovl_set_color_mode(enum omap_plane_id plane, u32 fourcc)
{
u32 m = 0;
if (plane != OMAP_DSS_GFX) {
- switch (color_mode) {
- case OMAP_DSS_COLOR_NV12:
+ switch (fourcc) {
+ case DRM_FORMAT_NV12:
m = 0x0; break;
- case OMAP_DSS_COLOR_RGBX16:
+ case DRM_FORMAT_XRGB4444:
m = 0x1; break;
- case OMAP_DSS_COLOR_RGBA16:
+ case DRM_FORMAT_RGBA4444:
m = 0x2; break;
- case OMAP_DSS_COLOR_RGB12U:
+ case DRM_FORMAT_RGBX4444:
m = 0x4; break;
- case OMAP_DSS_COLOR_ARGB16:
+ case DRM_FORMAT_ARGB4444:
m = 0x5; break;
- case OMAP_DSS_COLOR_RGB16:
+ case DRM_FORMAT_RGB565:
m = 0x6; break;
- case OMAP_DSS_COLOR_ARGB16_1555:
+ case DRM_FORMAT_ARGB1555:
m = 0x7; break;
- case OMAP_DSS_COLOR_RGB24U:
+ case DRM_FORMAT_XRGB8888:
m = 0x8; break;
- case OMAP_DSS_COLOR_RGB24P:
+ case DRM_FORMAT_RGB888:
m = 0x9; break;
- case OMAP_DSS_COLOR_YUV2:
+ case DRM_FORMAT_YUYV:
m = 0xa; break;
- case OMAP_DSS_COLOR_UYVY:
+ case DRM_FORMAT_UYVY:
m = 0xb; break;
- case OMAP_DSS_COLOR_ARGB32:
+ case DRM_FORMAT_ARGB8888:
m = 0xc; break;
- case OMAP_DSS_COLOR_RGBA32:
+ case DRM_FORMAT_RGBA8888:
m = 0xd; break;
- case OMAP_DSS_COLOR_RGBX32:
+ case DRM_FORMAT_RGBX8888:
m = 0xe; break;
- case OMAP_DSS_COLOR_XRGB16_1555:
+ case DRM_FORMAT_XRGB1555:
m = 0xf; break;
default:
BUG(); return;
}
} else {
- switch (color_mode) {
- case OMAP_DSS_COLOR_CLUT1:
- m = 0x0; break;
- case OMAP_DSS_COLOR_CLUT2:
- m = 0x1; break;
- case OMAP_DSS_COLOR_CLUT4:
- m = 0x2; break;
- case OMAP_DSS_COLOR_CLUT8:
- m = 0x3; break;
- case OMAP_DSS_COLOR_RGB12U:
+ switch (fourcc) {
+ case DRM_FORMAT_RGBX4444:
m = 0x4; break;
- case OMAP_DSS_COLOR_ARGB16:
+ case DRM_FORMAT_ARGB4444:
m = 0x5; break;
- case OMAP_DSS_COLOR_RGB16:
+ case DRM_FORMAT_RGB565:
m = 0x6; break;
- case OMAP_DSS_COLOR_ARGB16_1555:
+ case DRM_FORMAT_ARGB1555:
m = 0x7; break;
- case OMAP_DSS_COLOR_RGB24U:
+ case DRM_FORMAT_XRGB8888:
m = 0x8; break;
- case OMAP_DSS_COLOR_RGB24P:
+ case DRM_FORMAT_RGB888:
m = 0x9; break;
- case OMAP_DSS_COLOR_RGBX16:
+ case DRM_FORMAT_XRGB4444:
m = 0xa; break;
- case OMAP_DSS_COLOR_RGBA16:
+ case DRM_FORMAT_RGBA4444:
m = 0xb; break;
- case OMAP_DSS_COLOR_ARGB32:
+ case DRM_FORMAT_ARGB8888:
m = 0xc; break;
- case OMAP_DSS_COLOR_RGBA32:
+ case DRM_FORMAT_RGBA8888:
m = 0xd; break;
- case OMAP_DSS_COLOR_RGBX32:
+ case DRM_FORMAT_RGBX8888:
m = 0xe; break;
- case OMAP_DSS_COLOR_XRGB16_1555:
+ case DRM_FORMAT_XRGB1555:
m = 0xf; break;
default:
BUG(); return;
REG_FLD_MOD(DISPC_OVL_ATTRIBUTES(plane), m, 4, 1);
}
+static bool format_is_yuv(u32 fourcc)
+{
+ switch (fourcc) {
+ case DRM_FORMAT_YUYV:
+ case DRM_FORMAT_UYVY:
+ case DRM_FORMAT_NV12:
+ return true;
+ default:
+ return false;
+ }
+}
+
static void dispc_ovl_configure_burst_type(enum omap_plane_id plane,
enum omap_dss_rotation_type rotation_type)
{
return unit * 8;
}
-static enum omap_color_mode dispc_ovl_get_color_modes(enum omap_plane_id plane)
+static const u32 *dispc_ovl_get_color_modes(enum omap_plane_id plane)
{
return dss_feat_get_supported_color_modes(plane);
}
static void dispc_ovl_set_accu_uv(enum omap_plane_id plane,
u16 orig_width, u16 orig_height, u16 out_width, u16 out_height,
- bool ilace, enum omap_color_mode color_mode, u8 rotation)
+ bool ilace, u32 fourcc, u8 rotation)
{
int h_accu2_0, h_accu2_1;
int v_accu2_0, v_accu2_1;
{ 0, 1, 0, 1, -1, 1, 0, 1 },
};
- switch (rotation) {
- case OMAP_DSS_ROT_0:
+ /* Note: DSS HW rotates clockwise, DRM_MODE_ROTATE_* counter-clockwise */
+ switch (rotation & DRM_MODE_ROTATE_MASK) {
+ default:
+ case DRM_MODE_ROTATE_0:
idx = 0;
break;
- case OMAP_DSS_ROT_90:
- idx = 1;
+ case DRM_MODE_ROTATE_90:
+ idx = 3;
break;
- case OMAP_DSS_ROT_180:
+ case DRM_MODE_ROTATE_180:
idx = 2;
break;
- case OMAP_DSS_ROT_270:
- idx = 3;
+ case DRM_MODE_ROTATE_270:
+ idx = 1;
break;
- default:
- BUG();
- return;
}
- switch (color_mode) {
- case OMAP_DSS_COLOR_NV12:
+ switch (fourcc) {
+ case DRM_FORMAT_NV12:
if (ilace)
accu_table = accu_nv12_ilace;
else
accu_table = accu_nv12;
break;
- case OMAP_DSS_COLOR_YUV2:
- case OMAP_DSS_COLOR_UYVY:
+ case DRM_FORMAT_YUYV:
+ case DRM_FORMAT_UYVY:
accu_table = accu_yuv;
break;
default:
u16 orig_width, u16 orig_height,
u16 out_width, u16 out_height,
bool ilace, bool five_taps,
- bool fieldmode, enum omap_color_mode color_mode,
+ bool fieldmode, u32 fourcc,
u8 rotation)
{
int accu0 = 0;
u16 orig_width, u16 orig_height,
u16 out_width, u16 out_height,
bool ilace, bool five_taps,
- bool fieldmode, enum omap_color_mode color_mode,
+ bool fieldmode, u32 fourcc,
u8 rotation)
{
int scale_x = out_width != orig_width;
if (!dss_has_feature(FEAT_HANDLE_UV_SEPARATE))
return;
- if ((color_mode != OMAP_DSS_COLOR_YUV2 &&
- color_mode != OMAP_DSS_COLOR_UYVY &&
- color_mode != OMAP_DSS_COLOR_NV12)) {
+
+ if (!format_is_yuv(fourcc)) {
/* reset chroma resampling for RGB formats */
if (plane != OMAP_DSS_WB)
REG_FLD_MOD(DISPC_OVL_ATTRIBUTES2(plane), 0, 8, 8);
}
dispc_ovl_set_accu_uv(plane, orig_width, orig_height, out_width,
- out_height, ilace, color_mode, rotation);
+ out_height, ilace, fourcc, rotation);
- switch (color_mode) {
- case OMAP_DSS_COLOR_NV12:
+ switch (fourcc) {
+ case DRM_FORMAT_NV12:
if (chroma_upscale) {
/* UV is subsampled by 2 horizontally and vertically */
orig_height >>= 1;
}
break;
- case OMAP_DSS_COLOR_YUV2:
- case OMAP_DSS_COLOR_UYVY:
+ case DRM_FORMAT_YUYV:
+ case DRM_FORMAT_UYVY:
/* For YUV422 with 90/270 rotation, we don't upsample chroma */
- if (rotation == OMAP_DSS_ROT_0 ||
- rotation == OMAP_DSS_ROT_180) {
+ if (!drm_rotation_90_or_270(rotation)) {
if (chroma_upscale)
/* UV is subsampled by 2 horizontally */
orig_width >>= 1;
}
/* must use FIR for YUV422 if rotated */
- if (rotation != OMAP_DSS_ROT_0)
+ if ((rotation & DRM_MODE_ROTATE_MASK) != DRM_MODE_ROTATE_0)
scale_x = scale_y = true;
break;
u16 orig_width, u16 orig_height,
u16 out_width, u16 out_height,
bool ilace, bool five_taps,
- bool fieldmode, enum omap_color_mode color_mode,
+ bool fieldmode, u32 fourcc,
u8 rotation)
{
BUG_ON(plane == OMAP_DSS_GFX);
orig_width, orig_height,
out_width, out_height,
ilace, five_taps,
- fieldmode, color_mode,
+ fieldmode, fourcc,
rotation);
dispc_ovl_set_scaling_uv(plane,
orig_width, orig_height,
out_width, out_height,
ilace, five_taps,
- fieldmode, color_mode,
+ fieldmode, fourcc,
rotation);
}
static void dispc_ovl_set_rotation_attrs(enum omap_plane_id plane, u8 rotation,
- enum omap_dss_rotation_type rotation_type,
- bool mirroring, enum omap_color_mode color_mode)
+ enum omap_dss_rotation_type rotation_type, u32 fourcc)
{
bool row_repeat = false;
int vidrot = 0;
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY) {
+ /* Note: DSS HW rotates clockwise, DRM_MODE_ROTATE_* counter-clockwise */
+ if (fourcc == DRM_FORMAT_YUYV || fourcc == DRM_FORMAT_UYVY) {
- if (mirroring) {
- switch (rotation) {
- case OMAP_DSS_ROT_0:
+ if (rotation & DRM_MODE_REFLECT_X) {
+ switch (rotation & DRM_MODE_ROTATE_MASK) {
+ case DRM_MODE_ROTATE_0:
vidrot = 2;
break;
- case OMAP_DSS_ROT_90:
+ case DRM_MODE_ROTATE_90:
vidrot = 1;
break;
- case OMAP_DSS_ROT_180:
+ case DRM_MODE_ROTATE_180:
vidrot = 0;
break;
- case OMAP_DSS_ROT_270:
+ case DRM_MODE_ROTATE_270:
vidrot = 3;
break;
}
} else {
- switch (rotation) {
- case OMAP_DSS_ROT_0:
+ switch (rotation & DRM_MODE_ROTATE_MASK) {
+ case DRM_MODE_ROTATE_0:
vidrot = 0;
break;
- case OMAP_DSS_ROT_90:
- vidrot = 1;
+ case DRM_MODE_ROTATE_90:
+ vidrot = 3;
break;
- case OMAP_DSS_ROT_180:
+ case DRM_MODE_ROTATE_180:
vidrot = 2;
break;
- case OMAP_DSS_ROT_270:
- vidrot = 3;
+ case DRM_MODE_ROTATE_270:
+ vidrot = 1;
break;
}
}
- if (rotation == OMAP_DSS_ROT_90 || rotation == OMAP_DSS_ROT_270)
+ if (drm_rotation_90_or_270(rotation))
row_repeat = true;
else
row_repeat = false;
* NV12 in 1D mode must use ROTATION=1. Otherwise DSS will fetch extra
* rows beyond the framebuffer, which may cause OCP error.
*/
- if (color_mode == OMAP_DSS_COLOR_NV12 &&
- rotation_type != OMAP_DSS_ROT_TILER)
+ if (fourcc == DRM_FORMAT_NV12 && rotation_type != OMAP_DSS_ROT_TILER)
vidrot = 1;
REG_FLD_MOD(DISPC_OVL_ATTRIBUTES(plane), vidrot, 13, 12);
REG_FLD_MOD(DISPC_OVL_ATTRIBUTES(plane),
row_repeat ? 1 : 0, 18, 18);
- if (color_mode == OMAP_DSS_COLOR_NV12) {
- bool doublestride = (rotation_type == OMAP_DSS_ROT_TILER) &&
- (rotation == OMAP_DSS_ROT_0 ||
- rotation == OMAP_DSS_ROT_180);
+ if (dss_feat_color_mode_supported(plane, DRM_FORMAT_NV12)) {
+ bool doublestride =
+ fourcc == DRM_FORMAT_NV12 &&
+ rotation_type == OMAP_DSS_ROT_TILER &&
+ !drm_rotation_90_or_270(rotation);
+
/* DOUBLESTRIDE */
REG_FLD_MOD(DISPC_OVL_ATTRIBUTES(plane), doublestride, 22, 22);
}
-
}
-static int color_mode_to_bpp(enum omap_color_mode color_mode)
+static int color_mode_to_bpp(u32 fourcc)
{
- switch (color_mode) {
- case OMAP_DSS_COLOR_CLUT1:
- return 1;
- case OMAP_DSS_COLOR_CLUT2:
- return 2;
- case OMAP_DSS_COLOR_CLUT4:
- return 4;
- case OMAP_DSS_COLOR_CLUT8:
- case OMAP_DSS_COLOR_NV12:
+ switch (fourcc) {
+ case DRM_FORMAT_NV12:
return 8;
- case OMAP_DSS_COLOR_RGB12U:
- case OMAP_DSS_COLOR_RGB16:
- case OMAP_DSS_COLOR_ARGB16:
- case OMAP_DSS_COLOR_YUV2:
- case OMAP_DSS_COLOR_UYVY:
- case OMAP_DSS_COLOR_RGBA16:
- case OMAP_DSS_COLOR_RGBX16:
- case OMAP_DSS_COLOR_ARGB16_1555:
- case OMAP_DSS_COLOR_XRGB16_1555:
+ case DRM_FORMAT_RGBX4444:
+ case DRM_FORMAT_RGB565:
+ case DRM_FORMAT_ARGB4444:
+ case DRM_FORMAT_YUYV:
+ case DRM_FORMAT_UYVY:
+ case DRM_FORMAT_RGBA4444:
+ case DRM_FORMAT_XRGB4444:
+ case DRM_FORMAT_ARGB1555:
+ case DRM_FORMAT_XRGB1555:
return 16;
- case OMAP_DSS_COLOR_RGB24P:
+ case DRM_FORMAT_RGB888:
return 24;
- case OMAP_DSS_COLOR_RGB24U:
- case OMAP_DSS_COLOR_ARGB32:
- case OMAP_DSS_COLOR_RGBA32:
- case OMAP_DSS_COLOR_RGBX32:
+ case DRM_FORMAT_XRGB8888:
+ case DRM_FORMAT_ARGB8888:
+ case DRM_FORMAT_RGBA8888:
+ case DRM_FORMAT_RGBX8888:
return 32;
default:
BUG();
return 0;
}
-static void calc_vrfb_rotation_offset(u8 rotation, bool mirror,
- u16 screen_width,
- u16 width, u16 height,
- enum omap_color_mode color_mode, bool fieldmode,
- unsigned int field_offset,
- unsigned *offset0, unsigned *offset1,
- s32 *row_inc, s32 *pix_inc, int x_predecim, int y_predecim)
+static void calc_offset(u16 screen_width, u16 width,
+ u32 fourcc, bool fieldmode,
+ unsigned int field_offset, unsigned *offset0, unsigned *offset1,
+ s32 *row_inc, s32 *pix_inc, int x_predecim, int y_predecim,
+ enum omap_dss_rotation_type rotation_type, u8 rotation)
{
u8 ps;
- /* FIXME CLUT formats */
- switch (color_mode) {
- case OMAP_DSS_COLOR_CLUT1:
- case OMAP_DSS_COLOR_CLUT2:
- case OMAP_DSS_COLOR_CLUT4:
- case OMAP_DSS_COLOR_CLUT8:
- BUG();
- return;
- case OMAP_DSS_COLOR_YUV2:
- case OMAP_DSS_COLOR_UYVY:
- ps = 4;
- break;
- default:
- ps = color_mode_to_bpp(color_mode) / 8;
- break;
- }
+ ps = color_mode_to_bpp(fourcc) / 8;
- DSSDBG("calc_rot(%d): scrw %d, %dx%d\n", rotation, screen_width,
- width, height);
+ DSSDBG("scrw %d, width %d\n", screen_width, width);
- /*
- * field 0 = even field = bottom field
- * field 1 = odd field = top field
- */
- switch (rotation + mirror * 4) {
- case OMAP_DSS_ROT_0:
- case OMAP_DSS_ROT_180:
+ if (rotation_type == OMAP_DSS_ROT_TILER &&
+ (fourcc == DRM_FORMAT_UYVY || fourcc == DRM_FORMAT_YUYV) &&
+ drm_rotation_90_or_270(rotation)) {
/*
- * If the pixel format is YUV or UYVY divide the width
- * of the image by 2 for 0 and 180 degree rotation.
+ * HACK: ROW_INC needs to be calculated with TILER units.
+ * We get such 'screen_width' that multiplying it with the
+ * YUV422 pixel size gives the correct TILER container width.
+ * However, 'width' is in pixels and multiplying it with YUV422
+ * pixel size gives incorrect result. We thus multiply it here
+ * with 2 to match the 32 bit TILER unit size.
*/
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
- width = width >> 1;
- case OMAP_DSS_ROT_90:
- case OMAP_DSS_ROT_270:
- *offset1 = 0;
- if (field_offset)
- *offset0 = field_offset * screen_width * ps;
- else
- *offset0 = 0;
-
- *row_inc = pixinc(1 +
- (y_predecim * screen_width - x_predecim * width) +
- (fieldmode ? screen_width : 0), ps);
- *pix_inc = pixinc(x_predecim, ps);
- break;
-
- case OMAP_DSS_ROT_0 + 4:
- case OMAP_DSS_ROT_180 + 4:
- /* If the pixel format is YUV or UYVY divide the width
- * of the image by 2 for 0 degree and 180 degree
- */
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
- width = width >> 1;
- case OMAP_DSS_ROT_90 + 4:
- case OMAP_DSS_ROT_270 + 4:
- *offset1 = 0;
- if (field_offset)
- *offset0 = field_offset * screen_width * ps;
- else
- *offset0 = 0;
- *row_inc = pixinc(1 -
- (y_predecim * screen_width + x_predecim * width) -
- (fieldmode ? screen_width : 0), ps);
- *pix_inc = pixinc(x_predecim, ps);
- break;
-
- default:
- BUG();
- return;
- }
-}
-
-static void calc_dma_rotation_offset(u8 rotation, bool mirror,
- u16 screen_width,
- u16 width, u16 height,
- enum omap_color_mode color_mode, bool fieldmode,
- unsigned int field_offset,
- unsigned *offset0, unsigned *offset1,
- s32 *row_inc, s32 *pix_inc, int x_predecim, int y_predecim)
-{
- u8 ps;
- u16 fbw, fbh;
-
- /* FIXME CLUT formats */
- switch (color_mode) {
- case OMAP_DSS_COLOR_CLUT1:
- case OMAP_DSS_COLOR_CLUT2:
- case OMAP_DSS_COLOR_CLUT4:
- case OMAP_DSS_COLOR_CLUT8:
- BUG();
- return;
- default:
- ps = color_mode_to_bpp(color_mode) / 8;
- break;
- }
-
- DSSDBG("calc_rot(%d): scrw %d, %dx%d\n", rotation, screen_width,
- width, height);
-
- /* width & height are overlay sizes, convert to fb sizes */
-
- if (rotation == OMAP_DSS_ROT_0 || rotation == OMAP_DSS_ROT_180) {
- fbw = width;
- fbh = height;
- } else {
- fbw = height;
- fbh = width;
+ width *= 2;
}
/*
* field 0 = even field = bottom field
* field 1 = odd field = top field
*/
- switch (rotation + mirror * 4) {
- case OMAP_DSS_ROT_0:
- *offset1 = 0;
- if (field_offset)
- *offset0 = *offset1 + field_offset * screen_width * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(1 +
- (y_predecim * screen_width - fbw * x_predecim) +
- (fieldmode ? screen_width : 0), ps);
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
- *pix_inc = pixinc(x_predecim, 2 * ps);
- else
- *pix_inc = pixinc(x_predecim, ps);
- break;
- case OMAP_DSS_ROT_90:
- *offset1 = screen_width * (fbh - 1) * ps;
- if (field_offset)
- *offset0 = *offset1 + field_offset * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(screen_width * (fbh * x_predecim - 1) +
- y_predecim + (fieldmode ? 1 : 0), ps);
- *pix_inc = pixinc(-x_predecim * screen_width, ps);
- break;
- case OMAP_DSS_ROT_180:
- *offset1 = (screen_width * (fbh - 1) + fbw - 1) * ps;
- if (field_offset)
- *offset0 = *offset1 - field_offset * screen_width * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(-1 -
- (y_predecim * screen_width - fbw * x_predecim) -
- (fieldmode ? screen_width : 0), ps);
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
- *pix_inc = pixinc(-x_predecim, 2 * ps);
- else
- *pix_inc = pixinc(-x_predecim, ps);
- break;
- case OMAP_DSS_ROT_270:
- *offset1 = (fbw - 1) * ps;
- if (field_offset)
- *offset0 = *offset1 - field_offset * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(-screen_width * (fbh * x_predecim - 1) -
- y_predecim - (fieldmode ? 1 : 0), ps);
- *pix_inc = pixinc(x_predecim * screen_width, ps);
- break;
-
- /* mirroring */
- case OMAP_DSS_ROT_0 + 4:
- *offset1 = (fbw - 1) * ps;
- if (field_offset)
- *offset0 = *offset1 + field_offset * screen_width * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(y_predecim * screen_width * 2 - 1 +
- (fieldmode ? screen_width : 0),
- ps);
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
- *pix_inc = pixinc(-x_predecim, 2 * ps);
- else
- *pix_inc = pixinc(-x_predecim, ps);
- break;
-
- case OMAP_DSS_ROT_90 + 4:
- *offset1 = 0;
- if (field_offset)
- *offset0 = *offset1 + field_offset * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(-screen_width * (fbh * x_predecim - 1) +
- y_predecim + (fieldmode ? 1 : 0),
- ps);
- *pix_inc = pixinc(x_predecim * screen_width, ps);
- break;
-
- case OMAP_DSS_ROT_180 + 4:
- *offset1 = screen_width * (fbh - 1) * ps;
- if (field_offset)
- *offset0 = *offset1 - field_offset * screen_width * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(1 - y_predecim * screen_width * 2 -
- (fieldmode ? screen_width : 0),
- ps);
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
- *pix_inc = pixinc(x_predecim, 2 * ps);
- else
- *pix_inc = pixinc(x_predecim, ps);
- break;
-
- case OMAP_DSS_ROT_270 + 4:
- *offset1 = (screen_width * (fbh - 1) + fbw - 1) * ps;
- if (field_offset)
- *offset0 = *offset1 - field_offset * ps;
- else
- *offset0 = *offset1;
- *row_inc = pixinc(screen_width * (fbh * x_predecim - 1) -
- y_predecim - (fieldmode ? 1 : 0),
- ps);
- *pix_inc = pixinc(-x_predecim * screen_width, ps);
- break;
-
- default:
- BUG();
- return;
- }
-}
-
-static void calc_tiler_rotation_offset(u16 screen_width, u16 width,
- enum omap_color_mode color_mode, bool fieldmode,
- unsigned int field_offset, unsigned *offset0, unsigned *offset1,
- s32 *row_inc, s32 *pix_inc, int x_predecim, int y_predecim)
-{
- u8 ps;
-
- switch (color_mode) {
- case OMAP_DSS_COLOR_CLUT1:
- case OMAP_DSS_COLOR_CLUT2:
- case OMAP_DSS_COLOR_CLUT4:
- case OMAP_DSS_COLOR_CLUT8:
- BUG();
- return;
- default:
- ps = color_mode_to_bpp(color_mode) / 8;
- break;
- }
-
- DSSDBG("scrw %d, width %d\n", screen_width, width);
-
- /*
- * field 0 = even field = bottom field
- * field 1 = odd field = top field
- */
+ *offset0 = field_offset * screen_width * ps;
*offset1 = 0;
- if (field_offset)
- *offset0 = *offset1 + field_offset * screen_width * ps;
- else
- *offset0 = *offset1;
+
*row_inc = pixinc(1 + (y_predecim * screen_width - width * x_predecim) +
(fieldmode ? screen_width : 0), ps);
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY)
+ if (fourcc == DRM_FORMAT_YUYV || fourcc == DRM_FORMAT_UYVY)
*pix_inc = pixinc(x_predecim, 2 * ps);
else
*pix_inc = pixinc(x_predecim, ps);
static unsigned long calc_core_clk_five_taps(unsigned long pclk,
const struct videomode *vm, u16 width,
u16 height, u16 out_width, u16 out_height,
- enum omap_color_mode color_mode)
+ u32 fourcc)
{
u32 core_clk = 0;
u64 tmp;
do_div(tmp, out_width);
core_clk = max_t(u32, core_clk, tmp);
- if (color_mode == OMAP_DSS_COLOR_RGB24U)
+ if (fourcc == DRM_FORMAT_XRGB8888)
core_clk <<= 1;
}
static int dispc_ovl_calc_scaling_24xx(unsigned long pclk, unsigned long lclk,
const struct videomode *vm,
u16 width, u16 height, u16 out_width, u16 out_height,
- enum omap_color_mode color_mode, bool *five_taps,
+ u32 fourcc, bool *five_taps,
int *x_predecim, int *y_predecim, int *decim_x, int *decim_y,
u16 pos_x, unsigned long *core_clk, bool mem_to_mem)
{
static int dispc_ovl_calc_scaling_34xx(unsigned long pclk, unsigned long lclk,
const struct videomode *vm,
u16 width, u16 height, u16 out_width, u16 out_height,
- enum omap_color_mode color_mode, bool *five_taps,
+ u32 fourcc, bool *five_taps,
int *x_predecim, int *y_predecim, int *decim_x, int *decim_y,
u16 pos_x, unsigned long *core_clk, bool mem_to_mem)
{
if (*five_taps)
*core_clk = calc_core_clk_five_taps(pclk, vm,
in_width, in_height, out_width,
- out_height, color_mode);
+ out_height, fourcc);
else
*core_clk = dispc.feat->calc_core_clk(pclk, in_width,
in_height, out_width, out_height,
static int dispc_ovl_calc_scaling_44xx(unsigned long pclk, unsigned long lclk,
const struct videomode *vm,
u16 width, u16 height, u16 out_width, u16 out_height,
- enum omap_color_mode color_mode, bool *five_taps,
+ u32 fourcc, bool *five_taps,
int *x_predecim, int *y_predecim, int *decim_x, int *decim_y,
u16 pos_x, unsigned long *core_clk, bool mem_to_mem)
{
return -EINVAL;
}
- if (*decim_x > 4 && color_mode != OMAP_DSS_COLOR_NV12) {
+ if (*decim_x > 4 && fourcc != DRM_FORMAT_NV12) {
/*
* Let's disable all scaling that requires horizontal
* decimation with higher factor than 4, until we have
enum omap_overlay_caps caps,
const struct videomode *vm,
u16 width, u16 height, u16 out_width, u16 out_height,
- enum omap_color_mode color_mode, bool *five_taps,
+ u32 fourcc, bool *five_taps,
int *x_predecim, int *y_predecim, u16 pos_x,
enum omap_dss_rotation_type rotation_type, bool mem_to_mem)
{
2 : max_decim_limit;
}
- if (color_mode == OMAP_DSS_COLOR_CLUT1 ||
- color_mode == OMAP_DSS_COLOR_CLUT2 ||
- color_mode == OMAP_DSS_COLOR_CLUT4 ||
- color_mode == OMAP_DSS_COLOR_CLUT8) {
- *x_predecim = 1;
- *y_predecim = 1;
- *five_taps = false;
- return 0;
- }
-
decim_x = DIV_ROUND_UP(DIV_ROUND_UP(width, out_width), maxdownscale);
decim_y = DIV_ROUND_UP(DIV_ROUND_UP(height, out_height), maxdownscale);
return -EINVAL;
ret = dispc.feat->calc_scaling(pclk, lclk, vm, width, height,
- out_width, out_height, color_mode, five_taps,
+ out_width, out_height, fourcc, five_taps,
x_predecim, y_predecim, &decim_x, &decim_y, pos_x, &core_clk,
mem_to_mem);
if (ret)
static int dispc_ovl_setup_common(enum omap_plane_id plane,
enum omap_overlay_caps caps, u32 paddr, u32 p_uv_addr,
u16 screen_width, int pos_x, int pos_y, u16 width, u16 height,
- u16 out_width, u16 out_height, enum omap_color_mode color_mode,
- u8 rotation, bool mirror, u8 zorder, u8 pre_mult_alpha,
+ u16 out_width, u16 out_height, u32 fourcc,
+ u8 rotation, u8 zorder, u8 pre_mult_alpha,
u8 global_alpha, enum omap_dss_rotation_type rotation_type,
bool replication, const struct videomode *vm,
bool mem_to_mem)
if (paddr == 0 && rotation_type != OMAP_DSS_ROT_TILER)
return -EINVAL;
- switch (color_mode) {
- case OMAP_DSS_COLOR_YUV2:
- case OMAP_DSS_COLOR_UYVY:
- case OMAP_DSS_COLOR_NV12:
- if (in_width & 1) {
- DSSERR("input width %d is not even for YUV format\n",
- in_width);
- return -EINVAL;
- }
- break;
-
- default:
- break;
+ if (format_is_yuv(fourcc) && (in_width & 1)) {
+ DSSERR("input width %d is not even for YUV format\n", in_width);
+ return -EINVAL;
}
out_width = out_width == 0 ? width : out_width;
out_height);
}
- if (!dss_feat_color_mode_supported(plane, color_mode))
+ if (!dss_feat_color_mode_supported(plane, fourcc))
return -EINVAL;
r = dispc_ovl_calc_scaling(pclk, lclk, caps, vm, in_width,
- in_height, out_width, out_height, color_mode,
+ in_height, out_width, out_height, fourcc,
&five_taps, &x_predecim, &y_predecim, pos_x,
rotation_type, mem_to_mem);
if (r)
DSSDBG("predecimation %d x %x, new input size %d x %d\n",
x_predecim, y_predecim, in_width, in_height);
- switch (color_mode) {
- case OMAP_DSS_COLOR_YUV2:
- case OMAP_DSS_COLOR_UYVY:
- case OMAP_DSS_COLOR_NV12:
- if (in_width & 1) {
- DSSDBG("predecimated input width is not even for YUV format\n");
- DSSDBG("adjusting input width %d -> %d\n",
- in_width, in_width & ~1);
+ if (format_is_yuv(fourcc) && (in_width & 1)) {
+ DSSDBG("predecimated input width is not even for YUV format\n");
+ DSSDBG("adjusting input width %d -> %d\n",
+ in_width, in_width & ~1);
- in_width &= ~1;
- }
- break;
-
- default:
- break;
+ in_width &= ~1;
}
- if (color_mode == OMAP_DSS_COLOR_YUV2 ||
- color_mode == OMAP_DSS_COLOR_UYVY ||
- color_mode == OMAP_DSS_COLOR_NV12)
+ if (format_is_yuv(fourcc))
cconv = 1;
if (ilace && !fieldmode) {
frame_height = height;
}
- if (rotation_type == OMAP_DSS_ROT_TILER)
- calc_tiler_rotation_offset(screen_width, frame_width,
- color_mode, fieldmode, field_offset,
- &offset0, &offset1, &row_inc, &pix_inc,
- x_predecim, y_predecim);
- else if (rotation_type == OMAP_DSS_ROT_DMA)
- calc_dma_rotation_offset(rotation, mirror, screen_width,
- frame_width, frame_height,
- color_mode, fieldmode, field_offset,
- &offset0, &offset1, &row_inc, &pix_inc,
- x_predecim, y_predecim);
- else
- calc_vrfb_rotation_offset(rotation, mirror,
- screen_width, frame_width, frame_height,
- color_mode, fieldmode, field_offset,
- &offset0, &offset1, &row_inc, &pix_inc,
- x_predecim, y_predecim);
+ calc_offset(screen_width, frame_width,
+ fourcc, fieldmode, field_offset,
+ &offset0, &offset1, &row_inc, &pix_inc,
+ x_predecim, y_predecim,
+ rotation_type, rotation);
DSSDBG("offset0 %u, offset1 %u, row_inc %d, pix_inc %d\n",
offset0, offset1, row_inc, pix_inc);
- dispc_ovl_set_color_mode(plane, color_mode);
+ dispc_ovl_set_color_mode(plane, fourcc);
dispc_ovl_configure_burst_type(plane, rotation_type);
dispc_ovl_set_ba0(plane, paddr + offset0);
dispc_ovl_set_ba1(plane, paddr + offset1);
- if (OMAP_DSS_COLOR_NV12 == color_mode) {
+ if (fourcc == DRM_FORMAT_NV12) {
dispc_ovl_set_ba0_uv(plane, p_uv_addr + offset0);
dispc_ovl_set_ba1_uv(plane, p_uv_addr + offset1);
}
if (caps & OMAP_DSS_OVL_CAP_SCALE) {
dispc_ovl_set_scaling(plane, in_width, in_height, out_width,
out_height, ilace, five_taps, fieldmode,
- color_mode, rotation);
+ fourcc, rotation);
dispc_ovl_set_output_size(plane, out_width, out_height);
dispc_ovl_set_vid_color_conv(plane, cconv);
}
- dispc_ovl_set_rotation_attrs(plane, rotation, rotation_type, mirror,
- color_mode);
+ dispc_ovl_set_rotation_attrs(plane, rotation, rotation_type, fourcc);
dispc_ovl_set_zorder(plane, caps, zorder);
dispc_ovl_set_pre_mult_alpha(plane, caps, pre_mult_alpha);
static int dispc_ovl_setup(enum omap_plane_id plane,
const struct omap_overlay_info *oi,
- const struct videomode *vm, bool mem_to_mem)
+ const struct videomode *vm, bool mem_to_mem,
+ enum omap_channel channel)
{
int r;
enum omap_overlay_caps caps = dss_feat_get_overlay_caps(plane);
- enum omap_channel channel;
const bool replication = true;
- channel = dispc_ovl_get_channel_out(plane);
-
DSSDBG("dispc_ovl_setup %d, pa %pad, pa_uv %pad, sw %d, %d,%d, %dx%d ->"
- " %dx%d, cmode %x, rot %d, mir %d, chan %d repl %d\n",
+ " %dx%d, cmode %x, rot %d, chan %d repl %d\n",
plane, &oi->paddr, &oi->p_uv_addr, oi->screen_width, oi->pos_x,
oi->pos_y, oi->width, oi->height, oi->out_width, oi->out_height,
- oi->color_mode, oi->rotation, oi->mirror, channel, replication);
+ oi->fourcc, oi->rotation, channel, replication);
+
+ dispc_ovl_set_channel_out(plane, channel);
r = dispc_ovl_setup_common(plane, caps, oi->paddr, oi->p_uv_addr,
oi->screen_width, oi->pos_x, oi->pos_y, oi->width, oi->height,
- oi->out_width, oi->out_height, oi->color_mode, oi->rotation,
- oi->mirror, oi->zorder, oi->pre_mult_alpha, oi->global_alpha,
+ oi->out_width, oi->out_height, oi->fourcc, oi->rotation,
+ oi->zorder, oi->pre_mult_alpha, oi->global_alpha,
oi->rotation_type, replication, vm, mem_to_mem);
return r;
OMAP_DSS_OVL_CAP_SCALE | OMAP_DSS_OVL_CAP_PRE_MULT_ALPHA;
DSSDBG("dispc_wb_setup, pa %x, pa_uv %x, %d,%d -> %dx%d, cmode %x, "
- "rot %d, mir %d\n", wi->paddr, wi->p_uv_addr, in_width,
- in_height, wi->width, wi->height, wi->color_mode, wi->rotation,
- wi->mirror);
+ "rot %d\n", wi->paddr, wi->p_uv_addr, in_width,
+ in_height, wi->width, wi->height, wi->fourcc, wi->rotation);
r = dispc_ovl_setup_common(plane, caps, wi->paddr, wi->p_uv_addr,
wi->buf_width, pos_x, pos_y, in_width, in_height, wi->width,
- wi->height, wi->color_mode, wi->rotation, wi->mirror, zorder,
+ wi->height, wi->fourcc, wi->rotation, zorder,
wi->pre_mult_alpha, global_alpha, wi->rotation_type,
replication, vm, mem_to_mem);
- switch (wi->color_mode) {
- case OMAP_DSS_COLOR_RGB16:
- case OMAP_DSS_COLOR_RGB24P:
- case OMAP_DSS_COLOR_ARGB16:
- case OMAP_DSS_COLOR_RGBA16:
- case OMAP_DSS_COLOR_RGB12U:
- case OMAP_DSS_COLOR_ARGB16_1555:
- case OMAP_DSS_COLOR_XRGB16_1555:
- case OMAP_DSS_COLOR_RGBX16:
+ switch (wi->fourcc) {
+ case DRM_FORMAT_RGB565:
+ case DRM_FORMAT_RGB888:
+ case DRM_FORMAT_ARGB4444:
+ case DRM_FORMAT_RGBA4444:
+ case DRM_FORMAT_RGBX4444:
+ case DRM_FORMAT_ARGB1555:
+ case DRM_FORMAT_XRGB1555:
+ case DRM_FORMAT_XRGB4444:
truncation = true;
break;
default:
return 0;
}
-static bool dispc_ovl_enabled(enum omap_plane_id plane)
-{
- return REG_GET(DISPC_OVL_ATTRIBUTES(plane), 0, 0);
-}
-
static enum omap_dss_output_id dispc_mgr_get_supported_outputs(enum omap_channel channel)
{
return dss_feat_get_supported_outputs(channel);
dispc_write_reg(DISPC_IRQSTATUS, mask);
}
-static u32 dispc_read_irqenable(void)
-{
- return dispc_read_reg(DISPC_IRQENABLE);
-}
-
static void dispc_write_irqenable(u32 mask)
{
u32 old_mask = dispc_read_reg(DISPC_IRQENABLE);
dispc_clear_irqstatus((mask ^ old_mask) & mask);
dispc_write_reg(DISPC_IRQENABLE, mask);
+
+ /* flush posted write */
+ dispc_read_reg(DISPC_IRQENABLE);
}
void dispc_enable_sidle(void)
.ovli = {
.screen_width = 1,
.width = 1, .height = 1,
- .color_mode = OMAP_DSS_COLOR_RGB24U,
- .rotation = OMAP_DSS_ROT_0,
- .rotation_type = OMAP_DSS_ROT_DMA,
- .mirror = 0,
+ .fourcc = DRM_FORMAT_XRGB8888,
+ .rotation = DRM_MODE_ROTATE_0,
+ .rotation_type = OMAP_DSS_ROT_NONE,
.pos_x = 0, .pos_y = 0,
.out_width = 0, .out_height = 0,
.global_alpha = 0xff,
return 0;
i734_buf.size = i734.ovli.width * i734.ovli.height *
- color_mode_to_bpp(i734.ovli.color_mode) / 8;
+ color_mode_to_bpp(i734.ovli.fourcc) / 8;
i734_buf.vaddr = dma_alloc_writecombine(&dispc.pdev->dev, i734_buf.size,
&i734_buf.paddr, GFP_KERNEL);
REG_FLD_MOD(DISPC_CONFIG, 0x1f, 8, 4);
/* Setup and enable GFX plane */
- dispc_ovl_set_channel_out(OMAP_DSS_GFX, OMAP_DSS_CHANNEL_LCD);
- dispc_ovl_setup(OMAP_DSS_GFX, &ovli, &i734.vm, false);
+ dispc_ovl_setup(OMAP_DSS_GFX, &ovli, &i734.vm, false,
+ OMAP_DSS_CHANNEL_LCD);
dispc_ovl_enable(OMAP_DSS_GFX, true);
/* Set up and enable display manager for LCD1 */
static const struct dispc_ops dispc_ops = {
.read_irqstatus = dispc_read_irqstatus,
.clear_irqstatus = dispc_clear_irqstatus,
- .read_irqenable = dispc_read_irqenable,
.write_irqenable = dispc_write_irqenable,
.request_irq = dispc_request_irq,
.mgr_set_gamma = dispc_mgr_set_gamma,
.ovl_enable = dispc_ovl_enable,
- .ovl_enabled = dispc_ovl_enabled,
- .ovl_set_channel_out = dispc_ovl_set_channel_out,
.ovl_setup = dispc_ovl_setup,
.ovl_get_color_modes = dispc_ovl_get_color_modes,
};
return r;
dispc_mem = platform_get_resource(dispc.pdev, IORESOURCE_MEM, 0);
- if (!dispc_mem) {
- DSSERR("can't get IORESOURCE_MEM DISPC\n");
- return -EINVAL;
- }
-
- dispc.base = devm_ioremap(&pdev->dev, dispc_mem->start,
- resource_size(dispc_mem));
- if (!dispc.base) {
- DSSERR("can't ioremap DISPC\n");
- return -ENOMEM;
- }
+ dispc.base = devm_ioremap_resource(&pdev->dev, dispc_mem);
+ if (IS_ERR(dispc.base))
+ return PTR_ERR(dispc.base);
dispc.irq = platform_get_irq(dispc.pdev, 0);
if (dispc.irq < 0) {
#include "omapdss.h"
-void omapdss_default_get_resolution(struct omap_dss_device *dssdev,
- u16 *xres, u16 *yres)
-{
- *xres = dssdev->panel.vm.hactive;
- *yres = dssdev->panel.vm.vactive;
-}
-EXPORT_SYMBOL(omapdss_default_get_resolution);
-
-int omapdss_default_get_recommended_bpp(struct omap_dss_device *dssdev)
-{
- switch (dssdev->type) {
- case OMAP_DISPLAY_TYPE_DPI:
- if (dssdev->phy.dpi.data_lines == 24)
- return 24;
- else
- return 16;
-
- case OMAP_DISPLAY_TYPE_DBI:
- if (dssdev->ctrl.pixel_size == 24)
- return 24;
- else
- return 16;
- case OMAP_DISPLAY_TYPE_DSI:
- if (dssdev->panel.dsi_pix_fmt == OMAP_DSS_DSI_FMT_RGB565)
- return 16;
- else
- return 24;
- case OMAP_DISPLAY_TYPE_VENC:
- case OMAP_DISPLAY_TYPE_SDI:
- case OMAP_DISPLAY_TYPE_HDMI:
- case OMAP_DISPLAY_TYPE_DVI:
- return 24;
- default:
- BUG();
- return 0;
- }
-}
-EXPORT_SYMBOL(omapdss_default_get_recommended_bpp);
-
void omapdss_default_get_timings(struct omap_dss_device *dssdev,
struct videomode *vm)
{
int id;
/*
- * Note: this presumes all the displays are either using DT or non-DT,
- * which normally should be the case. This also presumes that all
- * displays either have an DT alias, or none has.
+ * Note: this presumes that all displays either have an DT alias, or
+ * none has.
*/
-
- if (dssdev->dev->of_node) {
- id = of_alias_get_id(dssdev->dev->of_node, "display");
-
- if (id < 0)
- id = disp_num_counter++;
- } else {
+ id = of_alias_get_id(dssdev->dev->of_node, "display");
+ if (id < 0)
id = disp_num_counter++;
- }
snprintf(dssdev->alias, sizeof(dssdev->alias), "display%d", id);
/* Use 'label' property for name, if it exists */
- if (dssdev->dev->of_node)
- of_property_read_string(dssdev->dev->of_node, "label",
- &dssdev->name);
+ of_property_read_string(dssdev->dev->of_node, "label", &dssdev->name);
if (dssdev->name == NULL)
dssdev->name = dssdev->alias;
- if (drv && drv->get_resolution == NULL)
- drv->get_resolution = omapdss_default_get_resolution;
- if (drv && drv->get_recommended_bpp == NULL)
- drv->get_recommended_bpp = omapdss_default_get_recommended_bpp;
if (drv && drv->get_timings == NULL)
drv->get_timings = omapdss_default_get_timings;
#include <linux/string.h>
#include <linux/of.h>
#include <linux/clk.h>
-#include <linux/component.h>
#include "omapdss.h"
#include "dss.h"
return container_of(dssdev, struct dpi_data, output);
}
-/* only used in non-DT mode */
-static struct dpi_data *dpi_get_data_from_pdev(struct platform_device *pdev)
-{
- return dev_get_drvdata(&pdev->dev);
-}
-
static enum dss_clk_source dpi_get_clk_src_dra7xx(enum omap_channel channel)
{
/*
return 0;
}
-static void dpi_set_data_lines(struct omap_dss_device *dssdev, int data_lines)
-{
- struct dpi_data *dpi = dpi_get_data_from_dssdev(dssdev);
-
- mutex_lock(&dpi->lock);
-
- dpi->data_lines = data_lines;
-
- mutex_unlock(&dpi->lock);
-}
-
static int dpi_verify_pll(struct dss_pll *pll)
{
int r;
.check_timings = dpi_check_timings,
.set_timings = dpi_set_timings,
.get_timings = dpi_get_timings,
-
- .set_data_lines = dpi_set_data_lines,
};
-static void dpi_init_output(struct platform_device *pdev)
-{
- struct dpi_data *dpi = dpi_get_data_from_pdev(pdev);
- struct omap_dss_device *out = &dpi->output;
-
- out->dev = &pdev->dev;
- out->id = OMAP_DSS_OUTPUT_DPI;
- out->output_type = OMAP_DISPLAY_TYPE_DPI;
- out->name = "dpi.0";
- out->dispc_channel = dpi_get_channel(0);
- out->ops.dpi = &dpi_ops;
- out->owner = THIS_MODULE;
-
- omapdss_register_output(out);
-}
-
-static void dpi_uninit_output(struct platform_device *pdev)
-{
- struct dpi_data *dpi = dpi_get_data_from_pdev(pdev);
- struct omap_dss_device *out = &dpi->output;
-
- omapdss_unregister_output(out);
-}
-
static void dpi_init_output_port(struct platform_device *pdev,
struct device_node *port)
{
omapdss_unregister_output(out);
}
-static int dpi_bind(struct device *dev, struct device *master, void *data)
-{
- struct platform_device *pdev = to_platform_device(dev);
- struct dpi_data *dpi;
-
- dpi = devm_kzalloc(&pdev->dev, sizeof(*dpi), GFP_KERNEL);
- if (!dpi)
- return -ENOMEM;
-
- dpi->pdev = pdev;
-
- dev_set_drvdata(&pdev->dev, dpi);
-
- mutex_init(&dpi->lock);
-
- dpi_init_output(pdev);
-
- return 0;
-}
-
-static void dpi_unbind(struct device *dev, struct device *master, void *data)
-{
- struct platform_device *pdev = to_platform_device(dev);
-
- dpi_uninit_output(pdev);
-}
-
-static const struct component_ops dpi_component_ops = {
- .bind = dpi_bind,
- .unbind = dpi_unbind,
-};
-
-static int dpi_probe(struct platform_device *pdev)
-{
- return component_add(&pdev->dev, &dpi_component_ops);
-}
-
-static int dpi_remove(struct platform_device *pdev)
-{
- component_del(&pdev->dev, &dpi_component_ops);
- return 0;
-}
-
-static struct platform_driver omap_dpi_driver = {
- .probe = dpi_probe,
- .remove = dpi_remove,
- .driver = {
- .name = "omapdss_dpi",
- .suppress_bind_attrs = true,
- },
-};
-
-int __init dpi_init_platform_driver(void)
-{
- return platform_driver_register(&omap_dpi_driver);
-}
-
-void dpi_uninit_platform_driver(void)
-{
- platform_driver_unregister(&omap_dpi_driver);
-}
-
int dpi_init_port(struct platform_device *pdev, struct device_node *port)
{
struct dpi_data *dpi;
static int dsi_bind(struct device *dev, struct device *master, void *data)
{
struct platform_device *dsidev = to_platform_device(dev);
+ const struct dsi_module_id_data *d;
u32 rev;
int r, i;
struct dsi_data *dsi;
struct resource *dsi_mem;
struct resource *res;
- struct resource temp_res;
dsi = devm_kzalloc(&dsidev->dev, sizeof(*dsi), GFP_KERNEL);
if (!dsi)
dsi->te_timer.data = 0;
#endif
- res = platform_get_resource_byname(dsidev, IORESOURCE_MEM, "proto");
- if (!res) {
- res = platform_get_resource(dsidev, IORESOURCE_MEM, 0);
- if (!res) {
- DSSERR("can't get IORESOURCE_MEM DSI\n");
- return -EINVAL;
- }
-
- temp_res.start = res->start;
- temp_res.end = temp_res.start + DSI_PROTO_SZ - 1;
- res = &temp_res;
- }
-
- dsi_mem = res;
-
- dsi->proto_base = devm_ioremap(&dsidev->dev, res->start,
- resource_size(res));
- if (!dsi->proto_base) {
- DSSERR("can't ioremap DSI protocol engine\n");
- return -ENOMEM;
- }
+ dsi_mem = platform_get_resource_byname(dsidev, IORESOURCE_MEM, "proto");
+ dsi->proto_base = devm_ioremap_resource(&dsidev->dev, dsi_mem);
+ if (IS_ERR(dsi->proto_base))
+ return PTR_ERR(dsi->proto_base);
res = platform_get_resource_byname(dsidev, IORESOURCE_MEM, "phy");
- if (!res) {
- res = platform_get_resource(dsidev, IORESOURCE_MEM, 0);
- if (!res) {
- DSSERR("can't get IORESOURCE_MEM DSI\n");
- return -EINVAL;
- }
-
- temp_res.start = res->start + DSI_PHY_OFFSET;
- temp_res.end = temp_res.start + DSI_PHY_SZ - 1;
- res = &temp_res;
- }
-
- dsi->phy_base = devm_ioremap(&dsidev->dev, res->start,
- resource_size(res));
- if (!dsi->phy_base) {
- DSSERR("can't ioremap DSI PHY\n");
- return -ENOMEM;
- }
+ dsi->phy_base = devm_ioremap_resource(&dsidev->dev, res);
+ if (IS_ERR(dsi->phy_base))
+ return PTR_ERR(dsi->phy_base);
res = platform_get_resource_byname(dsidev, IORESOURCE_MEM, "pll");
- if (!res) {
- res = platform_get_resource(dsidev, IORESOURCE_MEM, 0);
- if (!res) {
- DSSERR("can't get IORESOURCE_MEM DSI\n");
- return -EINVAL;
- }
-
- temp_res.start = res->start + DSI_PLL_OFFSET;
- temp_res.end = temp_res.start + DSI_PLL_SZ - 1;
- res = &temp_res;
- }
-
- dsi->pll_base = devm_ioremap(&dsidev->dev, res->start,
- resource_size(res));
- if (!dsi->pll_base) {
- DSSERR("can't ioremap DSI PLL\n");
- return -ENOMEM;
- }
+ dsi->pll_base = devm_ioremap_resource(&dsidev->dev, res);
+ if (IS_ERR(dsi->pll_base))
+ return PTR_ERR(dsi->pll_base);
dsi->irq = platform_get_irq(dsi->pdev, 0);
if (dsi->irq < 0) {
return r;
}
- if (dsidev->dev.of_node) {
- const struct of_device_id *match;
- const struct dsi_module_id_data *d;
-
- match = of_match_node(dsi_of_match, dsidev->dev.of_node);
- if (!match) {
- DSSERR("unsupported DSI module\n");
- return -ENODEV;
- }
-
- d = match->data;
-
- while (d->address != 0 && d->address != dsi_mem->start)
- d++;
-
- if (d->address == 0) {
- DSSERR("unsupported DSI module\n");
- return -ENODEV;
- }
+ d = of_match_node(dsi_of_match, dsidev->dev.of_node)->data;
+ while (d->address != 0 && d->address != dsi_mem->start)
+ d++;
- dsi->module_id = d->id;
- } else {
- dsi->module_id = dsidev->id;
+ if (d->address == 0) {
+ DSSERR("unsupported DSI module\n");
+ return -ENODEV;
}
+ dsi->module_id = d->id;
+
/* DSI VCs initialization */
for (i = 0; i < ARRAY_SIZE(dsi->vc); i++) {
dsi->vc[i].source = DSI_VC_SOURCE_L4;
dsi_init_output(dsidev);
- if (dsidev->dev.of_node) {
- r = dsi_probe_of(dsidev);
- if (r) {
- DSSERR("Invalid DSI DT data\n");
- goto err_probe_of;
- }
-
- r = of_platform_populate(dsidev->dev.of_node, NULL, NULL,
- &dsidev->dev);
- if (r)
- DSSERR("Failed to populate DSI child devices: %d\n", r);
+ r = dsi_probe_of(dsidev);
+ if (r) {
+ DSSERR("Invalid DSI DT data\n");
+ goto err_probe_of;
}
+ r = of_platform_populate(dsidev->dev.of_node, NULL, NULL, &dsidev->dev);
+ if (r)
+ DSSERR("Failed to populate DSI child devices: %d\n", r);
+
dsi_runtime_put(dsidev);
if (dsi->module_id == 0)
return r;
dss_mem = platform_get_resource(dss.pdev, IORESOURCE_MEM, 0);
- if (!dss_mem) {
- DSSERR("can't get IORESOURCE_MEM DSS\n");
- return -EINVAL;
- }
-
- dss.base = devm_ioremap(&pdev->dev, dss_mem->start,
- resource_size(dss_mem));
- if (!dss.base) {
- DSSERR("can't ioremap DSS\n");
- return -ENOMEM;
- }
+ dss.base = devm_ioremap_resource(&pdev->dev, dss_mem);
+ if (IS_ERR(dss.base))
+ return PTR_ERR(dss.base);
r = dss_get_clocks();
if (r)
struct platform_device;
/* core */
-struct platform_device *dss_get_core_pdev(void);
int dss_dsi_enable_pads(int dsi_id, unsigned lane_mask);
void dss_dsi_disable_pads(int dsi_id, unsigned lane_mask);
int dss_set_min_bus_tput(struct device *dev, unsigned long tput);
dss_div_calc_func func, void *data);
/* SDI */
-int sdi_init_platform_driver(void) __init;
-void sdi_uninit_platform_driver(void);
-
#ifdef CONFIG_OMAP2_DSS_SDI
int sdi_init_port(struct platform_device *pdev, struct device_node *port);
void sdi_uninit_port(struct device_node *port);
#endif
/* DPI */
-int dpi_init_platform_driver(void) __init;
-void dpi_uninit_platform_driver(void);
-
#ifdef CONFIG_OMAP2_DSS_DPI
int dpi_init_port(struct platform_device *pdev, struct device_node *port);
void dpi_uninit_port(struct device_node *port);
int hdmi5_init_platform_driver(void) __init;
void hdmi5_uninit_platform_driver(void);
-/* RFBI */
-int rfbi_init_platform_driver(void) __init;
-void rfbi_uninit_platform_driver(void);
-
#ifdef CONFIG_OMAP2_DSS_COLLECT_IRQ_STATS
static inline void dss_collect_irq_stats(u32 irqstatus, unsigned *irq_arr)
#include <linux/types.h>
#include <linux/err.h>
#include <linux/slab.h>
+#include <drm/drm_fourcc.h>
#include "omapdss.h"
#include "dss.h"
const int num_ovls;
const enum omap_display_type *supported_displays;
const enum omap_dss_output_id *supported_outputs;
- const enum omap_color_mode *supported_color_modes;
+ const u32 **supported_color_modes;
const enum omap_overlay_caps *overlay_caps;
const struct dss_param_range *dss_params;
- const enum omap_dss_rotation_type supported_rotation_types;
-
const u32 buffer_size_unit;
const u32 burst_size_unit;
};
OMAP_DSS_OUTPUT_DSI2,
};
-static const enum omap_color_mode omap2_dss_supported_color_modes[] = {
+#define COLOR_ARRAY(arr...) (const u32[]) { arr, 0 }
+
+static const u32 *omap2_dss_supported_color_modes[] = {
+
/* OMAP_DSS_GFX */
- OMAP_DSS_COLOR_CLUT1 | OMAP_DSS_COLOR_CLUT2 |
- OMAP_DSS_COLOR_CLUT4 | OMAP_DSS_COLOR_CLUT8 |
- OMAP_DSS_COLOR_RGB12U | OMAP_DSS_COLOR_RGB16 |
- OMAP_DSS_COLOR_RGB24U | OMAP_DSS_COLOR_RGB24P,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGBX4444, DRM_FORMAT_RGB565,
+ DRM_FORMAT_XRGB8888, DRM_FORMAT_RGB888),
/* OMAP_DSS_VIDEO1 */
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_YUV2 |
- OMAP_DSS_COLOR_UYVY,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGB565, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_YUYV,
+ DRM_FORMAT_UYVY),
/* OMAP_DSS_VIDEO2 */
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_YUV2 |
- OMAP_DSS_COLOR_UYVY,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGB565, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_YUYV,
+ DRM_FORMAT_UYVY),
};
-static const enum omap_color_mode omap3_dss_supported_color_modes[] = {
+static const u32 *omap3_dss_supported_color_modes[] = {
/* OMAP_DSS_GFX */
- OMAP_DSS_COLOR_CLUT1 | OMAP_DSS_COLOR_CLUT2 |
- OMAP_DSS_COLOR_CLUT4 | OMAP_DSS_COLOR_CLUT8 |
- OMAP_DSS_COLOR_RGB12U | OMAP_DSS_COLOR_ARGB16 |
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_ARGB32 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_RGBX32,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGBX4444, DRM_FORMAT_ARGB4444,
+ DRM_FORMAT_RGB565, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_RGBX8888),
/* OMAP_DSS_VIDEO1 */
- OMAP_DSS_COLOR_RGB24U | OMAP_DSS_COLOR_RGB24P |
- OMAP_DSS_COLOR_RGB12U | OMAP_DSS_COLOR_RGB16 |
- OMAP_DSS_COLOR_YUV2 | OMAP_DSS_COLOR_UYVY,
+ COLOR_ARRAY(
+ DRM_FORMAT_XRGB8888, DRM_FORMAT_RGB888,
+ DRM_FORMAT_RGBX4444, DRM_FORMAT_RGB565,
+ DRM_FORMAT_YUYV, DRM_FORMAT_UYVY),
/* OMAP_DSS_VIDEO2 */
- OMAP_DSS_COLOR_RGB12U | OMAP_DSS_COLOR_ARGB16 |
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_YUV2 |
- OMAP_DSS_COLOR_UYVY | OMAP_DSS_COLOR_ARGB32 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_RGBX32,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGBX4444, DRM_FORMAT_ARGB4444,
+ DRM_FORMAT_RGB565, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_YUYV,
+ DRM_FORMAT_UYVY, DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_RGBX8888),
};
-static const enum omap_color_mode omap4_dss_supported_color_modes[] = {
+static const u32 *omap4_dss_supported_color_modes[] = {
/* OMAP_DSS_GFX */
- OMAP_DSS_COLOR_CLUT1 | OMAP_DSS_COLOR_CLUT2 |
- OMAP_DSS_COLOR_CLUT4 | OMAP_DSS_COLOR_CLUT8 |
- OMAP_DSS_COLOR_RGB12U | OMAP_DSS_COLOR_ARGB16 |
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_ARGB32 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_RGBX32 |
- OMAP_DSS_COLOR_ARGB16_1555 | OMAP_DSS_COLOR_RGBX16 |
- OMAP_DSS_COLOR_RGBA16 | OMAP_DSS_COLOR_XRGB16_1555,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGBX4444, DRM_FORMAT_ARGB4444,
+ DRM_FORMAT_RGB565, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_ARGB8888,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_RGBX8888,
+ DRM_FORMAT_ARGB1555, DRM_FORMAT_XRGB4444,
+ DRM_FORMAT_RGBA4444, DRM_FORMAT_XRGB1555),
/* OMAP_DSS_VIDEO1 */
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB12U |
- OMAP_DSS_COLOR_YUV2 | OMAP_DSS_COLOR_ARGB16_1555 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_NV12 |
- OMAP_DSS_COLOR_RGBA16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_UYVY |
- OMAP_DSS_COLOR_ARGB16 | OMAP_DSS_COLOR_XRGB16_1555 |
- OMAP_DSS_COLOR_ARGB32 | OMAP_DSS_COLOR_RGBX16 |
- OMAP_DSS_COLOR_RGBX32,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGB565, DRM_FORMAT_RGBX4444,
+ DRM_FORMAT_YUYV, DRM_FORMAT_ARGB1555,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_NV12,
+ DRM_FORMAT_RGBA4444, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_UYVY,
+ DRM_FORMAT_ARGB4444, DRM_FORMAT_XRGB1555,
+ DRM_FORMAT_ARGB8888, DRM_FORMAT_XRGB4444,
+ DRM_FORMAT_RGBX8888),
/* OMAP_DSS_VIDEO2 */
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB12U |
- OMAP_DSS_COLOR_YUV2 | OMAP_DSS_COLOR_ARGB16_1555 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_NV12 |
- OMAP_DSS_COLOR_RGBA16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_UYVY |
- OMAP_DSS_COLOR_ARGB16 | OMAP_DSS_COLOR_XRGB16_1555 |
- OMAP_DSS_COLOR_ARGB32 | OMAP_DSS_COLOR_RGBX16 |
- OMAP_DSS_COLOR_RGBX32,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGB565, DRM_FORMAT_RGBX4444,
+ DRM_FORMAT_YUYV, DRM_FORMAT_ARGB1555,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_NV12,
+ DRM_FORMAT_RGBA4444, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_UYVY,
+ DRM_FORMAT_ARGB4444, DRM_FORMAT_XRGB1555,
+ DRM_FORMAT_ARGB8888, DRM_FORMAT_XRGB4444,
+ DRM_FORMAT_RGBX8888),
/* OMAP_DSS_VIDEO3 */
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB12U |
- OMAP_DSS_COLOR_YUV2 | OMAP_DSS_COLOR_ARGB16_1555 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_NV12 |
- OMAP_DSS_COLOR_RGBA16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_UYVY |
- OMAP_DSS_COLOR_ARGB16 | OMAP_DSS_COLOR_XRGB16_1555 |
- OMAP_DSS_COLOR_ARGB32 | OMAP_DSS_COLOR_RGBX16 |
- OMAP_DSS_COLOR_RGBX32,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGB565, DRM_FORMAT_RGBX4444,
+ DRM_FORMAT_YUYV, DRM_FORMAT_ARGB1555,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_NV12,
+ DRM_FORMAT_RGBA4444, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_UYVY,
+ DRM_FORMAT_ARGB4444, DRM_FORMAT_XRGB1555,
+ DRM_FORMAT_ARGB8888, DRM_FORMAT_XRGB4444,
+ DRM_FORMAT_RGBX8888),
/* OMAP_DSS_WB */
- OMAP_DSS_COLOR_RGB16 | OMAP_DSS_COLOR_RGB12U |
- OMAP_DSS_COLOR_YUV2 | OMAP_DSS_COLOR_ARGB16_1555 |
- OMAP_DSS_COLOR_RGBA32 | OMAP_DSS_COLOR_NV12 |
- OMAP_DSS_COLOR_RGBA16 | OMAP_DSS_COLOR_RGB24U |
- OMAP_DSS_COLOR_RGB24P | OMAP_DSS_COLOR_UYVY |
- OMAP_DSS_COLOR_ARGB16 | OMAP_DSS_COLOR_XRGB16_1555 |
- OMAP_DSS_COLOR_ARGB32 | OMAP_DSS_COLOR_RGBX16 |
- OMAP_DSS_COLOR_RGBX32,
+ COLOR_ARRAY(
+ DRM_FORMAT_RGB565, DRM_FORMAT_RGBX4444,
+ DRM_FORMAT_YUYV, DRM_FORMAT_ARGB1555,
+ DRM_FORMAT_RGBA8888, DRM_FORMAT_NV12,
+ DRM_FORMAT_RGBA4444, DRM_FORMAT_XRGB8888,
+ DRM_FORMAT_RGB888, DRM_FORMAT_UYVY,
+ DRM_FORMAT_ARGB4444, DRM_FORMAT_XRGB1555,
+ DRM_FORMAT_ARGB8888, DRM_FORMAT_XRGB4444,
+ DRM_FORMAT_RGBX8888),
};
static const enum omap_overlay_caps omap2_dss_overlay_caps[] = {
.supported_color_modes = omap2_dss_supported_color_modes,
.overlay_caps = omap2_dss_overlay_caps,
.dss_params = omap2_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_VRFB,
.buffer_size_unit = 1,
.burst_size_unit = 8,
};
.supported_color_modes = omap3_dss_supported_color_modes,
.overlay_caps = omap3430_dss_overlay_caps,
.dss_params = omap3_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_VRFB,
.buffer_size_unit = 1,
.burst_size_unit = 8,
};
.supported_color_modes = omap3_dss_supported_color_modes,
.overlay_caps = omap3430_dss_overlay_caps,
.dss_params = omap3_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_VRFB,
.buffer_size_unit = 1,
.burst_size_unit = 8,
};
.supported_color_modes = omap3_dss_supported_color_modes,
.overlay_caps = omap3430_dss_overlay_caps,
.dss_params = am43xx_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA,
.buffer_size_unit = 1,
.burst_size_unit = 8,
};
.supported_color_modes = omap3_dss_supported_color_modes,
.overlay_caps = omap3630_dss_overlay_caps,
.dss_params = omap3_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_VRFB,
.buffer_size_unit = 1,
.burst_size_unit = 8,
};
.supported_color_modes = omap4_dss_supported_color_modes,
.overlay_caps = omap4_dss_overlay_caps,
.dss_params = omap4_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_TILER,
.buffer_size_unit = 16,
.burst_size_unit = 16,
};
.supported_color_modes = omap4_dss_supported_color_modes,
.overlay_caps = omap4_dss_overlay_caps,
.dss_params = omap4_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_TILER,
.buffer_size_unit = 16,
.burst_size_unit = 16,
};
.supported_color_modes = omap4_dss_supported_color_modes,
.overlay_caps = omap4_dss_overlay_caps,
.dss_params = omap4_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_TILER,
.buffer_size_unit = 16,
.burst_size_unit = 16,
};
.supported_color_modes = omap4_dss_supported_color_modes,
.overlay_caps = omap4_dss_overlay_caps,
.dss_params = omap5_dss_param_range,
- .supported_rotation_types = OMAP_DSS_ROT_DMA | OMAP_DSS_ROT_TILER,
.buffer_size_unit = 16,
.burst_size_unit = 16,
};
return omap_current_dss_features->supported_outputs[channel];
}
-enum omap_color_mode dss_feat_get_supported_color_modes(enum omap_plane_id plane)
+const u32 *dss_feat_get_supported_color_modes(enum omap_plane_id plane)
{
return omap_current_dss_features->supported_color_modes[plane];
}
return omap_current_dss_features->overlay_caps[plane];
}
-bool dss_feat_color_mode_supported(enum omap_plane_id plane,
- enum omap_color_mode color_mode)
+bool dss_feat_color_mode_supported(enum omap_plane_id plane, u32 fourcc)
{
- return omap_current_dss_features->supported_color_modes[plane] &
- color_mode;
+ const u32 *modes;
+ unsigned int i;
+
+ modes = omap_current_dss_features->supported_color_modes[plane];
+
+ for (i = 0; modes[i]; ++i) {
+ if (modes[i] == fourcc)
+ return true;
+ }
+
+ return false;
}
u32 dss_feat_get_buffer_size_unit(void)
*end = omap_current_dss_features->reg_fields[id].end;
}
-bool dss_feat_rotation_type_supported(enum omap_dss_rotation_type rot_type)
-{
- return omap_current_dss_features->supported_rotation_types & rot_type;
-}
-
void dss_features_init(enum omapdss_version version)
{
switch (version) {
unsigned long dss_feat_get_param_max(enum dss_range_param param);
enum omap_overlay_caps dss_feat_get_overlay_caps(enum omap_plane_id plane);
bool dss_feat_color_mode_supported(enum omap_plane_id plane,
- enum omap_color_mode color_mode);
+ u32 fourcc);
u32 dss_feat_get_buffer_size_unit(void); /* in bytes */
u32 dss_feat_get_burst_size_unit(void); /* in bytes */
-bool dss_feat_rotation_type_supported(enum omap_dss_rotation_type rot_type);
-
bool dss_has_feature(enum dss_feat_id id);
void dss_feat_get_reg_field(enum dss_feat_reg_field id, u8 *start, u8 *end);
void dss_features_init(enum omapdss_version version);
int dss_feat_get_num_mgrs(void);
int dss_feat_get_num_ovls(void);
-enum omap_color_mode dss_feat_get_supported_color_modes(enum omap_plane_id plane);
+const u32 *dss_feat_get_supported_color_modes(enum omap_plane_id plane);
#endif
mutex_init(&hdmi.lock);
spin_lock_init(&hdmi.audio_playing_lock);
- if (pdev->dev.of_node) {
- r = hdmi_probe_of(pdev);
- if (r)
- return r;
- }
+ r = hdmi_probe_of(pdev);
+ if (r)
+ return r;
r = hdmi_wp_init(pdev, &hdmi.wp);
if (r)
struct resource *res;
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "core");
- if (!res) {
- DSSERR("can't get CORE mem resource\n");
- return -EINVAL;
- }
-
core->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(core->base)) {
- DSSERR("can't ioremap CORE\n");
+ if (IS_ERR(core->base))
return PTR_ERR(core->base);
- }
return 0;
}
mutex_init(&hdmi.lock);
spin_lock_init(&hdmi.audio_playing_lock);
- if (pdev->dev.of_node) {
- r = hdmi_probe_of(pdev);
- if (r)
- return r;
- }
+ r = hdmi_probe_of(pdev);
+ if (r)
+ return r;
r = hdmi_wp_init(pdev, &hdmi.wp);
if (r)
struct resource *res;
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "core");
- if (!res) {
- DSSERR("can't get CORE IORESOURCE_MEM HDMI\n");
- return -EINVAL;
- }
-
core->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(core->base)) {
- DSSERR("can't ioremap HDMI core\n");
+ if (IS_ERR(core->base))
return PTR_ERR(core->base);
- }
return 0;
}
return r;
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "phy");
- if (!res) {
- DSSERR("can't get PHY mem resource\n");
- return -EINVAL;
- }
-
phy->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(phy->base)) {
- DSSERR("can't ioremap TX PHY\n");
+ if (IS_ERR(phy->base))
return PTR_ERR(phy->base);
- }
return 0;
}
pll->wp = wp;
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "pll");
- if (!res) {
- DSSERR("can't get PLL mem resource\n");
- return -EINVAL;
- }
-
pll->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(pll->base)) {
- DSSERR("can't ioremap PLLCTRL\n");
+ if (IS_ERR(pll->base))
return PTR_ERR(pll->base);
- }
r = dsi_init_pll_data(pdev, pll);
if (r) {
struct resource *res;
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "wp");
- if (!res) {
- DSSERR("can't get WP mem resource\n");
- return -EINVAL;
- }
- wp->phys_base = res->start;
-
wp->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(wp->base)) {
- DSSERR("can't ioremap HDMI WP\n");
+ if (IS_ERR(wp->base))
return PTR_ERR(wp->base);
- }
+
+ wp->phys_base = res->start;
return 0;
}
#define DISPC_IRQ_FRAMEDONE3 (1 << 30)
struct omap_dss_device;
-struct omap_overlay_manager;
struct dss_lcd_mgr_config;
struct snd_aes_iec958;
struct snd_cea_861_aud_if;
};
enum omap_color_mode {
- OMAP_DSS_COLOR_CLUT1 = 1 << 0, /* BITMAP 1 */
- OMAP_DSS_COLOR_CLUT2 = 1 << 1, /* BITMAP 2 */
- OMAP_DSS_COLOR_CLUT4 = 1 << 2, /* BITMAP 4 */
- OMAP_DSS_COLOR_CLUT8 = 1 << 3, /* BITMAP 8 */
- OMAP_DSS_COLOR_RGB12U = 1 << 4, /* RGB12, 16-bit container */
- OMAP_DSS_COLOR_ARGB16 = 1 << 5, /* ARGB16 */
- OMAP_DSS_COLOR_RGB16 = 1 << 6, /* RGB16 */
- OMAP_DSS_COLOR_RGB24U = 1 << 7, /* RGB24, 32-bit container */
- OMAP_DSS_COLOR_RGB24P = 1 << 8, /* RGB24, 24-bit container */
- OMAP_DSS_COLOR_YUV2 = 1 << 9, /* YUV2 4:2:2 co-sited */
- OMAP_DSS_COLOR_UYVY = 1 << 10, /* UYVY 4:2:2 co-sited */
- OMAP_DSS_COLOR_ARGB32 = 1 << 11, /* ARGB32 */
- OMAP_DSS_COLOR_RGBA32 = 1 << 12, /* RGBA32 */
- OMAP_DSS_COLOR_RGBX32 = 1 << 13, /* RGBx32 */
- OMAP_DSS_COLOR_NV12 = 1 << 14, /* NV12 format: YUV 4:2:0 */
- OMAP_DSS_COLOR_RGBA16 = 1 << 15, /* RGBA16 - 4444 */
- OMAP_DSS_COLOR_RGBX16 = 1 << 16, /* RGBx16 - 4444 */
- OMAP_DSS_COLOR_ARGB16_1555 = 1 << 17, /* ARGB16 - 1555 */
- OMAP_DSS_COLOR_XRGB16_1555 = 1 << 18, /* xRGB16 - 1555 */
+ _UNUSED_,
};
enum omap_dss_load_mode {
OMAP_DSS_COLOR_KEY_VID_SRC = 1,
};
-enum omap_rfbi_te_mode {
- OMAP_DSS_RFBI_TE_MODE_1 = 1,
- OMAP_DSS_RFBI_TE_MODE_2 = 2,
-};
-
enum omap_dss_signal_level {
OMAPDSS_SIG_ACTIVE_LOW,
OMAPDSS_SIG_ACTIVE_HIGH,
};
enum omap_dss_rotation_type {
- OMAP_DSS_ROT_DMA = 1 << 0,
- OMAP_DSS_ROT_VRFB = 1 << 1,
- OMAP_DSS_ROT_TILER = 1 << 2,
-};
-
-/* clockwise rotation angle */
-enum omap_dss_rotation_angle {
- OMAP_DSS_ROT_0 = 0,
- OMAP_DSS_ROT_90 = 1,
- OMAP_DSS_ROT_180 = 2,
- OMAP_DSS_ROT_270 = 3,
+ OMAP_DSS_ROT_NONE = 0,
+ OMAP_DSS_ROT_TILER = 1 << 0,
};
enum omap_overlay_caps {
OMAP_DSS_OVL_CAP_REPLICATION = 1 << 5,
};
-enum omap_overlay_manager_caps {
- OMAP_DSS_DUMMY_VALUE, /* add a dummy value to prevent compiler error */
-};
-
enum omap_dss_clk_source {
OMAP_DSS_CLK_SRC_FCK = 0, /* OMAP2/3: DSS1_ALWON_FCLK
* OMAP4: DSS_FCLK */
OMAP_DSS_OUTPUT_HDMI = 1 << 6,
};
-/* RFBI */
-
-struct rfbi_timings {
- int cs_on_time;
- int cs_off_time;
- int we_on_time;
- int we_off_time;
- int re_on_time;
- int re_off_time;
- int we_cycle_time;
- int re_cycle_time;
- int cs_pulse_width;
- int access_time;
-
- int clk_div;
-
- u32 tim[5]; /* set by rfbi_convert_timings() */
-
- int converted;
-};
-
/* DSI */
enum omap_dss_dsi_trans_mode {
u16 screen_width;
u16 width;
u16 height;
- enum omap_color_mode color_mode;
+ u32 fourcc;
u8 rotation;
enum omap_dss_rotation_type rotation_type;
- bool mirror;
u16 pos_x;
u16 pos_y;
u8 zorder;
};
-struct omap_overlay {
- struct kobject kobj;
- struct list_head list;
-
- /* static fields */
- const char *name;
- enum omap_plane_id id;
- enum omap_color_mode supported_modes;
- enum omap_overlay_caps caps;
-
- /* dynamic fields */
- struct omap_overlay_manager *manager;
-
- /*
- * The following functions do not block:
- *
- * is_enabled
- * set_overlay_info
- * get_overlay_info
- *
- * The rest of the functions may block and cannot be called from
- * interrupt context
- */
-
- int (*enable)(struct omap_overlay *ovl);
- int (*disable)(struct omap_overlay *ovl);
- bool (*is_enabled)(struct omap_overlay *ovl);
-
- int (*set_manager)(struct omap_overlay *ovl,
- struct omap_overlay_manager *mgr);
- int (*unset_manager)(struct omap_overlay *ovl);
-
- int (*set_overlay_info)(struct omap_overlay *ovl,
- struct omap_overlay_info *info);
- void (*get_overlay_info)(struct omap_overlay *ovl,
- struct omap_overlay_info *info);
-
- int (*wait_for_go)(struct omap_overlay *ovl);
-
- struct omap_dss_device *(*get_device)(struct omap_overlay *ovl);
-};
-
struct omap_overlay_manager_info {
u32 default_color;
struct omap_dss_cpr_coefs cpr_coefs;
};
-struct omap_overlay_manager {
- struct kobject kobj;
-
- /* static fields */
- const char *name;
- enum omap_channel id;
- enum omap_overlay_manager_caps caps;
- struct list_head overlays;
- enum omap_display_type supported_displays;
- enum omap_dss_output_id supported_outputs;
-
- /* dynamic fields */
- struct omap_dss_device *output;
-
- /*
- * The following functions do not block:
- *
- * set_manager_info
- * get_manager_info
- * apply
- *
- * The rest of the functions may block and cannot be called from
- * interrupt context
- */
-
- int (*set_output)(struct omap_overlay_manager *mgr,
- struct omap_dss_device *output);
- int (*unset_output)(struct omap_overlay_manager *mgr);
-
- int (*set_manager_info)(struct omap_overlay_manager *mgr,
- struct omap_overlay_manager_info *info);
- void (*get_manager_info)(struct omap_overlay_manager *mgr,
- struct omap_overlay_manager_info *info);
-
- int (*apply)(struct omap_overlay_manager *mgr);
- int (*wait_for_go)(struct omap_overlay_manager *mgr);
- int (*wait_for_vsync)(struct omap_overlay_manager *mgr);
-
- struct omap_dss_device *(*get_device)(struct omap_overlay_manager *mgr);
-};
-
/* 22 pins means 1 clk lane and 10 data lanes */
#define OMAP_DSS_MAX_DSI_PINS 22
u16 buf_width;
u16 width;
u16 height;
- enum omap_color_mode color_mode;
+ u32 fourcc;
u8 rotation;
enum omap_dss_rotation_type rotation_type;
- bool mirror;
u8 pre_mult_alpha;
};
struct videomode *vm);
void (*get_timings)(struct omap_dss_device *dssdev,
struct videomode *vm);
-
- void (*set_data_lines)(struct omap_dss_device *dssdev, int data_lines);
};
struct omapdss_sdi_ops {
struct videomode *vm);
void (*get_timings)(struct omap_dss_device *dssdev,
struct videomode *vm);
-
- void (*set_datapairs)(struct omap_dss_device *dssdev, int datapairs);
};
struct omapdss_dvi_ops {
void (*get_timings)(struct omap_dss_device *dssdev,
struct videomode *vm);
- void (*set_type)(struct omap_dss_device *dssdev,
- enum omap_dss_venc_type type);
- void (*invert_vid_out_polarity)(struct omap_dss_device *dssdev,
- bool invert_polarity);
-
int (*set_wss)(struct omap_dss_device *dssdev, u32 wss);
u32 (*get_wss)(struct omap_dss_device *dssdev);
};
enum omap_display_type type;
enum omap_display_type output_type;
- union {
- struct {
- u8 data_lines;
- } dpi;
-
- struct {
- u8 channel;
- u8 data_lines;
- } rfbi;
-
- struct {
- u8 datapairs;
- } sdi;
-
- struct {
- int module;
- } dsi;
-
- struct {
- enum omap_dss_venc_type type;
- bool invert_polarity;
- } venc;
- } phy;
-
struct {
struct videomode vm;
enum omap_dss_dsi_mode dsi_mode;
} panel;
- struct {
- u8 pixel_size;
- struct rfbi_timings rfbi_timings;
- } ctrl;
-
const char *name;
- /* used to match device to driver */
- const char *driver_name;
-
- void *data;
-
struct omap_dss_driver *driver;
union {
int port_num;
/* dynamic fields */
- struct omap_overlay_manager *manager;
-
struct omap_dss_device *dst;
};
void *buf, size_t size,
u16 x, u16 y, u16 w, u16 h);
- void (*get_resolution)(struct omap_dss_device *dssdev,
- u16 *xres, u16 *yres);
- void (*get_dimensions)(struct omap_dss_device *dssdev,
- u32 *width, u32 *height);
- int (*get_recommended_bpp)(struct omap_dss_device *dssdev);
-
int (*check_timings)(struct omap_dss_device *dssdev,
struct videomode *vm);
void (*set_timings)(struct omap_dss_device *dssdev,
struct omap_dss_device *omap_dss_get_next_device(struct omap_dss_device *from);
struct omap_dss_device *omap_dss_find_device(void *data,
int (*match)(struct omap_dss_device *dssdev, void *data));
-const char *omapdss_get_default_display_name(void);
-
-int dss_feat_get_num_mgrs(void);
-int dss_feat_get_num_ovls(void);
-enum omap_color_mode dss_feat_get_supported_color_modes(enum omap_plane_id plane);
-
int omap_dss_get_num_overlay_managers(void);
-struct omap_overlay_manager *omap_dss_get_overlay_manager(int num);
int omap_dss_get_num_overlays(void);
-struct omap_overlay *omap_dss_get_overlay(int num);
int omapdss_register_output(struct omap_dss_device *output);
void omapdss_unregister_output(struct omap_dss_device *output);
struct omap_dss_device *omap_dss_get_output(enum omap_dss_output_id id);
-struct omap_dss_device *omap_dss_find_output(const char *name);
struct omap_dss_device *omap_dss_find_output_by_port_node(struct device_node *port);
int omapdss_output_set_device(struct omap_dss_device *out,
struct omap_dss_device *dssdev);
int omapdss_output_unset_device(struct omap_dss_device *out);
struct omap_dss_device *omapdss_find_output_from_display(struct omap_dss_device *dssdev);
-struct omap_overlay_manager *omapdss_find_mgr_from_display(struct omap_dss_device *dssdev);
-void omapdss_default_get_resolution(struct omap_dss_device *dssdev,
- u16 *xres, u16 *yres);
-int omapdss_default_get_recommended_bpp(struct omap_dss_device *dssdev);
void omapdss_default_get_timings(struct omap_dss_device *dssdev,
struct videomode *vm);
struct dispc_ops {
u32 (*read_irqstatus)(void);
void (*clear_irqstatus)(u32 mask);
- u32 (*read_irqenable)(void);
void (*write_irqenable)(u32 mask);
int (*request_irq)(irq_handler_t handler, void *dev_id);
unsigned int length);
int (*ovl_enable)(enum omap_plane_id plane, bool enable);
- bool (*ovl_enabled)(enum omap_plane_id plane);
- void (*ovl_set_channel_out)(enum omap_plane_id plane,
- enum omap_channel channel);
int (*ovl_setup)(enum omap_plane_id plane,
const struct omap_overlay_info *oi,
- const struct videomode *vm, bool mem_to_mem);
+ const struct videomode *vm, bool mem_to_mem,
+ enum omap_channel channel);
- enum omap_color_mode (*ovl_get_color_modes)(enum omap_plane_id plane);
+ const u32 *(*ovl_get_color_modes)(enum omap_plane_id plane);
};
void dispc_set_ops(const struct dispc_ops *o);
}
EXPORT_SYMBOL(omap_dss_get_output);
-struct omap_dss_device *omap_dss_find_output(const char *name)
-{
- struct omap_dss_device *out;
-
- list_for_each_entry(out, &output_list, list) {
- if (strcmp(out->name, name) == 0)
- return omap_dss_get_device(out);
- }
-
- return NULL;
-}
-EXPORT_SYMBOL(omap_dss_find_output);
-
struct omap_dss_device *omap_dss_find_output_by_port_node(struct device_node *port)
{
struct device_node *src_node;
+++ /dev/null
-/*
- * linux/drivers/video/omap2/dss/rfbi.c
- *
- * Copyright (C) 2009 Nokia Corporation
- * Author: Tomi Valkeinen <tomi.valkeinen@nokia.com>
- *
- * Some code and ideas taken from drivers/video/omap/ driver
- * by Imre Deak.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License version 2 as published by
- * the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along with
- * this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-#define DSS_SUBSYS_NAME "RFBI"
-
-#include <linux/kernel.h>
-#include <linux/dma-mapping.h>
-#include <linux/export.h>
-#include <linux/vmalloc.h>
-#include <linux/clk.h>
-#include <linux/io.h>
-#include <linux/delay.h>
-#include <linux/kfifo.h>
-#include <linux/ktime.h>
-#include <linux/hrtimer.h>
-#include <linux/seq_file.h>
-#include <linux/semaphore.h>
-#include <linux/platform_device.h>
-#include <linux/pm_runtime.h>
-#include <linux/component.h>
-
-#include "omapdss.h"
-#include "dss.h"
-
-struct rfbi_reg { u16 idx; };
-
-#define RFBI_REG(idx) ((const struct rfbi_reg) { idx })
-
-#define RFBI_REVISION RFBI_REG(0x0000)
-#define RFBI_SYSCONFIG RFBI_REG(0x0010)
-#define RFBI_SYSSTATUS RFBI_REG(0x0014)
-#define RFBI_CONTROL RFBI_REG(0x0040)
-#define RFBI_PIXEL_CNT RFBI_REG(0x0044)
-#define RFBI_LINE_NUMBER RFBI_REG(0x0048)
-#define RFBI_CMD RFBI_REG(0x004c)
-#define RFBI_PARAM RFBI_REG(0x0050)
-#define RFBI_DATA RFBI_REG(0x0054)
-#define RFBI_READ RFBI_REG(0x0058)
-#define RFBI_STATUS RFBI_REG(0x005c)
-
-#define RFBI_CONFIG(n) RFBI_REG(0x0060 + (n)*0x18)
-#define RFBI_ONOFF_TIME(n) RFBI_REG(0x0064 + (n)*0x18)
-#define RFBI_CYCLE_TIME(n) RFBI_REG(0x0068 + (n)*0x18)
-#define RFBI_DATA_CYCLE1(n) RFBI_REG(0x006c + (n)*0x18)
-#define RFBI_DATA_CYCLE2(n) RFBI_REG(0x0070 + (n)*0x18)
-#define RFBI_DATA_CYCLE3(n) RFBI_REG(0x0074 + (n)*0x18)
-
-#define RFBI_VSYNC_WIDTH RFBI_REG(0x0090)
-#define RFBI_HSYNC_WIDTH RFBI_REG(0x0094)
-
-#define REG_FLD_MOD(idx, val, start, end) \
- rfbi_write_reg(idx, FLD_MOD(rfbi_read_reg(idx), val, start, end))
-
-enum omap_rfbi_cycleformat {
- OMAP_DSS_RFBI_CYCLEFORMAT_1_1 = 0,
- OMAP_DSS_RFBI_CYCLEFORMAT_2_1 = 1,
- OMAP_DSS_RFBI_CYCLEFORMAT_3_1 = 2,
- OMAP_DSS_RFBI_CYCLEFORMAT_3_2 = 3,
-};
-
-enum omap_rfbi_datatype {
- OMAP_DSS_RFBI_DATATYPE_12 = 0,
- OMAP_DSS_RFBI_DATATYPE_16 = 1,
- OMAP_DSS_RFBI_DATATYPE_18 = 2,
- OMAP_DSS_RFBI_DATATYPE_24 = 3,
-};
-
-enum omap_rfbi_parallelmode {
- OMAP_DSS_RFBI_PARALLELMODE_8 = 0,
- OMAP_DSS_RFBI_PARALLELMODE_9 = 1,
- OMAP_DSS_RFBI_PARALLELMODE_12 = 2,
- OMAP_DSS_RFBI_PARALLELMODE_16 = 3,
-};
-
-static int rfbi_convert_timings(struct rfbi_timings *t);
-static void rfbi_get_clk_info(u32 *clk_period, u32 *max_clk_div);
-
-static struct {
- struct platform_device *pdev;
- void __iomem *base;
-
- unsigned long l4_khz;
-
- enum omap_rfbi_datatype datatype;
- enum omap_rfbi_parallelmode parallelmode;
-
- enum omap_rfbi_te_mode te_mode;
- int te_enabled;
-
- void (*framedone_callback)(void *data);
- void *framedone_callback_data;
-
- struct omap_dss_device *dssdev[2];
-
- struct semaphore bus_lock;
-
- struct videomode vm;
- int pixel_size;
- int data_lines;
- struct rfbi_timings intf_timings;
-
- struct omap_dss_device output;
-} rfbi;
-
-static inline void rfbi_write_reg(const struct rfbi_reg idx, u32 val)
-{
- __raw_writel(val, rfbi.base + idx.idx);
-}
-
-static inline u32 rfbi_read_reg(const struct rfbi_reg idx)
-{
- return __raw_readl(rfbi.base + idx.idx);
-}
-
-static int rfbi_runtime_get(void)
-{
- int r;
-
- DSSDBG("rfbi_runtime_get\n");
-
- r = pm_runtime_get_sync(&rfbi.pdev->dev);
- WARN_ON(r < 0);
- return r < 0 ? r : 0;
-}
-
-static void rfbi_runtime_put(void)
-{
- int r;
-
- DSSDBG("rfbi_runtime_put\n");
-
- r = pm_runtime_put_sync(&rfbi.pdev->dev);
- WARN_ON(r < 0 && r != -ENOSYS);
-}
-
-static void rfbi_bus_lock(void)
-{
- down(&rfbi.bus_lock);
-}
-
-static void rfbi_bus_unlock(void)
-{
- up(&rfbi.bus_lock);
-}
-
-static void rfbi_write_command(const void *buf, u32 len)
-{
- switch (rfbi.parallelmode) {
- case OMAP_DSS_RFBI_PARALLELMODE_8:
- {
- const u8 *b = buf;
- for (; len; len--)
- rfbi_write_reg(RFBI_CMD, *b++);
- break;
- }
-
- case OMAP_DSS_RFBI_PARALLELMODE_16:
- {
- const u16 *w = buf;
- BUG_ON(len & 1);
- for (; len; len -= 2)
- rfbi_write_reg(RFBI_CMD, *w++);
- break;
- }
-
- case OMAP_DSS_RFBI_PARALLELMODE_9:
- case OMAP_DSS_RFBI_PARALLELMODE_12:
- default:
- BUG();
- }
-}
-
-static void rfbi_read_data(void *buf, u32 len)
-{
- switch (rfbi.parallelmode) {
- case OMAP_DSS_RFBI_PARALLELMODE_8:
- {
- u8 *b = buf;
- for (; len; len--) {
- rfbi_write_reg(RFBI_READ, 0);
- *b++ = rfbi_read_reg(RFBI_READ);
- }
- break;
- }
-
- case OMAP_DSS_RFBI_PARALLELMODE_16:
- {
- u16 *w = buf;
- BUG_ON(len & ~1);
- for (; len; len -= 2) {
- rfbi_write_reg(RFBI_READ, 0);
- *w++ = rfbi_read_reg(RFBI_READ);
- }
- break;
- }
-
- case OMAP_DSS_RFBI_PARALLELMODE_9:
- case OMAP_DSS_RFBI_PARALLELMODE_12:
- default:
- BUG();
- }
-}
-
-static void rfbi_write_data(const void *buf, u32 len)
-{
- switch (rfbi.parallelmode) {
- case OMAP_DSS_RFBI_PARALLELMODE_8:
- {
- const u8 *b = buf;
- for (; len; len--)
- rfbi_write_reg(RFBI_PARAM, *b++);
- break;
- }
-
- case OMAP_DSS_RFBI_PARALLELMODE_16:
- {
- const u16 *w = buf;
- BUG_ON(len & 1);
- for (; len; len -= 2)
- rfbi_write_reg(RFBI_PARAM, *w++);
- break;
- }
-
- case OMAP_DSS_RFBI_PARALLELMODE_9:
- case OMAP_DSS_RFBI_PARALLELMODE_12:
- default:
- BUG();
-
- }
-}
-
-static void rfbi_write_pixels(const void __iomem *buf, int scr_width,
- u16 x, u16 y,
- u16 w, u16 h)
-{
- int start_offset = scr_width * y + x;
- int horiz_offset = scr_width - w;
- int i;
-
- if (rfbi.datatype == OMAP_DSS_RFBI_DATATYPE_16 &&
- rfbi.parallelmode == OMAP_DSS_RFBI_PARALLELMODE_8) {
- const u16 __iomem *pd = buf;
- pd += start_offset;
-
- for (; h; --h) {
- for (i = 0; i < w; ++i) {
- const u8 __iomem *b = (const u8 __iomem *)pd;
- rfbi_write_reg(RFBI_PARAM, __raw_readb(b+1));
- rfbi_write_reg(RFBI_PARAM, __raw_readb(b+0));
- ++pd;
- }
- pd += horiz_offset;
- }
- } else if (rfbi.datatype == OMAP_DSS_RFBI_DATATYPE_24 &&
- rfbi.parallelmode == OMAP_DSS_RFBI_PARALLELMODE_8) {
- const u32 __iomem *pd = buf;
- pd += start_offset;
-
- for (; h; --h) {
- for (i = 0; i < w; ++i) {
- const u8 __iomem *b = (const u8 __iomem *)pd;
- rfbi_write_reg(RFBI_PARAM, __raw_readb(b+2));
- rfbi_write_reg(RFBI_PARAM, __raw_readb(b+1));
- rfbi_write_reg(RFBI_PARAM, __raw_readb(b+0));
- ++pd;
- }
- pd += horiz_offset;
- }
- } else if (rfbi.datatype == OMAP_DSS_RFBI_DATATYPE_16 &&
- rfbi.parallelmode == OMAP_DSS_RFBI_PARALLELMODE_16) {
- const u16 __iomem *pd = buf;
- pd += start_offset;
-
- for (; h; --h) {
- for (i = 0; i < w; ++i) {
- rfbi_write_reg(RFBI_PARAM, __raw_readw(pd));
- ++pd;
- }
- pd += horiz_offset;
- }
- } else {
- BUG();
- }
-}
-
-static int rfbi_transfer_area(struct omap_dss_device *dssdev,
- void (*callback)(void *data), void *data)
-{
- u32 l;
- int r;
- struct omap_overlay_manager *mgr = rfbi.output.manager;
- u16 width = rfbi.vm.hactive;
- u16 height = rfbi.vm.vactive;
-
- /*BUG_ON(callback == 0);*/
- BUG_ON(rfbi.framedone_callback != NULL);
-
- DSSDBG("rfbi_transfer_area %dx%d\n", width, height);
-
- dss_mgr_set_timings(mgr, &rfbi.vm);
-
- r = dss_mgr_enable(mgr);
- if (r)
- return r;
-
- rfbi.framedone_callback = callback;
- rfbi.framedone_callback_data = data;
-
- rfbi_write_reg(RFBI_PIXEL_CNT, width * height);
-
- l = rfbi_read_reg(RFBI_CONTROL);
- l = FLD_MOD(l, 1, 0, 0); /* enable */
- if (!rfbi.te_enabled)
- l = FLD_MOD(l, 1, 4, 4); /* ITE */
-
- rfbi_write_reg(RFBI_CONTROL, l);
-
- return 0;
-}
-
-static void framedone_callback(void *data)
-{
- void (*callback)(void *data);
-
- DSSDBG("FRAMEDONE\n");
-
- REG_FLD_MOD(RFBI_CONTROL, 0, 0, 0);
-
- callback = rfbi.framedone_callback;
- rfbi.framedone_callback = NULL;
-
- if (callback != NULL)
- callback(rfbi.framedone_callback_data);
-}
-
-#if 1 /* VERBOSE */
-static void rfbi_print_timings(void)
-{
- u32 l;
- u32 time;
-
- l = rfbi_read_reg(RFBI_CONFIG(0));
- time = 1000000000 / rfbi.l4_khz;
- if (l & (1 << 4))
- time *= 2;
-
- DSSDBG("Tick time %u ps\n", time);
- l = rfbi_read_reg(RFBI_ONOFF_TIME(0));
- DSSDBG("CSONTIME %d, CSOFFTIME %d, WEONTIME %d, WEOFFTIME %d, "
- "REONTIME %d, REOFFTIME %d\n",
- l & 0x0f, (l >> 4) & 0x3f, (l >> 10) & 0x0f, (l >> 14) & 0x3f,
- (l >> 20) & 0x0f, (l >> 24) & 0x3f);
-
- l = rfbi_read_reg(RFBI_CYCLE_TIME(0));
- DSSDBG("WECYCLETIME %d, RECYCLETIME %d, CSPULSEWIDTH %d, "
- "ACCESSTIME %d\n",
- (l & 0x3f), (l >> 6) & 0x3f, (l >> 12) & 0x3f,
- (l >> 22) & 0x3f);
-}
-#else
-static void rfbi_print_timings(void) {}
-#endif
-
-
-
-
-static u32 extif_clk_period;
-
-static inline unsigned long round_to_extif_ticks(unsigned long ps, int div)
-{
- int bus_tick = extif_clk_period * div;
- return (ps + bus_tick - 1) / bus_tick * bus_tick;
-}
-
-static int calc_reg_timing(struct rfbi_timings *t, int div)
-{
- t->clk_div = div;
-
- t->cs_on_time = round_to_extif_ticks(t->cs_on_time, div);
-
- t->we_on_time = round_to_extif_ticks(t->we_on_time, div);
- t->we_off_time = round_to_extif_ticks(t->we_off_time, div);
- t->we_cycle_time = round_to_extif_ticks(t->we_cycle_time, div);
-
- t->re_on_time = round_to_extif_ticks(t->re_on_time, div);
- t->re_off_time = round_to_extif_ticks(t->re_off_time, div);
- t->re_cycle_time = round_to_extif_ticks(t->re_cycle_time, div);
-
- t->access_time = round_to_extif_ticks(t->access_time, div);
- t->cs_off_time = round_to_extif_ticks(t->cs_off_time, div);
- t->cs_pulse_width = round_to_extif_ticks(t->cs_pulse_width, div);
-
- DSSDBG("[reg]cson %d csoff %d reon %d reoff %d\n",
- t->cs_on_time, t->cs_off_time, t->re_on_time, t->re_off_time);
- DSSDBG("[reg]weon %d weoff %d recyc %d wecyc %d\n",
- t->we_on_time, t->we_off_time, t->re_cycle_time,
- t->we_cycle_time);
- DSSDBG("[reg]rdaccess %d cspulse %d\n",
- t->access_time, t->cs_pulse_width);
-
- return rfbi_convert_timings(t);
-}
-
-static int calc_extif_timings(struct rfbi_timings *t)
-{
- u32 max_clk_div;
- int div;
-
- rfbi_get_clk_info(&extif_clk_period, &max_clk_div);
- for (div = 1; div <= max_clk_div; div++) {
- if (calc_reg_timing(t, div) == 0)
- break;
- }
-
- if (div <= max_clk_div)
- return 0;
-
- DSSERR("can't setup timings\n");
- return -1;
-}
-
-
-static void rfbi_set_timings(int rfbi_module, struct rfbi_timings *t)
-{
- int r;
-
- if (!t->converted) {
- r = calc_extif_timings(t);
- if (r < 0)
- DSSERR("Failed to calc timings\n");
- }
-
- BUG_ON(!t->converted);
-
- rfbi_write_reg(RFBI_ONOFF_TIME(rfbi_module), t->tim[0]);
- rfbi_write_reg(RFBI_CYCLE_TIME(rfbi_module), t->tim[1]);
-
- /* TIMEGRANULARITY */
- REG_FLD_MOD(RFBI_CONFIG(rfbi_module),
- (t->tim[2] ? 1 : 0), 4, 4);
-
- rfbi_print_timings();
-}
-
-static int ps_to_rfbi_ticks(int time, int div)
-{
- unsigned long tick_ps;
- int ret;
-
- /* Calculate in picosecs to yield more exact results */
- tick_ps = 1000000000 / (rfbi.l4_khz) * div;
-
- ret = (time + tick_ps - 1) / tick_ps;
-
- return ret;
-}
-
-static void rfbi_get_clk_info(u32 *clk_period, u32 *max_clk_div)
-{
- *clk_period = 1000000000 / rfbi.l4_khz;
- *max_clk_div = 2;
-}
-
-static int rfbi_convert_timings(struct rfbi_timings *t)
-{
- u32 l;
- int reon, reoff, weon, weoff, cson, csoff, cs_pulse;
- int actim, recyc, wecyc;
- int div = t->clk_div;
-
- if (div <= 0 || div > 2)
- return -1;
-
- /* Make sure that after conversion it still holds that:
- * weoff > weon, reoff > reon, recyc >= reoff, wecyc >= weoff,
- * csoff > cson, csoff >= max(weoff, reoff), actim > reon
- */
- weon = ps_to_rfbi_ticks(t->we_on_time, div);
- weoff = ps_to_rfbi_ticks(t->we_off_time, div);
- if (weoff <= weon)
- weoff = weon + 1;
- if (weon > 0x0f)
- return -1;
- if (weoff > 0x3f)
- return -1;
-
- reon = ps_to_rfbi_ticks(t->re_on_time, div);
- reoff = ps_to_rfbi_ticks(t->re_off_time, div);
- if (reoff <= reon)
- reoff = reon + 1;
- if (reon > 0x0f)
- return -1;
- if (reoff > 0x3f)
- return -1;
-
- cson = ps_to_rfbi_ticks(t->cs_on_time, div);
- csoff = ps_to_rfbi_ticks(t->cs_off_time, div);
- if (csoff <= cson)
- csoff = cson + 1;
- if (csoff < max(weoff, reoff))
- csoff = max(weoff, reoff);
- if (cson > 0x0f)
- return -1;
- if (csoff > 0x3f)
- return -1;
-
- l = cson;
- l |= csoff << 4;
- l |= weon << 10;
- l |= weoff << 14;
- l |= reon << 20;
- l |= reoff << 24;
-
- t->tim[0] = l;
-
- actim = ps_to_rfbi_ticks(t->access_time, div);
- if (actim <= reon)
- actim = reon + 1;
- if (actim > 0x3f)
- return -1;
-
- wecyc = ps_to_rfbi_ticks(t->we_cycle_time, div);
- if (wecyc < weoff)
- wecyc = weoff;
- if (wecyc > 0x3f)
- return -1;
-
- recyc = ps_to_rfbi_ticks(t->re_cycle_time, div);
- if (recyc < reoff)
- recyc = reoff;
- if (recyc > 0x3f)
- return -1;
-
- cs_pulse = ps_to_rfbi_ticks(t->cs_pulse_width, div);
- if (cs_pulse > 0x3f)
- return -1;
-
- l = wecyc;
- l |= recyc << 6;
- l |= cs_pulse << 12;
- l |= actim << 22;
-
- t->tim[1] = l;
-
- t->tim[2] = div - 1;
-
- t->converted = 1;
-
- return 0;
-}
-
-/* xxx FIX module selection missing */
-static int rfbi_setup_te(enum omap_rfbi_te_mode mode,
- unsigned hs_pulse_time, unsigned vs_pulse_time,
- int hs_pol_inv, int vs_pol_inv, int extif_div)
-{
- int hs, vs;
- int min;
- u32 l;
-
- hs = ps_to_rfbi_ticks(hs_pulse_time, 1);
- vs = ps_to_rfbi_ticks(vs_pulse_time, 1);
- if (hs < 2)
- return -EDOM;
- if (mode == OMAP_DSS_RFBI_TE_MODE_2)
- min = 2;
- else /* OMAP_DSS_RFBI_TE_MODE_1 */
- min = 4;
- if (vs < min)
- return -EDOM;
- if (vs == hs)
- return -EINVAL;
- rfbi.te_mode = mode;
- DSSDBG("setup_te: mode %d hs %d vs %d hs_inv %d vs_inv %d\n",
- mode, hs, vs, hs_pol_inv, vs_pol_inv);
-
- rfbi_write_reg(RFBI_HSYNC_WIDTH, hs);
- rfbi_write_reg(RFBI_VSYNC_WIDTH, vs);
-
- l = rfbi_read_reg(RFBI_CONFIG(0));
- if (hs_pol_inv)
- l &= ~(1 << 21);
- else
- l |= 1 << 21;
- if (vs_pol_inv)
- l &= ~(1 << 20);
- else
- l |= 1 << 20;
-
- return 0;
-}
-
-/* xxx FIX module selection missing */
-static int rfbi_enable_te(bool enable, unsigned line)
-{
- u32 l;
-
- DSSDBG("te %d line %d mode %d\n", enable, line, rfbi.te_mode);
- if (line > (1 << 11) - 1)
- return -EINVAL;
-
- l = rfbi_read_reg(RFBI_CONFIG(0));
- l &= ~(0x3 << 2);
- if (enable) {
- rfbi.te_enabled = 1;
- l |= rfbi.te_mode << 2;
- } else
- rfbi.te_enabled = 0;
- rfbi_write_reg(RFBI_CONFIG(0), l);
- rfbi_write_reg(RFBI_LINE_NUMBER, line);
-
- return 0;
-}
-
-static int rfbi_configure_bus(int rfbi_module, int bpp, int lines)
-{
- u32 l;
- int cycle1 = 0, cycle2 = 0, cycle3 = 0;
- enum omap_rfbi_cycleformat cycleformat;
- enum omap_rfbi_datatype datatype;
- enum omap_rfbi_parallelmode parallelmode;
-
- switch (bpp) {
- case 12:
- datatype = OMAP_DSS_RFBI_DATATYPE_12;
- break;
- case 16:
- datatype = OMAP_DSS_RFBI_DATATYPE_16;
- break;
- case 18:
- datatype = OMAP_DSS_RFBI_DATATYPE_18;
- break;
- case 24:
- datatype = OMAP_DSS_RFBI_DATATYPE_24;
- break;
- default:
- BUG();
- return 1;
- }
- rfbi.datatype = datatype;
-
- switch (lines) {
- case 8:
- parallelmode = OMAP_DSS_RFBI_PARALLELMODE_8;
- break;
- case 9:
- parallelmode = OMAP_DSS_RFBI_PARALLELMODE_9;
- break;
- case 12:
- parallelmode = OMAP_DSS_RFBI_PARALLELMODE_12;
- break;
- case 16:
- parallelmode = OMAP_DSS_RFBI_PARALLELMODE_16;
- break;
- default:
- BUG();
- return 1;
- }
- rfbi.parallelmode = parallelmode;
-
- if ((bpp % lines) == 0) {
- switch (bpp / lines) {
- case 1:
- cycleformat = OMAP_DSS_RFBI_CYCLEFORMAT_1_1;
- break;
- case 2:
- cycleformat = OMAP_DSS_RFBI_CYCLEFORMAT_2_1;
- break;
- case 3:
- cycleformat = OMAP_DSS_RFBI_CYCLEFORMAT_3_1;
- break;
- default:
- BUG();
- return 1;
- }
- } else if ((2 * bpp % lines) == 0) {
- if ((2 * bpp / lines) == 3)
- cycleformat = OMAP_DSS_RFBI_CYCLEFORMAT_3_2;
- else {
- BUG();
- return 1;
- }
- } else {
- BUG();
- return 1;
- }
-
- switch (cycleformat) {
- case OMAP_DSS_RFBI_CYCLEFORMAT_1_1:
- cycle1 = lines;
- break;
-
- case OMAP_DSS_RFBI_CYCLEFORMAT_2_1:
- cycle1 = lines;
- cycle2 = lines;
- break;
-
- case OMAP_DSS_RFBI_CYCLEFORMAT_3_1:
- cycle1 = lines;
- cycle2 = lines;
- cycle3 = lines;
- break;
-
- case OMAP_DSS_RFBI_CYCLEFORMAT_3_2:
- cycle1 = lines;
- cycle2 = (lines / 2) | ((lines / 2) << 16);
- cycle3 = (lines << 16);
- break;
- }
-
- REG_FLD_MOD(RFBI_CONTROL, 0, 3, 2); /* clear CS */
-
- l = 0;
- l |= FLD_VAL(parallelmode, 1, 0);
- l |= FLD_VAL(0, 3, 2); /* TRIGGERMODE: ITE */
- l |= FLD_VAL(0, 4, 4); /* TIMEGRANULARITY */
- l |= FLD_VAL(datatype, 6, 5);
- /* l |= FLD_VAL(2, 8, 7); */ /* L4FORMAT, 2pix/L4 */
- l |= FLD_VAL(0, 8, 7); /* L4FORMAT, 1pix/L4 */
- l |= FLD_VAL(cycleformat, 10, 9);
- l |= FLD_VAL(0, 12, 11); /* UNUSEDBITS */
- l |= FLD_VAL(0, 16, 16); /* A0POLARITY */
- l |= FLD_VAL(0, 17, 17); /* REPOLARITY */
- l |= FLD_VAL(0, 18, 18); /* WEPOLARITY */
- l |= FLD_VAL(0, 19, 19); /* CSPOLARITY */
- l |= FLD_VAL(1, 20, 20); /* TE_VSYNC_POLARITY */
- l |= FLD_VAL(1, 21, 21); /* HSYNCPOLARITY */
- rfbi_write_reg(RFBI_CONFIG(rfbi_module), l);
-
- rfbi_write_reg(RFBI_DATA_CYCLE1(rfbi_module), cycle1);
- rfbi_write_reg(RFBI_DATA_CYCLE2(rfbi_module), cycle2);
- rfbi_write_reg(RFBI_DATA_CYCLE3(rfbi_module), cycle3);
-
-
- l = rfbi_read_reg(RFBI_CONTROL);
- l = FLD_MOD(l, rfbi_module+1, 3, 2); /* Select CSx */
- l = FLD_MOD(l, 0, 1, 1); /* clear bypass */
- rfbi_write_reg(RFBI_CONTROL, l);
-
-
- DSSDBG("RFBI config: bpp %d, lines %d, cycles: 0x%x 0x%x 0x%x\n",
- bpp, lines, cycle1, cycle2, cycle3);
-
- return 0;
-}
-
-static int rfbi_configure(struct omap_dss_device *dssdev)
-{
- return rfbi_configure_bus(dssdev->phy.rfbi.channel, rfbi.pixel_size,
- rfbi.data_lines);
-}
-
-static int rfbi_update(struct omap_dss_device *dssdev, void (*callback)(void *),
- void *data)
-{
- return rfbi_transfer_area(dssdev, callback, data);
-}
-
-static void rfbi_set_size(struct omap_dss_device *dssdev, u16 w, u16 h)
-{
- rfbi.vm.hactive = w;
- rfbi.vm.vactive = h;
-}
-
-static void rfbi_set_pixel_size(struct omap_dss_device *dssdev, int pixel_size)
-{
- rfbi.pixel_size = pixel_size;
-}
-
-static void rfbi_set_data_lines(struct omap_dss_device *dssdev, int data_lines)
-{
- rfbi.data_lines = data_lines;
-}
-
-static void rfbi_set_interface_timings(struct omap_dss_device *dssdev,
- struct rfbi_timings *timings)
-{
- rfbi.intf_timings = *timings;
-}
-
-static void rfbi_dump_regs(struct seq_file *s)
-{
-#define DUMPREG(r) seq_printf(s, "%-35s %08x\n", #r, rfbi_read_reg(r))
-
- if (rfbi_runtime_get())
- return;
-
- DUMPREG(RFBI_REVISION);
- DUMPREG(RFBI_SYSCONFIG);
- DUMPREG(RFBI_SYSSTATUS);
- DUMPREG(RFBI_CONTROL);
- DUMPREG(RFBI_PIXEL_CNT);
- DUMPREG(RFBI_LINE_NUMBER);
- DUMPREG(RFBI_CMD);
- DUMPREG(RFBI_PARAM);
- DUMPREG(RFBI_DATA);
- DUMPREG(RFBI_READ);
- DUMPREG(RFBI_STATUS);
-
- DUMPREG(RFBI_CONFIG(0));
- DUMPREG(RFBI_ONOFF_TIME(0));
- DUMPREG(RFBI_CYCLE_TIME(0));
- DUMPREG(RFBI_DATA_CYCLE1(0));
- DUMPREG(RFBI_DATA_CYCLE2(0));
- DUMPREG(RFBI_DATA_CYCLE3(0));
-
- DUMPREG(RFBI_CONFIG(1));
- DUMPREG(RFBI_ONOFF_TIME(1));
- DUMPREG(RFBI_CYCLE_TIME(1));
- DUMPREG(RFBI_DATA_CYCLE1(1));
- DUMPREG(RFBI_DATA_CYCLE2(1));
- DUMPREG(RFBI_DATA_CYCLE3(1));
-
- DUMPREG(RFBI_VSYNC_WIDTH);
- DUMPREG(RFBI_HSYNC_WIDTH);
-
- rfbi_runtime_put();
-#undef DUMPREG
-}
-
-static void rfbi_config_lcd_manager(struct omap_dss_device *dssdev)
-{
- struct omap_overlay_manager *mgr = rfbi.output.manager;
- struct dss_lcd_mgr_config mgr_config;
-
- mgr_config.io_pad_mode = DSS_IO_PAD_MODE_RFBI;
-
- mgr_config.stallmode = true;
- /* Do we need fifohandcheck for RFBI? */
- mgr_config.fifohandcheck = false;
-
- mgr_config.video_port_width = rfbi.pixel_size;
- mgr_config.lcden_sig_polarity = 0;
-
- dss_mgr_set_lcd_config(mgr, &mgr_config);
-
- /*
- * Set rfbi.timings with default values, the hactive and vactive fields
- * are expected to be already configured by the panel driver via
- * omapdss_rfbi_set_size()
- */
- rfbi.vm.hsync_len = 1;
- rfbi.vm.hfront_porch = 1;
- rfbi.vm.hback_porch = 1;
- rfbi.vm.vsync_len = 1;
- rfbi.vm.vfront_porch = 0;
- rfbi.vm.vback_porch = 0;
-
- rfbi.vm.flags &= ~DISPLAY_FLAGS_INTERLACED;
- rfbi.vm.flags &= ~DISPLAY_FLAGS_HSYNC_LOW;
- rfbi.vm.flags |= DISPLAY_FLAGS_HSYNC_HIGH;
- rfbi.vm.flags &= ~DISPLAY_FLAGS_VSYNC_LOW;
- rfbi.vm.flags |= DISPLAY_FLAGS_VSYNC_HIGH;
- rfbi.vm.flags &= ~DISPLAY_FLAGS_PIXDATA_NEGEDGE;
- rfbi.vm.flags |= DISPLAY_FLAGS_PIXDATA_POSEDGE;
- rfbi.vm.flags &= ~DISPLAY_FLAGS_DE_LOW;
- rfbi.vm.flags |= DISPLAY_FLAGS_DE_HIGH;
- rfbi.vm.flags &= ~DISPLAY_FLAGS_SYNC_POSEDGE;
- rfbi.vm.flags |= DISPLAY_FLAGS_SYNC_NEGEDGE;
-
- dss_mgr_set_timings(mgr, &rfbi.vm);
-}
-
-static int rfbi_display_enable(struct omap_dss_device *dssdev)
-{
- struct omap_dss_device *out = &rfbi.output;
- int r;
-
- if (!out->dispc_channel_connected) {
- DSSERR("failed to enable display: no output/manager\n");
- return -ENODEV;
- }
-
- r = rfbi_runtime_get();
- if (r)
- return r;
-
- r = dss_mgr_register_framedone_handler(out->manager,
- framedone_callback, NULL);
- if (r) {
- DSSERR("can't get FRAMEDONE irq\n");
- goto err1;
- }
-
- rfbi_config_lcd_manager(dssdev);
-
- rfbi_configure_bus(dssdev->phy.rfbi.channel, rfbi.pixel_size,
- rfbi.data_lines);
-
- rfbi_set_timings(dssdev->phy.rfbi.channel, &rfbi.intf_timings);
-
- return 0;
-err1:
- rfbi_runtime_put();
- return r;
-}
-
-static void rfbi_display_disable(struct omap_dss_device *dssdev)
-{
- struct omap_dss_device *out = &rfbi.output;
-
- dss_mgr_unregister_framedone_handler(out->manager,
- framedone_callback, NULL);
-
- rfbi_runtime_put();
-}
-
-static int rfbi_init_display(struct omap_dss_device *dssdev)
-{
- rfbi.dssdev[dssdev->phy.rfbi.channel] = dssdev;
- return 0;
-}
-
-static void rfbi_init_output(struct platform_device *pdev)
-{
- struct omap_dss_device *out = &rfbi.output;
-
- out->dev = &pdev->dev;
- out->id = OMAP_DSS_OUTPUT_DBI;
- out->output_type = OMAP_DISPLAY_TYPE_DBI;
- out->name = "rfbi.0";
- out->dispc_channel = OMAP_DSS_CHANNEL_LCD;
- out->owner = THIS_MODULE;
-
- omapdss_register_output(out);
-}
-
-static void rfbi_uninit_output(struct platform_device *pdev)
-{
- struct omap_dss_device *out = &rfbi.output;
-
- omapdss_unregister_output(out);
-}
-
-/* RFBI HW IP initialisation */
-static int rfbi_bind(struct device *dev, struct device *master, void *data)
-{
- struct platform_device *pdev = to_platform_device(dev);
- u32 rev;
- struct resource *rfbi_mem;
- struct clk *clk;
- int r;
-
- rfbi.pdev = pdev;
-
- sema_init(&rfbi.bus_lock, 1);
-
- rfbi_mem = platform_get_resource(rfbi.pdev, IORESOURCE_MEM, 0);
- if (!rfbi_mem) {
- DSSERR("can't get IORESOURCE_MEM RFBI\n");
- return -EINVAL;
- }
-
- rfbi.base = devm_ioremap(&pdev->dev, rfbi_mem->start,
- resource_size(rfbi_mem));
- if (!rfbi.base) {
- DSSERR("can't ioremap RFBI\n");
- return -ENOMEM;
- }
-
- clk = clk_get(&pdev->dev, "ick");
- if (IS_ERR(clk)) {
- DSSERR("can't get ick\n");
- return PTR_ERR(clk);
- }
-
- rfbi.l4_khz = clk_get_rate(clk) / 1000;
-
- clk_put(clk);
-
- pm_runtime_enable(&pdev->dev);
-
- r = rfbi_runtime_get();
- if (r)
- goto err_runtime_get;
-
- msleep(10);
-
- rev = rfbi_read_reg(RFBI_REVISION);
- dev_dbg(&pdev->dev, "OMAP RFBI rev %d.%d\n",
- FLD_GET(rev, 7, 4), FLD_GET(rev, 3, 0));
-
- rfbi_runtime_put();
-
- dss_debugfs_create_file("rfbi", rfbi_dump_regs);
-
- rfbi_init_output(pdev);
-
- return 0;
-
-err_runtime_get:
- pm_runtime_disable(&pdev->dev);
- return r;
-}
-
-static void rfbi_unbind(struct device *dev, struct device *master, void *data)
-{
- struct platform_device *pdev = to_platform_device(dev);
-
- rfbi_uninit_output(pdev);
-
- pm_runtime_disable(&pdev->dev);
-
- return 0;
-}
-
-static const struct component_ops rfbi_component_ops = {
- .bind = rfbi_bind,
- .unbind = rfbi_unbind,
-};
-
-static int rfbi_probe(struct platform_device *pdev)
-{
- return component_add(&pdev->dev, &rfbi_component_ops);
-}
-
-static int rfbi_remove(struct platform_device *pdev)
-{
- component_del(&pdev->dev, &rfbi_component_ops);
- return 0;
-}
-
-static int rfbi_runtime_suspend(struct device *dev)
-{
- dispc_runtime_put();
-
- return 0;
-}
-
-static int rfbi_runtime_resume(struct device *dev)
-{
- int r;
-
- r = dispc_runtime_get();
- if (r < 0)
- return r;
-
- return 0;
-}
-
-static const struct dev_pm_ops rfbi_pm_ops = {
- .runtime_suspend = rfbi_runtime_suspend,
- .runtime_resume = rfbi_runtime_resume,
-};
-
-static struct platform_driver omap_rfbihw_driver = {
- .probe = rfbi_probe,
- .remove = rfbi_remove,
- .driver = {
- .name = "omapdss_rfbi",
- .pm = &rfbi_pm_ops,
- .suppress_bind_attrs = true,
- },
-};
-
-int __init rfbi_init_platform_driver(void)
-{
- return platform_driver_register(&omap_rfbihw_driver);
-}
-
-void rfbi_uninit_platform_driver(void)
-{
- platform_driver_unregister(&omap_rfbihw_driver);
-}
#include <linux/platform_device.h>
#include <linux/string.h>
#include <linux/of.h>
-#include <linux/component.h>
#include "omapdss.h"
#include "dss.h"
return 0;
}
-static void sdi_set_datapairs(struct omap_dss_device *dssdev, int datapairs)
-{
- sdi.datapairs = datapairs;
-}
-
static int sdi_init_regulator(void)
{
struct regulator *vdds_sdi;
.check_timings = sdi_check_timings,
.set_timings = sdi_set_timings,
.get_timings = sdi_get_timings,
-
- .set_datapairs = sdi_set_datapairs,
};
static void sdi_init_output(struct platform_device *pdev)
omapdss_unregister_output(out);
}
-static int sdi_bind(struct device *dev, struct device *master, void *data)
-{
- struct platform_device *pdev = to_platform_device(dev);
-
- sdi.pdev = pdev;
-
- sdi_init_output(pdev);
-
- return 0;
-}
-
-static void sdi_unbind(struct device *dev, struct device *master, void *data)
-{
- struct platform_device *pdev = to_platform_device(dev);
-
- sdi_uninit_output(pdev);
-}
-
-static const struct component_ops sdi_component_ops = {
- .bind = sdi_bind,
- .unbind = sdi_unbind,
-};
-
-static int sdi_probe(struct platform_device *pdev)
-{
- return component_add(&pdev->dev, &sdi_component_ops);
-}
-
-static int sdi_remove(struct platform_device *pdev)
-{
- component_del(&pdev->dev, &sdi_component_ops);
- return 0;
-}
-
-static struct platform_driver omap_sdi_driver = {
- .probe = sdi_probe,
- .remove = sdi_remove,
- .driver = {
- .name = "omapdss_sdi",
- .suppress_bind_attrs = true,
- },
-};
-
-int __init sdi_init_platform_driver(void)
-{
- return platform_driver_register(&omap_sdi_driver);
-}
-
-void sdi_uninit_platform_driver(void)
-{
- platform_driver_unregister(&omap_sdi_driver);
-}
-
int sdi_init_port(struct platform_device *pdev, struct device_node *port)
{
struct device_node *ep;
return r;
}
-static void venc_set_type(struct omap_dss_device *dssdev,
- enum omap_dss_venc_type type)
-{
- mutex_lock(&venc.venc_lock);
-
- venc.type = type;
-
- mutex_unlock(&venc.venc_lock);
-}
-
-static void venc_invert_vid_out_polarity(struct omap_dss_device *dssdev,
- bool invert_polarity)
-{
- mutex_lock(&venc.venc_lock);
-
- venc.invert_polarity = invert_polarity;
-
- mutex_unlock(&venc.venc_lock);
-}
-
static int venc_init_regulator(void)
{
struct regulator *vdda_dac;
if (venc.vdda_dac_reg != NULL)
return 0;
- if (venc.pdev->dev.of_node)
- vdda_dac = devm_regulator_get(&venc.pdev->dev, "vdda");
- else
- vdda_dac = devm_regulator_get(&venc.pdev->dev, "vdda_dac");
-
+ vdda_dac = devm_regulator_get(&venc.pdev->dev, "vdda");
if (IS_ERR(vdda_dac)) {
if (PTR_ERR(vdda_dac) != -EPROBE_DEFER)
DSSERR("can't get VDDA_DAC regulator\n");
.set_timings = venc_set_timings,
.get_timings = venc_get_timings,
- .set_type = venc_set_type,
- .invert_vid_out_polarity = venc_invert_vid_out_polarity,
-
.set_wss = venc_set_wss,
.get_wss = venc_get_wss,
};
venc.wss_data = 0;
venc_mem = platform_get_resource(venc.pdev, IORESOURCE_MEM, 0);
- if (!venc_mem) {
- DSSERR("can't get IORESOURCE_MEM VENC\n");
- return -EINVAL;
- }
-
- venc.base = devm_ioremap(&pdev->dev, venc_mem->start,
- resource_size(venc_mem));
- if (!venc.base) {
- DSSERR("can't ioremap VENC\n");
- return -ENOMEM;
- }
+ venc.base = devm_ioremap_resource(&pdev->dev, venc_mem);
+ if (IS_ERR(venc.base))
+ return PTR_ERR(venc.base);
r = venc_get_clocks(pdev);
if (r)
venc_runtime_put();
- if (pdev->dev.of_node) {
- r = venc_probe_of(pdev);
- if (r) {
- DSSERR("Invalid DT data\n");
- goto err_probe_of;
- }
+ r = venc_probe_of(pdev);
+ if (r) {
+ DSSERR("Invalid DT data\n");
+ goto err_probe_of;
}
dss_debugfs_create_file("venc", venc_dump_regs);
/* PLL CONTROL */
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, reg_name[id]);
- if (!res) {
- dev_err(&pdev->dev,
- "missing platform resource data for pll%d\n", id);
- return ERR_PTR(-ENODEV);
- }
-
pll_base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(pll_base)) {
- dev_err(&pdev->dev, "failed to ioremap pll%d reg_name\n", id);
+ if (IS_ERR(pll_base))
return ERR_CAST(pll_base);
- }
/* CLOCK CONTROL */
res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
clkctrl_name[id]);
- if (!res) {
- dev_err(&pdev->dev,
- "missing platform resource data for pll%d\n", id);
- return ERR_PTR(-ENODEV);
- }
-
clkctrl_base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(clkctrl_base)) {
- dev_err(&pdev->dev, "failed to ioremap pll%d clkctrl\n", id);
+ if (IS_ERR(clkctrl_base))
return ERR_CAST(clkctrl_base);
- }
/* CLKIN */
kfree(omap_crtc);
}
+static void omap_crtc_arm_event(struct drm_crtc *crtc)
+{
+ struct omap_crtc *omap_crtc = to_omap_crtc(crtc);
+
+ WARN_ON(omap_crtc->pending);
+ omap_crtc->pending = true;
+
+ if (crtc->state->event) {
+ omap_crtc->event = crtc->state->event;
+ crtc->state->event = NULL;
+ }
+}
+
static void omap_crtc_enable(struct drm_crtc *crtc)
{
struct omap_crtc *omap_crtc = to_omap_crtc(crtc);
ret = drm_crtc_vblank_get(crtc);
WARN_ON(ret != 0);
- WARN_ON(omap_crtc->pending);
- omap_crtc->pending = true;
+ omap_crtc_arm_event(crtc);
spin_unlock_irq(&crtc->dev->event_lock);
}
DBG("%s", omap_crtc->name);
+ spin_lock_irq(&crtc->dev->event_lock);
+ if (crtc->state->event) {
+ drm_crtc_send_vblank_event(crtc, crtc->state->event);
+ crtc->state->event = NULL;
+ }
+ spin_unlock_irq(&crtc->dev->event_lock);
+
drm_crtc_vblank_off(crtc);
}
spin_lock_irq(&crtc->dev->event_lock);
priv->dispc_ops->mgr_go(omap_crtc->channel);
-
- WARN_ON(omap_crtc->pending);
- omap_crtc->pending = true;
-
- if (crtc->state->event)
- omap_crtc->event = crtc->state->event;
+ omap_crtc_arm_event(crtc);
spin_unlock_irq(&crtc->dev->event_lock);
}
u32 min_align = 128;
int ret;
unsigned long flags;
- size_t slot_bytes;
+ u32 slot_bytes;
BUG_ON(!validfmt(fmt));
* this program. If not, see <http://www.gnu.org/licenses/>.
*/
-#include <linux/wait.h>
+#include <linux/sys_soc.h>
#include <drm/drm_atomic.h>
#include <drm/drm_atomic_helper.h>
drm_fb_helper_hotplug_event(priv->fbdev);
}
-struct omap_atomic_state_commit {
- struct work_struct work;
- struct drm_device *dev;
- struct drm_atomic_state *state;
- u32 crtcs;
-};
-
static void omap_atomic_wait_for_completion(struct drm_device *dev,
struct drm_atomic_state *old_state)
{
}
}
-static void omap_atomic_complete(struct omap_atomic_state_commit *commit)
+static void omap_atomic_commit_tail(struct drm_atomic_state *old_state)
{
- struct drm_device *dev = commit->dev;
+ struct drm_device *dev = old_state->dev;
struct omap_drm_private *priv = dev->dev_private;
- struct drm_atomic_state *old_state = commit->state;
- /* Apply the atomic update. */
priv->dispc_ops->runtime_get();
+ /* Apply the atomic update. */
drm_atomic_helper_commit_modeset_disables(dev, old_state);
/* With the current dss dispc implementation we have to enable
drm_atomic_helper_commit_planes(dev, old_state, 0);
+ drm_atomic_helper_commit_hw_done(old_state);
+
+ /*
+ * Wait for completion of the page flips to ensure that old buffers
+ * can't be touched by the hardware anymore before cleaning up planes.
+ */
omap_atomic_wait_for_completion(dev, old_state);
drm_atomic_helper_cleanup_planes(dev, old_state);
priv->dispc_ops->runtime_put();
-
- drm_atomic_state_put(old_state);
-
- /* Complete the commit, wake up any waiter. */
- spin_lock(&priv->commit.lock);
- priv->commit.pending &= ~commit->crtcs;
- spin_unlock(&priv->commit.lock);
-
- wake_up_all(&priv->commit.wait);
-
- kfree(commit);
-}
-
-static void omap_atomic_work(struct work_struct *work)
-{
- struct omap_atomic_state_commit *commit =
- container_of(work, struct omap_atomic_state_commit, work);
-
- omap_atomic_complete(commit);
-}
-
-static bool omap_atomic_is_pending(struct omap_drm_private *priv,
- struct omap_atomic_state_commit *commit)
-{
- bool pending;
-
- spin_lock(&priv->commit.lock);
- pending = priv->commit.pending & commit->crtcs;
- spin_unlock(&priv->commit.lock);
-
- return pending;
}
-static int omap_atomic_commit(struct drm_device *dev,
- struct drm_atomic_state *state, bool nonblock)
-{
- struct omap_drm_private *priv = dev->dev_private;
- struct omap_atomic_state_commit *commit;
- struct drm_crtc *crtc;
- struct drm_crtc_state *crtc_state;
- int i, ret;
-
- ret = drm_atomic_helper_prepare_planes(dev, state);
- if (ret)
- return ret;
-
- /* Allocate the commit object. */
- commit = kzalloc(sizeof(*commit), GFP_KERNEL);
- if (commit == NULL) {
- ret = -ENOMEM;
- goto error;
- }
-
- INIT_WORK(&commit->work, omap_atomic_work);
- commit->dev = dev;
- commit->state = state;
-
- /* Wait until all affected CRTCs have completed previous commits and
- * mark them as pending.
- */
- for_each_crtc_in_state(state, crtc, crtc_state, i)
- commit->crtcs |= drm_crtc_mask(crtc);
-
- wait_event(priv->commit.wait, !omap_atomic_is_pending(priv, commit));
-
- spin_lock(&priv->commit.lock);
- priv->commit.pending |= commit->crtcs;
- spin_unlock(&priv->commit.lock);
-
- /* Swap the state, this is the point of no return. */
- drm_atomic_helper_swap_state(state, true);
-
- drm_atomic_state_get(state);
- if (nonblock)
- schedule_work(&commit->work);
- else
- omap_atomic_complete(commit);
-
- return 0;
-
-error:
- drm_atomic_helper_cleanup_planes(dev, state);
- return ret;
-}
+static const struct drm_mode_config_helper_funcs omap_mode_config_helper_funcs = {
+ .atomic_commit_tail = omap_atomic_commit_tail,
+};
static const struct drm_mode_config_funcs omap_mode_config_funcs = {
.fb_create = omap_framebuffer_create,
.output_poll_changed = omap_fb_output_poll_changed,
.atomic_check = drm_atomic_helper_check,
- .atomic_commit = omap_atomic_commit,
+ .atomic_commit = drm_atomic_helper_commit,
};
static int get_connector_type(struct omap_dss_device *dssdev)
return DRM_MODE_CONNECTOR_DVID;
case OMAP_DISPLAY_TYPE_DSI:
return DRM_MODE_CONNECTOR_DSI;
+ case OMAP_DISPLAY_TYPE_DPI:
+ case OMAP_DISPLAY_TYPE_DBI:
+ return DRM_MODE_CONNECTOR_DPI;
+ case OMAP_DISPLAY_TYPE_VENC:
+ /* TODO: This could also be composite */
+ return DRM_MODE_CONNECTOR_SVIDEO;
+ case OMAP_DISPLAY_TYPE_SDI:
+ return DRM_MODE_CONNECTOR_LVDS;
default:
return DRM_MODE_CONNECTOR_Unknown;
}
static int omap_modeset_init_properties(struct drm_device *dev)
{
struct omap_drm_private *priv = dev->dev_private;
+ unsigned int num_planes = priv->dispc_ops->get_num_ovls();
- priv->zorder_prop = drm_property_create_range(dev, 0, "zorder", 0, 3);
+ priv->zorder_prop = drm_property_create_range(dev, 0, "zorder", 0,
+ num_planes - 1);
if (!priv->zorder_prop)
return -ENOMEM;
dev->mode_config.max_height = 2048;
dev->mode_config.funcs = &omap_mode_config_funcs;
+ dev->mode_config.helper_private = &omap_mode_config_helper_funcs;
drm_mode_config_reset(dev);
&args->handle);
}
-static int ioctl_gem_cpu_prep(struct drm_device *dev, void *data,
- struct drm_file *file_priv)
-{
- struct drm_omap_gem_cpu_prep *args = data;
- struct drm_gem_object *obj;
- int ret;
-
- VERB("%p:%p: handle=%d, op=%x", dev, file_priv, args->handle, args->op);
-
- obj = drm_gem_object_lookup(file_priv, args->handle);
- if (!obj)
- return -ENOENT;
-
- ret = omap_gem_op_sync(obj, args->op);
-
- if (!ret)
- ret = omap_gem_op_start(obj, args->op);
-
- drm_gem_object_unreference_unlocked(obj);
-
- return ret;
-}
-
-static int ioctl_gem_cpu_fini(struct drm_device *dev, void *data,
- struct drm_file *file_priv)
-{
- struct drm_omap_gem_cpu_fini *args = data;
- struct drm_gem_object *obj;
- int ret;
-
- VERB("%p:%p: handle=%d", dev, file_priv, args->handle);
-
- obj = drm_gem_object_lookup(file_priv, args->handle);
- if (!obj)
- return -ENOENT;
-
- /* XXX flushy, flushy */
- ret = 0;
-
- if (!ret)
- ret = omap_gem_op_finish(obj, args->op);
-
- drm_gem_object_unreference_unlocked(obj);
-
- return ret;
-}
-
static int ioctl_gem_info(struct drm_device *dev, void *data,
struct drm_file *file_priv)
{
DRM_AUTH | DRM_MASTER | DRM_ROOT_ONLY),
DRM_IOCTL_DEF_DRV(OMAP_GEM_NEW, ioctl_gem_new,
DRM_AUTH | DRM_RENDER_ALLOW),
- DRM_IOCTL_DEF_DRV(OMAP_GEM_CPU_PREP, ioctl_gem_cpu_prep,
+ /* Deprecated, to be removed. */
+ DRM_IOCTL_DEF_DRV(OMAP_GEM_CPU_PREP, drm_noop,
DRM_AUTH | DRM_RENDER_ALLOW),
- DRM_IOCTL_DEF_DRV(OMAP_GEM_CPU_FINI, ioctl_gem_cpu_fini,
+ /* Deprecated, to be removed. */
+ DRM_IOCTL_DEF_DRV(OMAP_GEM_CPU_FINI, drm_noop,
DRM_AUTH | DRM_RENDER_ALLOW),
DRM_IOCTL_DEF_DRV(OMAP_GEM_INFO, ioctl_gem_info,
DRM_AUTH | DRM_RENDER_ALLOW),
.owner = THIS_MODULE,
.open = drm_open,
.unlocked_ioctl = drm_ioctl,
+ .compat_ioctl = drm_compat_ioctl,
.release = drm_release,
.mmap = omap_gem_mmap,
.poll = drm_poll,
.patchlevel = DRIVER_PATCHLEVEL,
};
+static const struct soc_device_attribute omapdrm_soc_devices[] = {
+ { .family = "OMAP3", .data = (void *)0x3430 },
+ { .family = "OMAP4", .data = (void *)0x4430 },
+ { .family = "OMAP5", .data = (void *)0x5430 },
+ { .family = "DRA7", .data = (void *)0x0752 },
+ { /* sentinel */ }
+};
+
static int pdev_probe(struct platform_device *pdev)
{
- struct omap_drm_platform_data *pdata = pdev->dev.platform_data;
+ const struct soc_device_attribute *soc;
struct omap_drm_private *priv;
struct drm_device *ddev;
unsigned int i;
priv->dispc_ops = dispc_get_ops();
- priv->omaprev = pdata->omaprev;
+ soc = soc_device_match(omapdrm_soc_devices);
+ priv->omaprev = soc ? (unsigned int)soc->data : 0;
priv->wq = alloc_ordered_workqueue("omapdrm", 0);
- init_waitqueue_head(&priv->commit.wait);
- spin_lock_init(&priv->commit.lock);
spin_lock_init(&priv->list_lock);
INIT_LIST_HEAD(&priv->obj_list);
#define __OMAP_DRV_H__
#include <linux/module.h>
-#include <linux/platform_data/omap_drm.h>
#include <linux/types.h>
-#include <linux/wait.h>
+#include <linux/workqueue.h>
#include <drm/drmP.h>
#include <drm/drm_crtc_helper.h>
struct omap_drm_usergart;
-/* parameters which describe (unrotated) coordinates of scanout within a fb: */
-struct omap_drm_window {
- uint32_t rotation;
- int32_t crtc_x, crtc_y; /* signed because can be offscreen */
- uint32_t crtc_w, crtc_h;
- uint32_t src_x, src_y;
- uint32_t src_w, src_h;
-};
-
/* For KMS code that needs to wait for a certain # of IRQs:
*/
struct omap_irq_wait;
spinlock_t wait_lock; /* protects the wait_list */
struct list_head wait_list; /* list of omap_irq_wait */
uint32_t irq_mask; /* enabled irqs in addition to wait_list */
-
- /* atomic commit */
- struct {
- wait_queue_head_t wait;
- u32 pending;
- spinlock_t lock; /* Protects commit.pending */
- } commit;
};
struct drm_connector *connector);
bool omap_connector_get_hdmi_mode(struct drm_connector *connector);
-uint32_t omap_framebuffer_get_formats(uint32_t *pixel_formats,
- uint32_t max_formats, enum omap_color_mode supported_modes);
struct drm_framebuffer *omap_framebuffer_create(struct drm_device *dev,
struct drm_file *file, const struct drm_mode_fb_cmd2 *mode_cmd);
struct drm_framebuffer *omap_framebuffer_init(struct drm_device *dev,
int omap_framebuffer_pin(struct drm_framebuffer *fb);
void omap_framebuffer_unpin(struct drm_framebuffer *fb);
void omap_framebuffer_update_scanout(struct drm_framebuffer *fb,
- struct omap_drm_window *win, struct omap_overlay_info *info);
+ struct drm_plane_state *state, struct omap_overlay_info *info);
struct drm_connector *omap_framebuffer_get_next_connector(
struct drm_framebuffer *fb, struct drm_connector *from);
bool omap_framebuffer_supports_rotation(struct drm_framebuffer *fb);
int omap_gem_mmap_obj(struct drm_gem_object *obj,
struct vm_area_struct *vma);
int omap_gem_fault(struct vm_fault *vmf);
-int omap_gem_op_start(struct drm_gem_object *obj, enum omap_gem_op op);
-int omap_gem_op_finish(struct drm_gem_object *obj, enum omap_gem_op op);
-int omap_gem_op_sync(struct drm_gem_object *obj, enum omap_gem_op op);
-int omap_gem_op_async(struct drm_gem_object *obj, enum omap_gem_op op,
- void (*fxn)(void *arg), void *arg);
int omap_gem_roll(struct drm_gem_object *obj, uint32_t roll);
-void omap_gem_cpu_sync(struct drm_gem_object *obj, int pgoff);
-void omap_gem_dma_sync(struct drm_gem_object *obj,
+void omap_gem_cpu_sync_page(struct drm_gem_object *obj, int pgoff);
+void omap_gem_dma_sync_buffer(struct drm_gem_object *obj,
enum dma_data_direction dir);
-int omap_gem_get_paddr(struct drm_gem_object *obj,
- dma_addr_t *paddr, bool remap);
-void omap_gem_put_paddr(struct drm_gem_object *obj);
+int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr);
+void omap_gem_unpin(struct drm_gem_object *obj);
int omap_gem_get_pages(struct drm_gem_object *obj, struct page ***pages,
bool remap);
int omap_gem_put_pages(struct drm_gem_object *obj);
uint32_t omap_gem_flags(struct drm_gem_object *obj);
-int omap_gem_rotated_paddr(struct drm_gem_object *obj, uint32_t orient,
- int x, int y, dma_addr_t *paddr);
+int omap_gem_rotated_dma_addr(struct drm_gem_object *obj, uint32_t orient,
+ int x, int y, dma_addr_t *dma_addr);
uint64_t omap_gem_mmap_offset(struct drm_gem_object *obj);
size_t omap_gem_mmap_size(struct drm_gem_object *obj);
int omap_gem_tiled_stride(struct drm_gem_object *obj, uint32_t orient);
* framebuffer funcs
*/
-/* DSS to DRM formats mapping */
-static const struct {
- enum omap_color_mode dss_format;
- uint32_t pixel_format;
-} formats[] = {
+static const u32 formats[] = {
/* 16bpp [A]RGB: */
- { OMAP_DSS_COLOR_RGB16, DRM_FORMAT_RGB565 }, /* RGB16-565 */
- { OMAP_DSS_COLOR_RGB12U, DRM_FORMAT_RGBX4444 }, /* RGB12x-4444 */
- { OMAP_DSS_COLOR_RGBX16, DRM_FORMAT_XRGB4444 }, /* xRGB12-4444 */
- { OMAP_DSS_COLOR_RGBA16, DRM_FORMAT_RGBA4444 }, /* RGBA12-4444 */
- { OMAP_DSS_COLOR_ARGB16, DRM_FORMAT_ARGB4444 }, /* ARGB16-4444 */
- { OMAP_DSS_COLOR_XRGB16_1555, DRM_FORMAT_XRGB1555 }, /* xRGB15-1555 */
- { OMAP_DSS_COLOR_ARGB16_1555, DRM_FORMAT_ARGB1555 }, /* ARGB16-1555 */
+ DRM_FORMAT_RGB565, /* RGB16-565 */
+ DRM_FORMAT_RGBX4444, /* RGB12x-4444 */
+ DRM_FORMAT_XRGB4444, /* xRGB12-4444 */
+ DRM_FORMAT_RGBA4444, /* RGBA12-4444 */
+ DRM_FORMAT_ARGB4444, /* ARGB16-4444 */
+ DRM_FORMAT_XRGB1555, /* xRGB15-1555 */
+ DRM_FORMAT_ARGB1555, /* ARGB16-1555 */
/* 24bpp RGB: */
- { OMAP_DSS_COLOR_RGB24P, DRM_FORMAT_RGB888 }, /* RGB24-888 */
+ DRM_FORMAT_RGB888, /* RGB24-888 */
/* 32bpp [A]RGB: */
- { OMAP_DSS_COLOR_RGBX32, DRM_FORMAT_RGBX8888 }, /* RGBx24-8888 */
- { OMAP_DSS_COLOR_RGB24U, DRM_FORMAT_XRGB8888 }, /* xRGB24-8888 */
- { OMAP_DSS_COLOR_RGBA32, DRM_FORMAT_RGBA8888 }, /* RGBA32-8888 */
- { OMAP_DSS_COLOR_ARGB32, DRM_FORMAT_ARGB8888 }, /* ARGB32-8888 */
+ DRM_FORMAT_RGBX8888, /* RGBx24-8888 */
+ DRM_FORMAT_XRGB8888, /* xRGB24-8888 */
+ DRM_FORMAT_RGBA8888, /* RGBA32-8888 */
+ DRM_FORMAT_ARGB8888, /* ARGB32-8888 */
/* YUV: */
- { OMAP_DSS_COLOR_NV12, DRM_FORMAT_NV12 },
- { OMAP_DSS_COLOR_YUV2, DRM_FORMAT_YUYV },
- { OMAP_DSS_COLOR_UYVY, DRM_FORMAT_UYVY },
+ DRM_FORMAT_NV12,
+ DRM_FORMAT_YUYV,
+ DRM_FORMAT_UYVY,
};
-/* convert from overlay's pixel formats bitmask to an array of fourcc's */
-uint32_t omap_framebuffer_get_formats(uint32_t *pixel_formats,
- uint32_t max_formats, enum omap_color_mode supported_modes)
-{
- uint32_t nformats = 0;
- int i = 0;
-
- for (i = 0; i < ARRAY_SIZE(formats) && nformats < max_formats; i++)
- if (formats[i].dss_format & supported_modes)
- pixel_formats[nformats++] = formats[i].pixel_format;
-
- return nformats;
-}
-
/* per-plane info for the fb: */
struct plane {
struct drm_gem_object *bo;
uint32_t pitch;
uint32_t offset;
- dma_addr_t paddr;
+ dma_addr_t dma_addr;
};
#define to_omap_framebuffer(x) container_of(x, struct omap_framebuffer, base)
struct drm_framebuffer base;
int pin_count;
const struct drm_format_info *format;
- enum omap_color_mode dss_format;
struct plane planes[2];
- /* lock for pinning (pin_count and planes.paddr) */
+ /* lock for pinning (pin_count and planes.dma_addr) */
struct mutex lock;
};
+ (x * format->cpp[n] / (n == 0 ? 1 : format->hsub))
+ (y * plane->pitch / (n == 0 ? 1 : format->vsub));
- return plane->paddr + offset;
+ return plane->dma_addr + offset;
}
bool omap_framebuffer_supports_rotation(struct drm_framebuffer *fb)
return omap_gem_flags(plane->bo) & OMAP_BO_TILED;
}
+/* Note: DRM rotates counter-clockwise, TILER & DSS rotates clockwise */
+static uint32_t drm_rotation_to_tiler(unsigned int drm_rot)
+{
+ uint32_t orient;
+
+ switch (drm_rot & DRM_MODE_ROTATE_MASK) {
+ default:
+ case DRM_MODE_ROTATE_0:
+ orient = 0;
+ break;
+ case DRM_MODE_ROTATE_90:
+ orient = MASK_XY_FLIP | MASK_X_INVERT;
+ break;
+ case DRM_MODE_ROTATE_180:
+ orient = MASK_X_INVERT | MASK_Y_INVERT;
+ break;
+ case DRM_MODE_ROTATE_270:
+ orient = MASK_XY_FLIP | MASK_Y_INVERT;
+ break;
+ }
+
+ if (drm_rot & DRM_MODE_REFLECT_X)
+ orient ^= MASK_X_INVERT;
+
+ if (drm_rot & DRM_MODE_REFLECT_Y)
+ orient ^= MASK_Y_INVERT;
+
+ return orient;
+}
+
/* update ovl info for scanout, handles cases of multi-planar fb's, etc.
*/
void omap_framebuffer_update_scanout(struct drm_framebuffer *fb,
- struct omap_drm_window *win, struct omap_overlay_info *info)
+ struct drm_plane_state *state, struct omap_overlay_info *info)
{
struct omap_framebuffer *omap_fb = to_omap_framebuffer(fb);
const struct drm_format_info *format = omap_fb->format;
struct plane *plane = &omap_fb->planes[0];
uint32_t x, y, orient = 0;
- info->color_mode = omap_fb->dss_format;
+ info->fourcc = fb->format->format;
- info->pos_x = win->crtc_x;
- info->pos_y = win->crtc_y;
- info->out_width = win->crtc_w;
- info->out_height = win->crtc_h;
- info->width = win->src_w;
- info->height = win->src_h;
+ info->pos_x = state->crtc_x;
+ info->pos_y = state->crtc_y;
+ info->out_width = state->crtc_w;
+ info->out_height = state->crtc_h;
+ info->width = state->src_w >> 16;
+ info->height = state->src_h >> 16;
- x = win->src_x;
- y = win->src_y;
+ /* DSS driver wants the w & h in rotated orientation */
+ if (drm_rotation_90_or_270(state->rotation))
+ swap(info->width, info->height);
- if (omap_gem_flags(plane->bo) & OMAP_BO_TILED) {
- uint32_t w = win->src_w;
- uint32_t h = win->src_h;
+ x = state->src_x >> 16;
+ y = state->src_y >> 16;
- switch (win->rotation & DRM_MODE_ROTATE_MASK) {
- default:
- dev_err(fb->dev->dev, "invalid rotation: %02x",
- (uint32_t)win->rotation);
- /* fallthru to default to no rotation */
- case 0:
- case DRM_MODE_ROTATE_0:
- orient = 0;
- break;
- case DRM_MODE_ROTATE_90:
- orient = MASK_XY_FLIP | MASK_X_INVERT;
- break;
- case DRM_MODE_ROTATE_180:
- orient = MASK_X_INVERT | MASK_Y_INVERT;
- break;
- case DRM_MODE_ROTATE_270:
- orient = MASK_XY_FLIP | MASK_Y_INVERT;
- break;
+ if (omap_gem_flags(plane->bo) & OMAP_BO_TILED) {
+ uint32_t w = state->src_w >> 16;
+ uint32_t h = state->src_h >> 16;
+
+ orient = drm_rotation_to_tiler(state->rotation);
+
+ /*
+ * omap_gem_rotated_paddr() wants the x & y in tiler units.
+ * Usually tiler unit size is the same as the pixel size, except
+ * for YUV422 formats, for which the tiler unit size is 32 bits
+ * and pixel size is 16 bits.
+ */
+ if (fb->format->format == DRM_FORMAT_UYVY ||
+ fb->format->format == DRM_FORMAT_YUYV) {
+ x /= 2;
+ w /= 2;
}
- if (win->rotation & DRM_MODE_REFLECT_X)
- orient ^= MASK_X_INVERT;
-
- if (win->rotation & DRM_MODE_REFLECT_Y)
- orient ^= MASK_Y_INVERT;
-
- /* adjust x,y offset for flip/invert: */
- if (orient & MASK_XY_FLIP)
- swap(w, h);
+ /* adjust x,y offset for invert: */
if (orient & MASK_Y_INVERT)
y += h - 1;
if (orient & MASK_X_INVERT)
x += w - 1;
- omap_gem_rotated_paddr(plane->bo, orient, x, y, &info->paddr);
+ /* Note: x and y are in TILER units, not pixels */
+ omap_gem_rotated_dma_addr(plane->bo, orient, x, y,
+ &info->paddr);
info->rotation_type = OMAP_DSS_ROT_TILER;
+ info->rotation = state->rotation ?: DRM_MODE_ROTATE_0;
+ /* Note: stride in TILER units, not pixels */
info->screen_width = omap_gem_tiled_stride(plane->bo, orient);
} else {
- switch (win->rotation & DRM_MODE_ROTATE_MASK) {
+ switch (state->rotation & DRM_MODE_ROTATE_MASK) {
case 0:
case DRM_MODE_ROTATE_0:
/* OK */
default:
dev_warn(fb->dev->dev,
"rotation '%d' ignored for non-tiled fb\n",
- win->rotation);
- win->rotation = 0;
+ state->rotation);
break;
}
info->paddr = get_linear_addr(plane, format, 0, x, y);
- info->rotation_type = OMAP_DSS_ROT_DMA;
+ info->rotation_type = OMAP_DSS_ROT_NONE;
+ info->rotation = DRM_MODE_ROTATE_0;
info->screen_width = plane->pitch;
}
/* convert to pixels: */
info->screen_width /= format->cpp[0];
- if (omap_fb->dss_format == OMAP_DSS_COLOR_NV12) {
+ if (fb->format->format == DRM_FORMAT_NV12) {
plane = &omap_fb->planes[1];
if (info->rotation_type == OMAP_DSS_ROT_TILER) {
WARN_ON(!(omap_gem_flags(plane->bo) & OMAP_BO_TILED));
- omap_gem_rotated_paddr(plane->bo, orient,
- x/2, y/2, &info->p_uv_addr);
+ omap_gem_rotated_dma_addr(plane->bo, orient, x/2, y/2,
+ &info->p_uv_addr);
} else {
info->p_uv_addr = get_linear_addr(plane, format, 1, x, y);
}
for (i = 0; i < n; i++) {
struct plane *plane = &omap_fb->planes[i];
- ret = omap_gem_get_paddr(plane->bo, &plane->paddr, true);
+ ret = omap_gem_pin(plane->bo, &plane->dma_addr);
if (ret)
goto fail;
- omap_gem_dma_sync(plane->bo, DMA_TO_DEVICE);
+ omap_gem_dma_sync_buffer(plane->bo, DMA_TO_DEVICE);
}
omap_fb->pin_count++;
fail:
for (i--; i >= 0; i--) {
struct plane *plane = &omap_fb->planes[i];
- omap_gem_put_paddr(plane->bo);
- plane->paddr = 0;
+ omap_gem_unpin(plane->bo);
+ plane->dma_addr = 0;
}
mutex_unlock(&omap_fb->lock);
for (i = 0; i < n; i++) {
struct plane *plane = &omap_fb->planes[i];
- omap_gem_put_paddr(plane->bo);
- plane->paddr = 0;
+ omap_gem_unpin(plane->bo);
+ plane->dma_addr = 0;
}
mutex_unlock(&omap_fb->lock);
const struct drm_format_info *format = NULL;
struct omap_framebuffer *omap_fb = NULL;
struct drm_framebuffer *fb = NULL;
- enum omap_color_mode dss_format = 0;
unsigned int pitch = mode_cmd->pitches[0];
int ret, i;
format = drm_format_info(mode_cmd->pixel_format);
for (i = 0; i < ARRAY_SIZE(formats); i++) {
- if (formats[i].pixel_format == mode_cmd->pixel_format) {
- dss_format = formats[i].dss_format;
+ if (formats[i] == mode_cmd->pixel_format)
break;
- }
}
- if (!format || !dss_format) {
+ if (!format || i == ARRAY_SIZE(formats)) {
dev_dbg(dev->dev, "unsupported pixel format: %4.4s\n",
(char *)&mode_cmd->pixel_format);
ret = -EINVAL;
fb = &omap_fb->base;
omap_fb->format = format;
- omap_fb->dss_format = dss_format;
mutex_init(&omap_fb->lock);
/*
if (size > omap_gem_mmap_size(bos[i]) - mode_cmd->offsets[i]) {
dev_dbg(dev->dev,
- "provided buffer object is too small! %d < %d\n",
+ "provided buffer object is too small! %zu < %d\n",
bos[i]->size - mode_cmd->offsets[i], size);
ret = -EINVAL;
goto fail;
plane->bo = bos[i];
plane->offset = mode_cmd->offsets[i];
plane->pitch = pitch;
- plane->paddr = 0;
+ plane->dma_addr = 0;
}
drm_helper_mode_fill_fb_struct(dev, fb, mode_cmd);
union omap_gem_size gsize;
struct fb_info *fbi = NULL;
struct drm_mode_fb_cmd2 mode_cmd = {0};
- dma_addr_t paddr;
+ dma_addr_t dma_addr;
int ret;
sizes->surface_bpp = 32;
* to it). Then we just need to be sure that we are able to re-
* pin it in case of an opps.
*/
- ret = omap_gem_get_paddr(fbdev->bo, &paddr, true);
+ ret = omap_gem_pin(fbdev->bo, &dma_addr);
if (ret) {
- dev_err(dev->dev,
- "could not map (paddr)! Skipping framebuffer alloc\n");
+ dev_err(dev->dev, "could not pin framebuffer\n");
ret = -ENOMEM;
goto fail;
}
drm_fb_helper_fill_fix(fbi, fb->pitches[0], fb->format->depth);
drm_fb_helper_fill_var(fbi, helper, sizes->fb_width, sizes->fb_height);
- dev->mode_config.fb_base = paddr;
+ dev->mode_config.fb_base = dma_addr;
fbi->screen_base = omap_gem_vaddr(fbdev->bo);
fbi->screen_size = fbdev->bo->size;
- fbi->fix.smem_start = paddr;
+ fbi->fix.smem_start = dma_addr;
fbi->fix.smem_len = fbdev->bo->size;
/* if we have DMM, then we can use it for scrolling by just
fbdev = to_omap_fbdev(priv->fbdev);
- /* release the ref taken in omap_fbdev_create() */
- omap_gem_put_paddr(fbdev->bo);
+ /* unpin the GEM object pinned in omap_fbdev_create() */
+ omap_gem_unpin(fbdev->bo);
/* this will free the backing object */
if (fbdev->fb)
uint32_t roll;
/**
- * paddr contains the buffer DMA address. It is valid for
+ * dma_addr contains the buffer DMA address. It is valid for
*
* - buffers allocated through the DMA mapping API (with the
* OMAP_BO_MEM_DMA_API flag set)
* - buffers imported from dmabuf (with the OMAP_BO_MEM_DMABUF flag set)
* if they are physically contiguous (when sgt->orig_nents == 1)
*
- * - buffers mapped through the TILER when paddr_cnt is not zero, in
+ * - buffers mapped through the TILER when dma_addr_cnt is not zero, in
* which case the DMA address points to the TILER aperture
*
* Physically contiguous buffers have their DMA address equal to the
* physical address as we don't remap those buffers through the TILER.
*
* Buffers mapped to the TILER have their DMA address pointing to the
- * TILER aperture. As TILER mappings are refcounted (through paddr_cnt)
- * the DMA address must be accessed through omap_get_get_paddr() to
- * ensure that the mapping won't disappear unexpectedly. References must
- * be released with omap_gem_put_paddr().
+ * TILER aperture. As TILER mappings are refcounted (through
+ * dma_addr_cnt) the DMA address must be accessed through omap_gem_pin()
+ * to ensure that the mapping won't disappear unexpectedly. References
+ * must be released with omap_gem_unpin().
*/
- dma_addr_t paddr;
+ dma_addr_t dma_addr;
/**
- * # of users of paddr
+ * # of users of dma_addr
*/
- uint32_t paddr_cnt;
+ uint32_t dma_addr_cnt;
/**
* If the buffer has been imported from a dmabuf the OMAP_DB_DMABUF flag
struct page **pages;
/** addresses corresponding to pages in above array */
- dma_addr_t *addrs;
+ dma_addr_t *dma_addrs;
/**
* Virtual address, if mapped.
*/
void *vaddr;
-
- /**
- * sync-object allocated on demand (if needed)
- *
- * Per-buffer sync-object for tracking pending and completed hw/dma
- * read and write operations.
- */
- struct {
- uint32_t write_pending;
- uint32_t write_complete;
- uint32_t read_pending;
- uint32_t read_complete;
- } *sync;
};
#define to_omap_bo(x) container_of(x, struct omap_gem_object, base)
#define NUM_USERGART_ENTRIES 2
struct omap_drm_usergart_entry {
struct tiler_block *block; /* the reserved tiler block */
- dma_addr_t paddr;
+ dma_addr_t dma_addr;
struct drm_gem_object *obj; /* the current pinned obj */
pgoff_t obj_pgoff; /* page offset of obj currently
mapped in */
size_t size = PAGE_SIZE * n;
loff_t off = mmap_offset(obj) +
(entry->obj_pgoff << PAGE_SHIFT);
- const int m = 1 + ((omap_obj->width << fmt) / PAGE_SIZE);
+ const int m = DIV_ROUND_UP(omap_obj->width << fmt, PAGE_SIZE);
if (m > 1) {
int i;
for (i = 0; i < npages; i++) {
addrs[i] = dma_map_page(dev->dev, pages[i],
- 0, PAGE_SIZE, DMA_BIDIRECTIONAL);
+ 0, PAGE_SIZE, DMA_TO_DEVICE);
if (dma_mapping_error(dev->dev, addrs[i])) {
dev_warn(dev->dev,
for (i = i - 1; i >= 0; --i) {
dma_unmap_page(dev->dev, addrs[i],
- PAGE_SIZE, DMA_BIDIRECTIONAL);
+ PAGE_SIZE, DMA_TO_DEVICE);
}
ret = -ENOMEM;
}
}
- omap_obj->addrs = addrs;
+ omap_obj->dma_addrs = addrs;
omap_obj->pages = pages;
return 0;
static void omap_gem_detach_pages(struct drm_gem_object *obj)
{
struct omap_gem_object *omap_obj = to_omap_bo(obj);
+ unsigned int npages = obj->size >> PAGE_SHIFT;
+ unsigned int i;
- /* for non-cached buffers, ensure the new pages are clean because
- * DSS, GPU, etc. are not cache coherent:
- */
- if (omap_obj->flags & (OMAP_BO_WC|OMAP_BO_UNCACHED)) {
- int i, npages = obj->size >> PAGE_SHIFT;
- for (i = 0; i < npages; i++) {
- if (omap_obj->addrs[i])
- dma_unmap_page(obj->dev->dev,
- omap_obj->addrs[i],
- PAGE_SIZE, DMA_BIDIRECTIONAL);
- }
+ for (i = 0; i < npages; i++) {
+ if (omap_obj->dma_addrs[i])
+ dma_unmap_page(obj->dev->dev, omap_obj->dma_addrs[i],
+ PAGE_SIZE, DMA_TO_DEVICE);
}
- kfree(omap_obj->addrs);
- omap_obj->addrs = NULL;
+ kfree(omap_obj->dma_addrs);
+ omap_obj->dma_addrs = NULL;
drm_gem_put_pages(obj, omap_obj->pages, true, false);
omap_obj->pages = NULL;
pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
if (omap_obj->pages) {
- omap_gem_cpu_sync(obj, pgoff);
+ omap_gem_cpu_sync_page(obj, pgoff);
pfn = page_to_pfn(omap_obj->pages[pgoff]);
} else {
BUG_ON(!is_contiguous(omap_obj));
- pfn = (omap_obj->paddr >> PAGE_SHIFT) + pgoff;
+ pfn = (omap_obj->dma_addr >> PAGE_SHIFT) + pgoff;
}
VERB("Inserting %p pfn %lx, pa %lx", (void *)vmf->address,
* into account in some of the math, so figure out virtual stride
* in pages
*/
- const int m = 1 + ((omap_obj->width << fmt) / PAGE_SIZE);
+ const int m = DIV_ROUND_UP(omap_obj->width << fmt, PAGE_SIZE);
/* We don't use vmf->pgoff since that has the fake offset: */
pgoff = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
return ret;
}
- pfn = entry->paddr >> PAGE_SHIFT;
+ pfn = entry->dma_addr >> PAGE_SHIFT;
VERB("Inserting %p pfn %lx, pa %lx", (void *)vmf->address,
pfn, pfn << PAGE_SHIFT);
* Memory Management & DMA Sync
*/
-/**
- * shmem buffers that are mapped cached can simulate coherency via using
- * page faulting to keep track of dirty pages
+/*
+ * shmem buffers that are mapped cached are not coherent.
+ *
+ * We keep track of dirty pages using page faulting to perform cache management.
+ * When a page is mapped to the CPU in read/write mode the device can't access
+ * it and omap_obj->dma_addrs[i] is NULL. When a page is mapped to the device
+ * the omap_obj->dma_addrs[i] is set to the DMA address, and the page is
+ * unmapped from the CPU.
*/
static inline bool is_cached_coherent(struct drm_gem_object *obj)
{
struct omap_gem_object *omap_obj = to_omap_bo(obj);
- return (omap_obj->flags & OMAP_BO_MEM_SHMEM) &&
- ((omap_obj->flags & OMAP_BO_CACHE_MASK) == OMAP_BO_CACHED);
+ return !((omap_obj->flags & OMAP_BO_MEM_SHMEM) &&
+ ((omap_obj->flags & OMAP_BO_CACHE_MASK) == OMAP_BO_CACHED));
}
/* Sync the buffer for CPU access.. note pages should already be
* attached, ie. omap_gem_get_pages()
*/
-void omap_gem_cpu_sync(struct drm_gem_object *obj, int pgoff)
+void omap_gem_cpu_sync_page(struct drm_gem_object *obj, int pgoff)
{
struct drm_device *dev = obj->dev;
struct omap_gem_object *omap_obj = to_omap_bo(obj);
- if (is_cached_coherent(obj) && omap_obj->addrs[pgoff]) {
- dma_unmap_page(dev->dev, omap_obj->addrs[pgoff],
- PAGE_SIZE, DMA_BIDIRECTIONAL);
- omap_obj->addrs[pgoff] = 0;
+ if (is_cached_coherent(obj))
+ return;
+
+ if (omap_obj->dma_addrs[pgoff]) {
+ dma_unmap_page(dev->dev, omap_obj->dma_addrs[pgoff],
+ PAGE_SIZE, DMA_TO_DEVICE);
+ omap_obj->dma_addrs[pgoff] = 0;
}
}
/* sync the buffer for DMA access */
-void omap_gem_dma_sync(struct drm_gem_object *obj,
+void omap_gem_dma_sync_buffer(struct drm_gem_object *obj,
enum dma_data_direction dir)
{
struct drm_device *dev = obj->dev;
struct omap_gem_object *omap_obj = to_omap_bo(obj);
+ int i, npages = obj->size >> PAGE_SHIFT;
+ struct page **pages = omap_obj->pages;
+ bool dirty = false;
- if (is_cached_coherent(obj)) {
- int i, npages = obj->size >> PAGE_SHIFT;
- struct page **pages = omap_obj->pages;
- bool dirty = false;
-
- for (i = 0; i < npages; i++) {
- if (!omap_obj->addrs[i]) {
- dma_addr_t addr;
-
- addr = dma_map_page(dev->dev, pages[i], 0,
- PAGE_SIZE, DMA_BIDIRECTIONAL);
+ if (is_cached_coherent(obj))
+ return;
- if (dma_mapping_error(dev->dev, addr)) {
- dev_warn(dev->dev,
- "%s: failed to map page\n",
- __func__);
- break;
- }
+ for (i = 0; i < npages; i++) {
+ if (!omap_obj->dma_addrs[i]) {
+ dma_addr_t addr;
- dirty = true;
- omap_obj->addrs[i] = addr;
+ addr = dma_map_page(dev->dev, pages[i], 0,
+ PAGE_SIZE, dir);
+ if (dma_mapping_error(dev->dev, addr)) {
+ dev_warn(dev->dev, "%s: failed to map page\n",
+ __func__);
+ break;
}
- }
- if (dirty) {
- unmap_mapping_range(obj->filp->f_mapping, 0,
- omap_gem_mmap_size(obj), 1);
+ dirty = true;
+ omap_obj->dma_addrs[i] = addr;
}
}
+
+ if (dirty) {
+ unmap_mapping_range(obj->filp->f_mapping, 0,
+ omap_gem_mmap_size(obj), 1);
+ }
}
-/* Get physical address for DMA.. if 'remap' is true, and the buffer is not
- * already contiguous, remap it to pin in physically contiguous memory.. (ie.
- * map in TILER)
+/**
+ * omap_gem_pin() - Pin a GEM object in memory
+ * @obj: the GEM object
+ * @dma_addr: the DMA address
+ *
+ * Pin the given GEM object in memory and fill the dma_addr pointer with the
+ * object's DMA address. If the buffer is not physically contiguous it will be
+ * remapped through the TILER to provide a contiguous view.
+ *
+ * Pins are reference-counted, calling this function multiple times is allowed
+ * as long the corresponding omap_gem_unpin() calls are balanced.
+ *
+ * Return 0 on success or a negative error code otherwise.
*/
-int omap_gem_get_paddr(struct drm_gem_object *obj,
- dma_addr_t *paddr, bool remap)
+int omap_gem_pin(struct drm_gem_object *obj, dma_addr_t *dma_addr)
{
struct omap_drm_private *priv = obj->dev->dev_private;
struct omap_gem_object *omap_obj = to_omap_bo(obj);
mutex_lock(&obj->dev->struct_mutex);
- if (!is_contiguous(omap_obj) && remap && priv->has_dmm) {
- if (omap_obj->paddr_cnt == 0) {
+ if (!is_contiguous(omap_obj) && priv->has_dmm) {
+ if (omap_obj->dma_addr_cnt == 0) {
struct page **pages;
uint32_t npages = obj->size >> PAGE_SHIFT;
enum tiler_fmt fmt = gem2fmt(omap_obj->flags);
goto fail;
}
- omap_obj->paddr = tiler_ssptr(block);
+ omap_obj->dma_addr = tiler_ssptr(block);
omap_obj->block = block;
- DBG("got paddr: %pad", &omap_obj->paddr);
+ DBG("got dma address: %pad", &omap_obj->dma_addr);
}
- omap_obj->paddr_cnt++;
+ omap_obj->dma_addr_cnt++;
- *paddr = omap_obj->paddr;
+ *dma_addr = omap_obj->dma_addr;
} else if (is_contiguous(omap_obj)) {
- *paddr = omap_obj->paddr;
+ *dma_addr = omap_obj->dma_addr;
} else {
ret = -EINVAL;
goto fail;
return ret;
}
-/* Release physical address, when DMA is no longer being performed.. this
- * could potentially unpin and unmap buffers from TILER
+/**
+ * omap_gem_unpin() - Unpin a GEM object from memory
+ * @obj: the GEM object
+ *
+ * Unpin the given GEM object previously pinned with omap_gem_pin(). Pins are
+ * reference-counted, the actualy unpin will only be performed when the number
+ * of calls to this function matches the number of calls to omap_gem_pin().
*/
-void omap_gem_put_paddr(struct drm_gem_object *obj)
+void omap_gem_unpin(struct drm_gem_object *obj)
{
struct omap_gem_object *omap_obj = to_omap_bo(obj);
int ret;
mutex_lock(&obj->dev->struct_mutex);
- if (omap_obj->paddr_cnt > 0) {
- omap_obj->paddr_cnt--;
- if (omap_obj->paddr_cnt == 0) {
+ if (omap_obj->dma_addr_cnt > 0) {
+ omap_obj->dma_addr_cnt--;
+ if (omap_obj->dma_addr_cnt == 0) {
ret = tiler_unpin(omap_obj->block);
if (ret) {
dev_err(obj->dev->dev,
dev_err(obj->dev->dev,
"could not release unmap: %d\n", ret);
}
- omap_obj->paddr = 0;
+ omap_obj->dma_addr = 0;
omap_obj->block = NULL;
}
}
* specified orientation and x,y offset from top-left corner of buffer
* (only valid for tiled 2d buffers)
*/
-int omap_gem_rotated_paddr(struct drm_gem_object *obj, uint32_t orient,
- int x, int y, dma_addr_t *paddr)
+int omap_gem_rotated_dma_addr(struct drm_gem_object *obj, uint32_t orient,
+ int x, int y, dma_addr_t *dma_addr)
{
struct omap_gem_object *omap_obj = to_omap_bo(obj);
int ret = -EINVAL;
mutex_lock(&obj->dev->struct_mutex);
- if ((omap_obj->paddr_cnt > 0) && omap_obj->block &&
+ if ((omap_obj->dma_addr_cnt > 0) && omap_obj->block &&
(omap_obj->flags & OMAP_BO_TILED)) {
- *paddr = tiler_tsptr(omap_obj->block, orient, x, y);
+ *dma_addr = tiler_tsptr(omap_obj->block, orient, x, y);
ret = 0;
}
mutex_unlock(&obj->dev->struct_mutex);
* increasing the pin count (which we don't really do yet anyways,
* because we don't support swapping pages back out). And 'remap'
* might not be quite the right name, but I wanted to keep it working
- * similarly to omap_gem_get_paddr(). Note though that mutex is not
+ * similarly to omap_gem_pin(). Note though that mutex is not
* aquired if !remap (because this can be called in atomic ctxt),
- * but probably omap_gem_get_paddr() should be changed to work in the
+ * but probably omap_gem_unpin() should be changed to work in the
* same way. If !remap, a matching omap_gem_put_pages() call is not
* required (and should not be made).
*/
seq_printf(m, "%08x: %2d (%2d) %08llx %pad (%2d) %p %4d",
omap_obj->flags, obj->name, kref_read(&obj->refcount),
- off, &omap_obj->paddr, omap_obj->paddr_cnt,
+ off, &omap_obj->dma_addr, omap_obj->dma_addr_cnt,
omap_obj->vaddr, omap_obj->roll);
if (omap_obj->flags & OMAP_BO_TILED) {
area->p1.x, area->p1.y);
}
} else {
- seq_printf(m, " %d", obj->size);
+ seq_printf(m, " %zu", obj->size);
}
seq_printf(m, "\n");
}
#endif
-/* -----------------------------------------------------------------------------
- * Buffer Synchronization
- */
-
-static DEFINE_SPINLOCK(sync_lock);
-
-struct omap_gem_sync_waiter {
- struct list_head list;
- struct omap_gem_object *omap_obj;
- enum omap_gem_op op;
- uint32_t read_target, write_target;
- /* notify called w/ sync_lock held */
- void (*notify)(void *arg);
- void *arg;
-};
-
-/* list of omap_gem_sync_waiter.. the notify fxn gets called back when
- * the read and/or write target count is achieved which can call a user
- * callback (ex. to kick 3d and/or 2d), wakeup blocked task (prep for
- * cpu access), etc.
- */
-static LIST_HEAD(waiters);
-
-static inline bool is_waiting(struct omap_gem_sync_waiter *waiter)
-{
- struct omap_gem_object *omap_obj = waiter->omap_obj;
- if ((waiter->op & OMAP_GEM_READ) &&
- (omap_obj->sync->write_complete < waiter->write_target))
- return true;
- if ((waiter->op & OMAP_GEM_WRITE) &&
- (omap_obj->sync->read_complete < waiter->read_target))
- return true;
- return false;
-}
-
-/* macro for sync debug.. */
-#define SYNCDBG 0
-#define SYNC(fmt, ...) do { if (SYNCDBG) \
- pr_err("%s:%d: " fmt "\n", __func__, __LINE__, ##__VA_ARGS__); \
- } while (0)
-
-
-static void sync_op_update(void)
-{
- struct omap_gem_sync_waiter *waiter, *n;
- list_for_each_entry_safe(waiter, n, &waiters, list) {
- if (!is_waiting(waiter)) {
- list_del(&waiter->list);
- SYNC("notify: %p", waiter);
- waiter->notify(waiter->arg);
- kfree(waiter);
- }
- }
-}
-
-static inline int sync_op(struct drm_gem_object *obj,
- enum omap_gem_op op, bool start)
-{
- struct omap_gem_object *omap_obj = to_omap_bo(obj);
- int ret = 0;
-
- spin_lock(&sync_lock);
-
- if (!omap_obj->sync) {
- omap_obj->sync = kzalloc(sizeof(*omap_obj->sync), GFP_ATOMIC);
- if (!omap_obj->sync) {
- ret = -ENOMEM;
- goto unlock;
- }
- }
-
- if (start) {
- if (op & OMAP_GEM_READ)
- omap_obj->sync->read_pending++;
- if (op & OMAP_GEM_WRITE)
- omap_obj->sync->write_pending++;
- } else {
- if (op & OMAP_GEM_READ)
- omap_obj->sync->read_complete++;
- if (op & OMAP_GEM_WRITE)
- omap_obj->sync->write_complete++;
- sync_op_update();
- }
-
-unlock:
- spin_unlock(&sync_lock);
-
- return ret;
-}
-
-/* mark the start of read and/or write operation */
-int omap_gem_op_start(struct drm_gem_object *obj, enum omap_gem_op op)
-{
- return sync_op(obj, op, true);
-}
-
-int omap_gem_op_finish(struct drm_gem_object *obj, enum omap_gem_op op)
-{
- return sync_op(obj, op, false);
-}
-
-static DECLARE_WAIT_QUEUE_HEAD(sync_event);
-
-static void sync_notify(void *arg)
-{
- struct task_struct **waiter_task = arg;
- *waiter_task = NULL;
- wake_up_all(&sync_event);
-}
-
-int omap_gem_op_sync(struct drm_gem_object *obj, enum omap_gem_op op)
-{
- struct omap_gem_object *omap_obj = to_omap_bo(obj);
- int ret = 0;
- if (omap_obj->sync) {
- struct task_struct *waiter_task = current;
- struct omap_gem_sync_waiter *waiter =
- kzalloc(sizeof(*waiter), GFP_KERNEL);
-
- if (!waiter)
- return -ENOMEM;
-
- waiter->omap_obj = omap_obj;
- waiter->op = op;
- waiter->read_target = omap_obj->sync->read_pending;
- waiter->write_target = omap_obj->sync->write_pending;
- waiter->notify = sync_notify;
- waiter->arg = &waiter_task;
-
- spin_lock(&sync_lock);
- if (is_waiting(waiter)) {
- SYNC("waited: %p", waiter);
- list_add_tail(&waiter->list, &waiters);
- spin_unlock(&sync_lock);
- ret = wait_event_interruptible(sync_event,
- (waiter_task == NULL));
- spin_lock(&sync_lock);
- if (waiter_task) {
- SYNC("interrupted: %p", waiter);
- /* we were interrupted */
- list_del(&waiter->list);
- waiter_task = NULL;
- } else {
- /* freed in sync_op_update() */
- waiter = NULL;
- }
- }
- spin_unlock(&sync_lock);
- kfree(waiter);
- }
- return ret;
-}
-
-/* call fxn(arg), either synchronously or asynchronously if the op
- * is currently blocked.. fxn() can be called from any context
- *
- * (TODO for now fxn is called back from whichever context calls
- * omap_gem_op_finish().. but this could be better defined later
- * if needed)
- *
- * TODO more code in common w/ _sync()..
- */
-int omap_gem_op_async(struct drm_gem_object *obj, enum omap_gem_op op,
- void (*fxn)(void *arg), void *arg)
-{
- struct omap_gem_object *omap_obj = to_omap_bo(obj);
- if (omap_obj->sync) {
- struct omap_gem_sync_waiter *waiter =
- kzalloc(sizeof(*waiter), GFP_ATOMIC);
-
- if (!waiter)
- return -ENOMEM;
-
- waiter->omap_obj = omap_obj;
- waiter->op = op;
- waiter->read_target = omap_obj->sync->read_pending;
- waiter->write_target = omap_obj->sync->write_pending;
- waiter->notify = fxn;
- waiter->arg = arg;
-
- spin_lock(&sync_lock);
- if (is_waiting(waiter)) {
- SYNC("waited: %p", waiter);
- list_add_tail(&waiter->list, &waiters);
- spin_unlock(&sync_lock);
- return 0;
- }
-
- spin_unlock(&sync_lock);
-
- kfree(waiter);
- }
-
- /* no waiting.. */
- fxn(arg);
-
- return 0;
-}
-
/* -----------------------------------------------------------------------------
* Constructor & Destructor
*/
/* this means the object is still pinned.. which really should
* not happen. I think..
*/
- WARN_ON(omap_obj->paddr_cnt > 0);
+ WARN_ON(omap_obj->dma_addr_cnt > 0);
if (omap_obj->pages) {
if (omap_obj->flags & OMAP_BO_MEM_DMABUF)
if (omap_obj->flags & OMAP_BO_MEM_DMA_API) {
dma_free_wc(dev->dev, obj->size, omap_obj->vaddr,
- omap_obj->paddr);
+ omap_obj->dma_addr);
} else if (omap_obj->vaddr) {
vunmap(omap_obj->vaddr);
} else if (obj->import_attach) {
drm_prime_gem_destroy(obj, omap_obj->sgt);
}
- kfree(omap_obj->sync);
-
drm_gem_object_release(obj);
kfree(omap_obj);
/* Allocate memory if needed. */
if (flags & OMAP_BO_MEM_DMA_API) {
omap_obj->vaddr = dma_alloc_wc(dev->dev, size,
- &omap_obj->paddr,
+ &omap_obj->dma_addr,
GFP_KERNEL);
if (!omap_obj->vaddr)
goto err_release;
omap_obj->sgt = sgt;
if (sgt->orig_nents == 1) {
- omap_obj->paddr = sg_dma_address(sgt->sgl);
+ omap_obj->dma_addr = sg_dma_address(sgt->sgl);
} else {
/* Create pages list from sgt */
struct sg_page_iter iter;
i, j, PTR_ERR(block));
return;
}
- entry->paddr = tiler_ssptr(block);
+ entry->dma_addr = tiler_ssptr(block);
entry->block = block;
- DBG("%d:%d: %dx%d: paddr=%pad stride=%d", i, j, w, h,
- &entry->paddr,
+ DBG("%d:%d: %dx%d: dma_addr=%pad stride=%d", i, j, w, h,
+ &entry->dma_addr,
usergart[i].stride_pfn << PAGE_SHIFT);
}
}
{
struct drm_gem_object *obj = attachment->dmabuf->priv;
struct sg_table *sg;
- dma_addr_t paddr;
+ dma_addr_t dma_addr;
int ret;
sg = kzalloc(sizeof(*sg), GFP_KERNEL);
/* camera, etc, need physically contiguous.. but we need a
* better way to know this..
*/
- ret = omap_gem_get_paddr(obj, &paddr, true);
+ ret = omap_gem_pin(obj, &dma_addr);
if (ret)
goto out;
sg_init_table(sg->sgl, 1);
sg_dma_len(sg->sgl) = obj->size;
- sg_set_page(sg->sgl, pfn_to_page(PFN_DOWN(paddr)), obj->size, 0);
- sg_dma_address(sg->sgl) = paddr;
+ sg_set_page(sg->sgl, pfn_to_page(PFN_DOWN(dma_addr)), obj->size, 0);
+ sg_dma_address(sg->sgl) = dma_addr;
- /* this should be after _get_paddr() to ensure we have pages attached */
- omap_gem_dma_sync(obj, dir);
+ /* this must be after omap_gem_pin() to ensure we have pages attached */
+ omap_gem_dma_sync_buffer(obj, dir);
return sg;
out:
struct sg_table *sg, enum dma_data_direction dir)
{
struct drm_gem_object *obj = attachment->dmabuf->priv;
- omap_gem_put_paddr(obj);
+ omap_gem_unpin(obj);
sg_free_table(sg);
kfree(sg);
}
-static void omap_gem_dmabuf_release(struct dma_buf *buffer)
-{
- struct drm_gem_object *obj = buffer->priv;
- /* release reference that was taken when dmabuf was exported
- * in omap_gem_prime_set()..
- */
- drm_gem_object_unreference_unlocked(obj);
-}
-
-
static int omap_gem_dmabuf_begin_cpu_access(struct dma_buf *buffer,
enum dma_data_direction dir)
{
struct drm_gem_object *obj = buffer->priv;
struct page **pages;
omap_gem_get_pages(obj, &pages, false);
- omap_gem_cpu_sync(obj, page_num);
+ omap_gem_cpu_sync_page(obj, page_num);
return kmap_atomic(pages[page_num]);
}
struct drm_gem_object *obj = buffer->priv;
struct page **pages;
omap_gem_get_pages(obj, &pages, false);
- omap_gem_cpu_sync(obj, page_num);
+ omap_gem_cpu_sync_page(obj, page_num);
return kmap(pages[page_num]);
}
static struct dma_buf_ops omap_dmabuf_ops = {
.map_dma_buf = omap_gem_map_dma_buf,
.unmap_dma_buf = omap_gem_unmap_dma_buf,
- .release = omap_gem_dmabuf_release,
+ .release = drm_gem_dmabuf_release,
.begin_cpu_access = omap_gem_dmabuf_begin_cpu_access,
.end_cpu_access = omap_gem_dmabuf_end_cpu_access,
.map_atomic = omap_gem_dmabuf_kmap_atomic,
exp_info.flags = flags;
exp_info.priv = obj;
- return dma_buf_export(&exp_info);
+ return drm_gem_dmabuf_export(dev, &exp_info);
}
/* -----------------------------------------------------------------------------
get_dma_buf(dma_buf);
- sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
+ sgt = dma_buf_map_attachment(attach, DMA_TO_DEVICE);
if (IS_ERR(sgt)) {
ret = PTR_ERR(sgt);
goto fail_detach;
return obj;
fail_unmap:
- dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
+ dma_buf_unmap_attachment(attach, sgt, DMA_TO_DEVICE);
fail_detach:
dma_buf_detach(dma_buf, attach);
dma_buf_put(dma_buf);
DBG("irqmask=%08x", irqmask);
priv->dispc_ops->write_irqenable(irqmask);
- priv->dispc_ops->read_irqenable(); /* flush posted write */
}
static void omap_irq_wait_handler(struct omap_irq_wait *wait)
pr_cont("(0x%08x)\n", irqstatus);
}
-static void omap_irq_ocp_error_handler(u32 irqstatus)
+static void omap_irq_ocp_error_handler(struct drm_device *dev,
+ u32 irqstatus)
{
if (!(irqstatus & DISPC_IRQ_OCP_ERR))
return;
- DRM_ERROR("OCP error\n");
+ dev_err_ratelimited(dev->dev, "OCP error\n");
}
static irqreturn_t omap_irq_handler(int irq, void *arg)
omap_crtc_error_irq(crtc, irqstatus);
}
- omap_irq_ocp_error_handler(irqstatus);
+ omap_irq_ocp_error_handler(dev, irqstatus);
omap_irq_fifo_underflow(priv, irqstatus);
spin_lock_irqsave(&priv->wait_lock, flags);
struct drm_plane base;
enum omap_plane_id id;
const char *name;
-
- uint32_t nformats;
- uint32_t formats[32];
-};
-
-struct omap_plane_state {
- struct drm_plane_state base;
-
- unsigned int zorder;
};
-static inline struct omap_plane_state *
-to_omap_plane_state(struct drm_plane_state *state)
-{
- return container_of(state, struct omap_plane_state, base);
-}
-
static int omap_plane_prepare_fb(struct drm_plane *plane,
struct drm_plane_state *new_state)
{
struct omap_drm_private *priv = plane->dev->dev_private;
struct omap_plane *omap_plane = to_omap_plane(plane);
struct drm_plane_state *state = plane->state;
- struct omap_plane_state *omap_state = to_omap_plane_state(state);
struct omap_overlay_info info;
- struct omap_drm_window win;
int ret;
DBG("%s, crtc=%p fb=%p", omap_plane->name, state->crtc, state->fb);
memset(&info, 0, sizeof(info));
- info.rotation_type = OMAP_DSS_ROT_DMA;
- info.rotation = OMAP_DSS_ROT_0;
+ info.rotation_type = OMAP_DSS_ROT_NONE;
+ info.rotation = DRM_MODE_ROTATE_0;
info.global_alpha = 0xff;
- info.mirror = 0;
- info.zorder = omap_state->zorder;
-
- memset(&win, 0, sizeof(win));
- win.rotation = state->rotation;
- win.crtc_x = state->crtc_x;
- win.crtc_y = state->crtc_y;
- win.crtc_w = state->crtc_w;
- win.crtc_h = state->crtc_h;
-
- /*
- * src values are in Q16 fixed point, convert to integer.
- * omap_framebuffer_update_scanout() takes adjusted src.
- */
- win.src_x = state->src_x >> 16;
- win.src_y = state->src_y >> 16;
-
- if (drm_rotation_90_or_270(state->rotation)) {
- win.src_w = state->src_h >> 16;
- win.src_h = state->src_w >> 16;
- } else {
- win.src_w = state->src_w >> 16;
- win.src_h = state->src_h >> 16;
- }
+ info.zorder = state->zpos;
/* update scanout: */
- omap_framebuffer_update_scanout(state->fb, &win, &info);
+ omap_framebuffer_update_scanout(state->fb, state, &info);
DBG("%dx%d -> %dx%d (%d)", info.width, info.height,
info.out_width, info.out_height,
DBG("%d,%d %pad %pad", info.pos_x, info.pos_y,
&info.paddr, &info.p_uv_addr);
- priv->dispc_ops->ovl_set_channel_out(omap_plane->id,
- omap_crtc_channel(state->crtc));
-
/* and finally, update omapdss: */
ret = priv->dispc_ops->ovl_setup(omap_plane->id, &info,
- omap_crtc_timings(state->crtc), false);
+ omap_crtc_timings(state->crtc), false,
+ omap_crtc_channel(state->crtc));
if (ret) {
dev_err(plane->dev->dev, "Failed to setup plane %s\n",
omap_plane->name);
struct drm_plane_state *old_state)
{
struct omap_drm_private *priv = plane->dev->dev_private;
- struct omap_plane_state *omap_state = to_omap_plane_state(plane->state);
struct omap_plane *omap_plane = to_omap_plane(plane);
plane->state->rotation = DRM_MODE_ROTATE_0;
- omap_state->zorder = plane->type == DRM_PLANE_TYPE_PRIMARY
+ plane->state->zpos = plane->type == DRM_PLANE_TYPE_PRIMARY
? 0 : omap_plane->id;
priv->dispc_ops->ovl_enable(omap_plane->id, false);
drm_object_attach_property(obj, priv->zorder_prop, 0);
}
-static struct drm_plane_state *
-omap_plane_atomic_duplicate_state(struct drm_plane *plane)
-{
- struct omap_plane_state *state;
- struct omap_plane_state *copy;
-
- if (WARN_ON(!plane->state))
- return NULL;
-
- state = to_omap_plane_state(plane->state);
- copy = kmemdup(state, sizeof(*state), GFP_KERNEL);
- if (copy == NULL)
- return NULL;
-
- __drm_atomic_helper_plane_duplicate_state(plane, ©->base);
-
- return ©->base;
-}
-
-static void omap_plane_atomic_destroy_state(struct drm_plane *plane,
- struct drm_plane_state *state)
-{
- __drm_atomic_helper_plane_destroy_state(state);
- kfree(to_omap_plane_state(state));
-}
-
static void omap_plane_reset(struct drm_plane *plane)
{
struct omap_plane *omap_plane = to_omap_plane(plane);
- struct omap_plane_state *omap_state;
- if (plane->state) {
- omap_plane_atomic_destroy_state(plane, plane->state);
- plane->state = NULL;
- }
-
- omap_state = kzalloc(sizeof(*omap_state), GFP_KERNEL);
- if (omap_state == NULL)
+ drm_atomic_helper_plane_reset(plane);
+ if (!plane->state)
return;
/*
- * Set defaults depending on whether we are a primary or overlay
+ * Set the zpos default depending on whether we are a primary or overlay
* plane.
*/
- omap_state->zorder = plane->type == DRM_PLANE_TYPE_PRIMARY
+ plane->state->zpos = plane->type == DRM_PLANE_TYPE_PRIMARY
? 0 : omap_plane->id;
- omap_state->base.rotation = DRM_MODE_ROTATE_0;
-
- plane->state = &omap_state->base;
- plane->state->plane = plane;
}
static int omap_plane_atomic_set_property(struct drm_plane *plane,
uint64_t val)
{
struct omap_drm_private *priv = plane->dev->dev_private;
- struct omap_plane_state *omap_state = to_omap_plane_state(state);
if (property == priv->zorder_prop)
- omap_state->zorder = val;
+ state->zpos = val;
else
return -EINVAL;
uint64_t *val)
{
struct omap_drm_private *priv = plane->dev->dev_private;
- const struct omap_plane_state *omap_state =
- container_of(state, const struct omap_plane_state, base);
if (property == priv->zorder_prop)
- *val = omap_state->zorder;
+ *val = state->zpos;
else
return -EINVAL;
.reset = omap_plane_reset,
.destroy = omap_plane_destroy,
.set_property = drm_atomic_helper_plane_set_property,
- .atomic_duplicate_state = omap_plane_atomic_duplicate_state,
- .atomic_destroy_state = omap_plane_atomic_destroy_state,
+ .atomic_duplicate_state = drm_atomic_helper_plane_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_plane_destroy_state,
.atomic_set_property = omap_plane_atomic_set_property,
.atomic_get_property = omap_plane_atomic_get_property,
};
u32 possible_crtcs)
{
struct omap_drm_private *priv = dev->dev_private;
+ unsigned int num_planes = priv->dispc_ops->get_num_ovls();
struct drm_plane *plane;
struct omap_plane *omap_plane;
enum omap_plane_id id;
int ret;
+ u32 nformats;
+ const u32 *formats;
if (WARN_ON(idx >= ARRAY_SIZE(plane_idx_to_id)))
return ERR_PTR(-EINVAL);
if (!omap_plane)
return ERR_PTR(-ENOMEM);
- omap_plane->nformats = omap_framebuffer_get_formats(
- omap_plane->formats, ARRAY_SIZE(omap_plane->formats),
- priv->dispc_ops->ovl_get_color_modes(id));
+ formats = priv->dispc_ops->ovl_get_color_modes(id);
+ for (nformats = 0; formats[nformats]; ++nformats)
+ ;
omap_plane->id = id;
omap_plane->name = plane_id_to_name[id];
plane = &omap_plane->base;
ret = drm_universal_plane_init(dev, plane, possible_crtcs,
- &omap_plane_funcs, omap_plane->formats,
- omap_plane->nformats, type, NULL);
+ &omap_plane_funcs, formats,
+ nformats, type, NULL);
if (ret < 0)
goto error;
drm_plane_helper_add(plane, &omap_plane_helper_funcs);
omap_plane_install_properties(plane, &plane->base);
+ drm_plane_create_zpos_property(plane, 0, 0, num_planes - 1);
return plane;
if (ret)
return;
- cmd = (struct qxl_cursor_cmd *) qxl_release_map(qdev, release);
-
if (fb != old_state->fb) {
obj = to_qxl_framebuffer(fb)->obj;
user_bo = gem_to_qxl_bo(obj);
qxl_bo_kunmap(cursor_bo);
qxl_bo_kunmap(user_bo);
+ cmd = (struct qxl_cursor_cmd *) qxl_release_map(qdev, release);
cmd->u.set.visible = 1;
cmd->u.set.shape = qxl_bo_physical_address(qdev,
cursor_bo, 0);
if (ret)
goto out_free_release;
+ cmd = (struct qxl_cursor_cmd *) qxl_release_map(qdev, release);
cmd->type = QXL_CURSOR_MOVE;
}
u32 vblank_time = r600_dpm_get_vblank_time(rdev);
u32 switch_limit = pi->mem_gddr5 ? 450 : 300;
+ /* disable mclk switching if the refresh is >120Hz, even if the
+ * blanking period would allow it
+ */
+ if (r600_dpm_get_vrefresh(rdev) > 120)
+ return true;
+
if (vblank_time < switch_limit)
return true;
else
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.cik.disp_int_cont5 & DC_HPD6_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.cik.disp_int_cont5 & DC_HPD6_RX_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_RX_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.evergreen.disp_int_cont5 & DC_HPD6_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.evergreen.disp_int_cont5 & DC_HPD6_RX_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_RX_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.r600.disp_int_cont2 & DC_HPD6_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
if ((radeon_runtime_pm != 0) &&
radeon_has_atpx() &&
((flags & RADEON_IS_IGP) == 0) &&
- !pci_is_thunderbolt_attached(rdev->pdev))
+ !pci_is_thunderbolt_attached(dev->pdev))
flags |= RADEON_IS_PX;
/* radeon_device_init should report only fatal error
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.evergreen.disp_int_cont5 & DC_HPD6_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
WREG32(DC_HPD5_INT_CONTROL, tmp);
}
if (rdev->irq.stat_regs.evergreen.disp_int_cont5 & DC_HPD6_RX_INTERRUPT) {
- tmp = RREG32(DC_HPD5_INT_CONTROL);
+ tmp = RREG32(DC_HPD6_INT_CONTROL);
tmp |= DC_HPDx_RX_INT_ACK;
WREG32(DC_HPD6_INT_CONTROL, tmp);
}
struct cpuinfo_x86 *c = &cpu_data(cpu);
struct platform_data *pdata;
+ /*
+ * Don't execute this on resume as the offline callback did
+ * not get executed on suspend.
+ */
+ if (cpuhp_tasks_frozen)
+ return 0;
+
/*
* CPUID.06H.EAX[0] indicates whether the CPU has thermal
* sensors. We check this bit only, all the early CPUs
struct temp_data *tdata;
int indx, target;
+ /*
+ * Don't execute this on suspend as the device remove locks
+ * up the machine.
+ */
+ if (cpuhp_tasks_frozen)
+ return 0;
+
/* If the physical CPU device does not exist, just return */
if (!pdev)
return 0;
static int dw_i2c_acpi_configure(struct platform_device *pdev)
{
struct dw_i2c_dev *dev = platform_get_drvdata(pdev);
+ u32 ss_ht = 0, fp_ht = 0, hs_ht = 0, fs_ht = 0;
acpi_handle handle = ACPI_HANDLE(&pdev->dev);
const struct acpi_device_id *id;
struct acpi_device *adev;
* Try to get SDA hold time and *CNT values from an ACPI method for
* selected speed modes.
*/
+ dw_i2c_acpi_params(pdev, "SSCN", &dev->ss_hcnt, &dev->ss_lcnt, &ss_ht);
+ dw_i2c_acpi_params(pdev, "FPCN", &dev->fp_hcnt, &dev->fp_lcnt, &fp_ht);
+ dw_i2c_acpi_params(pdev, "HSCN", &dev->hs_hcnt, &dev->hs_lcnt, &hs_ht);
+ dw_i2c_acpi_params(pdev, "FMCN", &dev->fs_hcnt, &dev->fs_lcnt, &fs_ht);
+
switch (dev->clk_freq) {
case 100000:
- dw_i2c_acpi_params(pdev, "SSCN", &dev->ss_hcnt, &dev->ss_lcnt,
- &dev->sda_hold_time);
+ dev->sda_hold_time = ss_ht;
break;
case 1000000:
- dw_i2c_acpi_params(pdev, "FPCN", &dev->fp_hcnt, &dev->fp_lcnt,
- &dev->sda_hold_time);
+ dev->sda_hold_time = fp_ht;
break;
case 3400000:
- dw_i2c_acpi_params(pdev, "HSCN", &dev->hs_hcnt, &dev->hs_lcnt,
- &dev->sda_hold_time);
+ dev->sda_hold_time = hs_ht;
break;
case 400000:
default:
- dw_i2c_acpi_params(pdev, "FMCN", &dev->fs_hcnt, &dev->fs_lcnt,
- &dev->sda_hold_time);
+ dev->sda_hold_time = fs_ht;
break;
}
rc = -EINVAL;
goto out;
}
- drv_data->irq = irq_of_parse_and_map(np, 0);
drv_data->rstc = devm_reset_control_get_optional(dev, NULL);
if (IS_ERR(drv_data->rstc)) {
if (!IS_ERR(drv_data->clk))
clk_prepare_enable(drv_data->clk);
+ drv_data->irq = platform_get_irq(pd, 0);
+
if (pdata) {
drv_data->freq_m = pdata->freq_m;
drv_data->freq_n = pdata->freq_n;
- drv_data->irq = platform_get_irq(pd, 0);
drv_data->adapter.timeout = msecs_to_jiffies(pdata->timeout);
drv_data->offload_enabled = false;
memcpy(&drv_data->reg_offsets, &mv64xxx_i2c_regs_mv64xxx, sizeof(drv_data->reg_offsets));
goto exit_clk;
}
if (drv_data->irq < 0) {
- rc = -ENXIO;
+ rc = drv_data->irq;
goto exit_reset;
}
int value, int index, void *data, int len)
{
struct i2c_tiny_usb *dev = (struct i2c_tiny_usb *)adapter->algo_data;
+ void *dmadata = kmalloc(len, GFP_KERNEL);
+ int ret;
+
+ if (!dmadata)
+ return -ENOMEM;
/* do control transfer */
- return usb_control_msg(dev->usb_dev, usb_rcvctrlpipe(dev->usb_dev, 0),
+ ret = usb_control_msg(dev->usb_dev, usb_rcvctrlpipe(dev->usb_dev, 0),
cmd, USB_TYPE_VENDOR | USB_RECIP_INTERFACE |
- USB_DIR_IN, value, index, data, len, 2000);
+ USB_DIR_IN, value, index, dmadata, len, 2000);
+
+ memcpy(data, dmadata, len);
+ kfree(dmadata);
+ return ret;
}
static int usb_write(struct i2c_adapter *adapter, int cmd,
int value, int index, void *data, int len)
{
struct i2c_tiny_usb *dev = (struct i2c_tiny_usb *)adapter->algo_data;
+ void *dmadata = kmemdup(data, len, GFP_KERNEL);
+ int ret;
+
+ if (!dmadata)
+ return -ENOMEM;
/* do control transfer */
- return usb_control_msg(dev->usb_dev, usb_sndctrlpipe(dev->usb_dev, 0),
+ ret = usb_control_msg(dev->usb_dev, usb_sndctrlpipe(dev->usb_dev, 0),
cmd, USB_TYPE_VENDOR | USB_RECIP_INTERFACE,
- value, index, data, len, 2000);
+ value, index, dmadata, len, 2000);
+
+ kfree(dmadata);
+ return ret;
}
static void i2c_tiny_usb_free(struct i2c_tiny_usb *dev)
adapter->class = I2C_CLASS_HWMON;
adapter->dev.parent = &pdev->dev;
adapter->dev.of_node = pdev->dev.of_node;
+ ACPI_COMPANION_SET(&adapter->dev, ACPI_COMPANION(&pdev->dev));
i2c_set_adapdata(adapter, ctx);
rc = i2c_add_adapter(adapter);
if (rc) {
if (force_nr) {
priv->adap.nr = force_nr;
ret = i2c_add_numbered_adapter(&priv->adap);
- dev_err(&parent->dev,
- "failed to add mux-adapter %u as bus %u (error=%d)\n",
- chan_id, force_nr, ret);
+ if (ret < 0) {
+ dev_err(&parent->dev,
+ "failed to add mux-adapter %u as bus %u (error=%d)\n",
+ chan_id, force_nr, ret);
+ goto err_free_priv;
+ }
} else {
ret = i2c_add_adapter(&priv->adap);
- dev_err(&parent->dev,
- "failed to add mux-adapter %u (error=%d)\n",
- chan_id, ret);
- }
- if (ret < 0) {
- kfree(priv);
- return ret;
+ if (ret < 0) {
+ dev_err(&parent->dev,
+ "failed to add mux-adapter %u (error=%d)\n",
+ chan_id, ret);
+ goto err_free_priv;
+ }
}
WARN(sysfs_create_link(&priv->adap.dev.kobj, &muxc->dev->kobj,
muxc->adapter[muxc->num_adapters++] = &priv->adap;
return 0;
+
+err_free_priv:
+ kfree(priv);
+ return ret;
}
EXPORT_SYMBOL_GPL(i2c_mux_add_adapter);
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
mux->data.reg_size = resource_size(res);
mux->data.reg = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(mux->data.reg))
- return PTR_ERR(mux->data.reg);
+ if (IS_ERR(mux->data.reg)) {
+ ret = PTR_ERR(mux->data.reg);
+ goto err_put_parent;
+ }
}
if (mux->data.reg_size != 4 && mux->data.reg_size != 2 &&
mux->data.reg_size != 1) {
dev_err(&pdev->dev, "Invalid register size\n");
- return -EINVAL;
+ ret = -EINVAL;
+ goto err_put_parent;
}
muxc = i2c_mux_alloc(parent, &pdev->dev, mux->data.n_values, 0, 0,
i2c_mux_reg_select, NULL);
- if (!muxc)
- return -ENOMEM;
+ if (!muxc) {
+ ret = -ENOMEM;
+ goto err_put_parent;
+ }
muxc->priv = mux;
platform_set_drvdata(pdev, muxc);
ret = i2c_mux_add_adapter(muxc, nr, mux->data.values[i], class);
if (ret)
- goto add_adapter_failed;
+ goto err_del_mux_adapters;
}
dev_dbg(&pdev->dev, "%d port mux on %s adapter\n",
return 0;
-add_adapter_failed:
+err_del_mux_adapters:
i2c_mux_del_adapters(muxc);
+err_put_parent:
+ i2c_put_adapter(parent);
return ret;
}
struct completion *completion)
{
struct device *dev = &client->dev;
- long ret;
int error;
int len;
- u8 buffer[ETP_I2C_INF_LENGTH];
+ u8 buffer[ETP_I2C_REPORT_LEN];
+
+ len = i2c_master_recv(client, buffer, ETP_I2C_REPORT_LEN);
+ if (len != ETP_I2C_REPORT_LEN) {
+ error = len < 0 ? len : -EIO;
+ dev_warn(dev, "failed to read I2C data after FW WDT reset: %d (%d)\n",
+ error, len);
+ }
reinit_completion(completion);
enable_irq(client->irq);
error = elan_i2c_write_cmd(client, ETP_I2C_STAND_CMD, ETP_I2C_RESET);
- if (!error)
- ret = wait_for_completion_interruptible_timeout(completion,
- msecs_to_jiffies(300));
- disable_irq(client->irq);
-
if (error) {
dev_err(dev, "device reset failed: %d\n", error);
- return error;
- } else if (ret == 0) {
+ } else if (!wait_for_completion_timeout(completion,
+ msecs_to_jiffies(300))) {
dev_err(dev, "timeout waiting for device reset\n");
- return -ETIMEDOUT;
- } else if (ret < 0) {
- error = ret;
- dev_err(dev, "error waiting for device reset: %d\n", error);
- return error;
+ error = -ETIMEDOUT;
}
+ disable_irq(client->irq);
+
+ if (error)
+ return error;
+
len = i2c_master_recv(client, buffer, ETP_I2C_INF_LENGTH);
if (len != ETP_I2C_INF_LENGTH) {
error = len < 0 ? len : -EIO;
case MXT_TOUCH_KEYARRAY_T15:
case MXT_TOUCH_PROXIMITY_T23:
case MXT_TOUCH_PROXKEY_T52:
+ case MXT_TOUCH_MULTITOUCHSCREEN_T100:
case MXT_PROCI_GRIPFACE_T20:
case MXT_PROCG_NOISE_T22:
case MXT_PROCI_ONETOUCH_T24:
static EDT_ATTR(offset, S_IWUSR | S_IRUGO, WORK_REGISTER_OFFSET,
M09_REGISTER_OFFSET, 0, 31);
static EDT_ATTR(threshold, S_IWUSR | S_IRUGO, WORK_REGISTER_THRESHOLD,
- M09_REGISTER_THRESHOLD, 20, 80);
+ M09_REGISTER_THRESHOLD, 0, 80);
static EDT_ATTR(report_rate, S_IWUSR | S_IRUGO, WORK_REGISTER_REPORT_RATE,
NO_REGISTER, 3, 14);
dma_addr_t iova, size_t size)
{
struct iova_domain *iovad = &cookie->iovad;
- unsigned long shift = iova_shift(iovad);
/* The MSI case is only ever cleaning up its most recent allocation */
if (cookie->type == IOMMU_DMA_MSI_COOKIE)
cookie->msi_iova -= size;
else
- free_iova_fast(iovad, iova >> shift, size >> shift);
+ free_iova_fast(iovad, iova_pfn(iovad, iova),
+ size >> iova_shift(iovad));
}
static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr,
{
struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
struct iommu_dma_cookie *cookie = domain->iova_cookie;
- struct iova_domain *iovad = &cookie->iovad;
- size_t iova_off = iova_offset(iovad, phys);
+ size_t iova_off = 0;
dma_addr_t iova;
- size = iova_align(iovad, size + iova_off);
+ if (cookie->type == IOMMU_DMA_IOVA_COOKIE) {
+ iova_off = iova_offset(&cookie->iovad, phys);
+ size = iova_align(&cookie->iovad, size + iova_off);
+ }
+
iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev);
if (!iova)
return DMA_ERROR_CODE;
if (context_copied(context)) {
u16 did_old = context_domain_id(context);
- if (did_old >= 0 && did_old < cap_ndoms(iommu->cap))
+ if (did_old >= 0 && did_old < cap_ndoms(iommu->cap)) {
iommu->flush.flush_context(iommu, did_old,
(((u16)bus) << 8) | devfn,
DMA_CCMD_MASK_NOBIT,
DMA_CCMD_DEVICE_INVL);
+ iommu->flush.flush_iotlb(iommu, did_old, 0, 0,
+ DMA_TLB_DSI_FLUSH);
+ }
}
pgd = domain->pgd;
#include <linux/clk.h>
#include <linux/component.h>
#include <linux/device.h>
+#include <linux/dma-mapping.h>
#include <linux/dma-iommu.h>
#include <linux/err.h>
#include <linux/interrupt.h>
static inline void get_mbigen_clear_reg(irq_hw_number_t hwirq,
u32 *mask, u32 *addr)
{
- unsigned int ofst;
-
- hwirq -= RESERVED_IRQ_PER_MBIGEN_CHIP;
- ofst = hwirq / 32 * 4;
+ unsigned int ofst = (hwirq / 32) * 4;
*mask = 1 << (hwirq % 32);
*addr = ofst + REG_MBIGEN_CLEAR_OFFSET;
mgn_chip->pdev = pdev;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- mgn_chip->base = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(mgn_chip->base))
- return PTR_ERR(mgn_chip->base);
+ if (!res)
+ return -EINVAL;
+
+ mgn_chip->base = devm_ioremap(&pdev->dev, res->start,
+ resource_size(res));
+ if (!mgn_chip->base) {
+ dev_err(&pdev->dev, "failed to ioremap %pR\n", res);
+ return -ENOMEM;
+ }
if (IS_ENABLED(CONFIG_OF) && pdev->dev.of_node)
err = mbigen_of_create_domain(pdev, mgn_chip);
"slave address 0x%02x\n",
client->name, chip->bits, client->addr);
- if (!i2c_check_functionality(adapter, I2C_FUNC_I2C))
+ if (!i2c_check_functionality(adapter, I2C_FUNC_SMBUS_BYTE_DATA))
return -EIO;
if (pdata) {
* Buffers are freed after this timeout
*/
static unsigned dm_bufio_max_age = DM_BUFIO_DEFAULT_AGE_SECS;
-static unsigned dm_bufio_retain_bytes = DM_BUFIO_DEFAULT_RETAIN_BYTES;
+static unsigned long dm_bufio_retain_bytes = DM_BUFIO_DEFAULT_RETAIN_BYTES;
static unsigned long dm_bufio_peak_allocated;
static unsigned long dm_bufio_allocated_kmem_cache;
return true;
}
-static unsigned get_retain_buffers(struct dm_bufio_client *c)
+static unsigned long get_retain_buffers(struct dm_bufio_client *c)
{
- unsigned retain_bytes = ACCESS_ONCE(dm_bufio_retain_bytes);
- return retain_bytes / c->block_size;
+ unsigned long retain_bytes = ACCESS_ONCE(dm_bufio_retain_bytes);
+ return retain_bytes >> (c->sectors_per_block_bits + SECTOR_SHIFT);
}
static unsigned long __scan(struct dm_bufio_client *c, unsigned long nr_to_scan,
struct dm_buffer *b, *tmp;
unsigned long freed = 0;
unsigned long count = nr_to_scan;
- unsigned retain_target = get_retain_buffers(c);
+ unsigned long retain_target = get_retain_buffers(c);
for (l = 0; l < LIST_SIZE; l++) {
list_for_each_entry_safe_reverse(b, tmp, &c->lru[l], lru_list) {
static void __evict_old_buffers(struct dm_bufio_client *c, unsigned long age_hz)
{
struct dm_buffer *b, *tmp;
- unsigned retain_target = get_retain_buffers(c);
- unsigned count;
+ unsigned long retain_target = get_retain_buffers(c);
+ unsigned long count;
LIST_HEAD(write_list);
dm_bufio_lock(c);
module_param_named(max_age_seconds, dm_bufio_max_age, uint, S_IRUGO | S_IWUSR);
MODULE_PARM_DESC(max_age_seconds, "Max age of a buffer in seconds");
-module_param_named(retain_bytes, dm_bufio_retain_bytes, uint, S_IRUGO | S_IWUSR);
+module_param_named(retain_bytes, dm_bufio_retain_bytes, ulong, S_IRUGO | S_IWUSR);
MODULE_PARM_DESC(retain_bytes, "Try to keep at least this many bytes cached in memory");
module_param_named(peak_allocated_bytes, dm_bufio_peak_allocated, ulong, S_IRUGO | S_IWUSR);
{
struct background_tracker *b = kmalloc(sizeof(*b), GFP_KERNEL);
+ if (!b) {
+ DMERR("couldn't create background_tracker");
+ return NULL;
+ }
+
b->max_work = max_work;
atomic_set(&b->pending_promotes, 0);
atomic_set(&b->pending_writebacks, 0);
* Cache entries may not be populated. So we cannot rely on the
* size of the clean queue.
*/
- unsigned nr_clean;
-
if (idle) {
/*
* We'd like to clean everything.
return q_size(&mq->dirty) == 0u;
}
- nr_clean = from_cblock(mq->cache_size) - q_size(&mq->dirty);
- return (nr_clean + btracker_nr_writebacks_queued(mq->bg_work)) >=
- percent_to_target(mq, CLEAN_TARGET);
+ /*
+ * If we're busy we don't worry about cleaning at all.
+ */
+ return true;
}
-static bool free_target_met(struct smq_policy *mq, bool idle)
+static bool free_target_met(struct smq_policy *mq)
{
unsigned nr_free;
- if (!idle)
- return true;
-
nr_free = from_cblock(mq->cache_size) - mq->cache_alloc.nr_allocated;
return (nr_free + btracker_nr_demotions_queued(mq->bg_work)) >=
percent_to_target(mq, FREE_TARGET);
if (unlikely(WARN_ON_ONCE(!mq->migrations_allowed)))
return;
- e = q_peek(&mq->clean, mq->clean.nr_levels, true);
+ e = q_peek(&mq->clean, mq->clean.nr_levels / 2, true);
if (!e) {
- if (!clean_target_met(mq, false))
+ if (!clean_target_met(mq, true))
queue_writeback(mq);
return;
}
* We always claim to be 'idle' to ensure some demotions happen
* with continuous loads.
*/
- if (!free_target_met(mq, true))
+ if (!free_target_met(mq))
queue_demotion(mq);
return;
}
spin_lock_irqsave(&mq->lock, flags);
r = btracker_issue(mq->bg_work, result);
if (r == -ENODATA) {
- /* find some writeback work to do */
- if (mq->migrations_allowed && !free_target_met(mq, idle))
- queue_demotion(mq);
-
- else if (!clean_target_met(mq, idle))
+ if (!clean_target_met(mq, idle)) {
queue_writeback(mq);
-
- r = btracker_issue(mq->bg_work, result);
+ r = btracker_issue(mq->bg_work, result);
+ }
}
spin_unlock_irqrestore(&mq->lock, flags);
clear_pending(mq, e);
if (success) {
e->oblock = work->oblock;
+ e->level = NR_CACHE_LEVELS - 1;
push(mq, e);
// h, q, a
} else {
static void __iot_io_end(struct io_tracker *iot, sector_t len)
{
+ if (!len)
+ return;
+
iot->in_flight -= len;
if (!iot->in_flight)
iot->idle_time = jiffies;
spinlock_t invalidation_lock;
struct list_head invalidation_requests;
- struct io_tracker origin_tracker;
+ struct io_tracker tracker;
struct work_struct commit_ws;
struct batcher committer;
static bool accountable_bio(struct cache *cache, struct bio *bio)
{
- return ((bio->bi_bdev == cache->origin_dev->bdev) &&
- bio_op(bio) != REQ_OP_DISCARD);
+ return bio_op(bio) != REQ_OP_DISCARD;
}
static void accounted_begin(struct cache *cache, struct bio *bio)
if (accountable_bio(cache, bio)) {
pb->len = bio_sectors(bio);
- iot_io_begin(&cache->origin_tracker, pb->len);
+ iot_io_begin(&cache->tracker, pb->len);
}
}
size_t pb_data_size = get_per_bio_data_size(cache);
struct per_bio_data *pb = get_per_bio_data(bio, pb_data_size);
- iot_io_end(&cache->origin_tracker, pb->len);
+ iot_io_end(&cache->tracker, pb->len);
}
static void accounted_request(struct cache *cache, struct bio *bio)
enum busy {
IDLE,
- MODERATE,
BUSY
};
static enum busy spare_migration_bandwidth(struct cache *cache)
{
- bool idle = iot_idle_for(&cache->origin_tracker, HZ);
+ bool idle = iot_idle_for(&cache->tracker, HZ);
sector_t current_volume = (atomic_read(&cache->nr_io_migrations) + 1) *
cache->sectors_per_block;
- if (current_volume <= cache->migration_threshold)
- return idle ? IDLE : MODERATE;
+ if (idle && current_volume <= cache->migration_threshold)
+ return IDLE;
else
- return idle ? MODERATE : BUSY;
+ return BUSY;
}
static void inc_hit_counter(struct cache *cache, struct bio *bio)
for (;;) {
b = spare_migration_bandwidth(cache);
- if (b == BUSY)
- break;
r = policy_get_background_work(cache->policy, b == IDLE, &op);
if (r == -ENODATA)
batcher_init(&cache->committer, commit_op, cache,
issue_op, cache, cache->wq);
- iot_init(&cache->origin_tracker);
+ iot_init(&cache->tracker);
init_rwsem(&cache->background_work_lock);
prevent_background_work(cache);
cancel_delayed_work(&cache->waker);
flush_workqueue(cache->wq);
- WARN_ON(cache->origin_tracker.in_flight);
+ WARN_ON(cache->tracker.in_flight);
/*
* If it's a flush suspend there won't be any deferred bios, so this
* it has been invoked.
*/
#define dm_report_EIO(m) \
-({ \
+do { \
struct mapped_device *md = dm_table_get_md((m)->ti->table); \
\
pr_debug("%s: returning EIO; QIFNP = %d; SQIFNP = %d; DNFS = %d\n", \
test_bit(MPATHF_QUEUE_IF_NO_PATH, &(m)->flags), \
test_bit(MPATHF_SAVED_QUEUE_IF_NO_PATH, &(m)->flags), \
dm_noflush_suspending((m)->ti)); \
- -EIO; \
-})
+} while (0)
/*
* Map cloned requests (request-based multipath)
if (!pgpath) {
if (test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags))
return DM_MAPIO_DELAY_REQUEUE;
- return dm_report_EIO(m); /* Failed */
+ dm_report_EIO(m); /* Failed */
+ return DM_MAPIO_KILL;
} else if (test_bit(MPATHF_QUEUE_IO, &m->flags) ||
test_bit(MPATHF_PG_INIT_REQUIRED, &m->flags)) {
if (pg_init_all_paths(m))
if (!pgpath) {
if (test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags))
return DM_MAPIO_REQUEUE;
- return dm_report_EIO(m);
+ dm_report_EIO(m);
+ return -EIO;
}
mpio->pgpath = pgpath;
if (atomic_read(&m->nr_valid_paths) == 0 &&
!test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags)) {
if (error == -EIO)
- error = dm_report_EIO(m);
+ dm_report_EIO(m);
/* complete with the original error */
r = DM_ENDIO_DONE;
}
fail_path(mpio->pgpath);
if (atomic_read(&m->nr_valid_paths) == 0 &&
- !test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags))
- return dm_report_EIO(m);
+ !test_bit(MPATHF_QUEUE_IF_NO_PATH, &m->flags)) {
+ dm_report_EIO(m);
+ return -EIO;
+ }
/* Queue for the daemon to resubmit */
dm_bio_restore(get_bio_details_from_bio(clone), clone);
case DM_MAPIO_KILL:
/* The target wants to complete the I/O */
dm_kill_unmapped_request(rq, -EIO);
+ break;
default:
DMWARN("unimplemented target map return value: %d", r);
BUG();
if (r < 0)
return r;
- r = save_sm_roots(pmd);
+ r = dm_tm_pre_commit(pmd->tm);
if (r < 0)
return r;
- r = dm_tm_pre_commit(pmd->tm);
+ r = save_sm_roots(pmd);
if (r < 0)
return r;
* may proceed without blocking. It is important to call this before
* attempting a GFP_KERNEL allocation while holding the mddev lock.
* Must be called with mddev_lock held.
- *
- * In the ->external case MD_SB_CHANGE_PENDING can not be cleared until mddev->lock
- * is dropped, so return -EAGAIN after notifying userspace.
*/
-int md_allow_write(struct mddev *mddev)
+void md_allow_write(struct mddev *mddev)
{
if (!mddev->pers)
- return 0;
+ return;
if (mddev->ro)
- return 0;
+ return;
if (!mddev->pers->sync_request)
- return 0;
+ return;
spin_lock(&mddev->lock);
if (mddev->in_sync) {
spin_unlock(&mddev->lock);
md_update_sb(mddev, 0);
sysfs_notify_dirent_safe(mddev->sysfs_state);
+ /* wait for the dirty state to be recorded in the metadata */
+ wait_event(mddev->sb_wait,
+ !test_bit(MD_SB_CHANGE_CLEAN, &mddev->sb_flags) &&
+ !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags));
} else
spin_unlock(&mddev->lock);
-
- if (test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags))
- return -EAGAIN;
- else
- return 0;
}
EXPORT_SYMBOL_GPL(md_allow_write);
bool metadata_op);
extern void md_do_sync(struct md_thread *thread);
extern void md_new_event(struct mddev *mddev);
-extern int md_allow_write(struct mddev *mddev);
+extern void md_allow_write(struct mddev *mddev);
extern void md_wait_for_blocked_rdev(struct md_rdev *rdev, struct mddev *mddev);
extern void md_set_array_sectors(struct mddev *mddev, sector_t array_sectors);
extern int md_check_no_bitmap(struct mddev *mddev);
static int sm_disk_dec_block(struct dm_space_map *sm, dm_block_t b)
{
+ int r;
+ uint32_t old_count;
enum allocation_event ev;
struct sm_disk *smd = container_of(sm, struct sm_disk, sm);
- return sm_ll_dec(&smd->ll, b, &ev);
+ r = sm_ll_dec(&smd->ll, b, &ev);
+ if (!r && (ev == SM_FREE)) {
+ /*
+ * It's only free if it's also free in the last
+ * transaction.
+ */
+ r = sm_ll_lookup(&smd->old_ll, b, &old_count);
+ if (!r && !old_count)
+ smd->nr_allocated_this_transaction--;
+ }
+
+ return r;
}
static int sm_disk_new_block(struct dm_space_map *sm, dm_block_t *b)
blk_queue_max_hw_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_max_write_same_sectors(mddev->queue, mddev->chunk_sectors);
blk_queue_max_write_zeroes_sectors(mddev->queue, mddev->chunk_sectors);
- blk_queue_max_discard_sectors(mddev->queue, mddev->chunk_sectors);
+ blk_queue_max_discard_sectors(mddev->queue, UINT_MAX);
blk_queue_io_min(mddev->queue, mddev->chunk_sectors << 9);
blk_queue_io_opt(mddev->queue,
}
}
+static void raid0_handle_discard(struct mddev *mddev, struct bio *bio)
+{
+ struct r0conf *conf = mddev->private;
+ struct strip_zone *zone;
+ sector_t start = bio->bi_iter.bi_sector;
+ sector_t end;
+ unsigned int stripe_size;
+ sector_t first_stripe_index, last_stripe_index;
+ sector_t start_disk_offset;
+ unsigned int start_disk_index;
+ sector_t end_disk_offset;
+ unsigned int end_disk_index;
+ unsigned int disk;
+
+ zone = find_zone(conf, &start);
+
+ if (bio_end_sector(bio) > zone->zone_end) {
+ struct bio *split = bio_split(bio,
+ zone->zone_end - bio->bi_iter.bi_sector, GFP_NOIO,
+ mddev->bio_set);
+ bio_chain(split, bio);
+ generic_make_request(bio);
+ bio = split;
+ end = zone->zone_end;
+ } else
+ end = bio_end_sector(bio);
+
+ if (zone != conf->strip_zone)
+ end = end - zone[-1].zone_end;
+
+ /* Now start and end is the offset in zone */
+ stripe_size = zone->nb_dev * mddev->chunk_sectors;
+
+ first_stripe_index = start;
+ sector_div(first_stripe_index, stripe_size);
+ last_stripe_index = end;
+ sector_div(last_stripe_index, stripe_size);
+
+ start_disk_index = (int)(start - first_stripe_index * stripe_size) /
+ mddev->chunk_sectors;
+ start_disk_offset = ((int)(start - first_stripe_index * stripe_size) %
+ mddev->chunk_sectors) +
+ first_stripe_index * mddev->chunk_sectors;
+ end_disk_index = (int)(end - last_stripe_index * stripe_size) /
+ mddev->chunk_sectors;
+ end_disk_offset = ((int)(end - last_stripe_index * stripe_size) %
+ mddev->chunk_sectors) +
+ last_stripe_index * mddev->chunk_sectors;
+
+ for (disk = 0; disk < zone->nb_dev; disk++) {
+ sector_t dev_start, dev_end;
+ struct bio *discard_bio = NULL;
+ struct md_rdev *rdev;
+
+ if (disk < start_disk_index)
+ dev_start = (first_stripe_index + 1) *
+ mddev->chunk_sectors;
+ else if (disk > start_disk_index)
+ dev_start = first_stripe_index * mddev->chunk_sectors;
+ else
+ dev_start = start_disk_offset;
+
+ if (disk < end_disk_index)
+ dev_end = (last_stripe_index + 1) * mddev->chunk_sectors;
+ else if (disk > end_disk_index)
+ dev_end = last_stripe_index * mddev->chunk_sectors;
+ else
+ dev_end = end_disk_offset;
+
+ if (dev_end <= dev_start)
+ continue;
+
+ rdev = conf->devlist[(zone - conf->strip_zone) *
+ conf->strip_zone[0].nb_dev + disk];
+ if (__blkdev_issue_discard(rdev->bdev,
+ dev_start + zone->dev_start + rdev->data_offset,
+ dev_end - dev_start, GFP_NOIO, 0, &discard_bio) ||
+ !discard_bio)
+ continue;
+ bio_chain(discard_bio, bio);
+ if (mddev->gendisk)
+ trace_block_bio_remap(bdev_get_queue(rdev->bdev),
+ discard_bio, disk_devt(mddev->gendisk),
+ bio->bi_iter.bi_sector);
+ generic_make_request(discard_bio);
+ }
+ bio_endio(bio);
+}
+
static void raid0_make_request(struct mddev *mddev, struct bio *bio)
{
struct strip_zone *zone;
return;
}
+ if (unlikely((bio_op(bio) == REQ_OP_DISCARD))) {
+ raid0_handle_discard(mddev, bio);
+ return;
+ }
+
bio_sector = bio->bi_iter.bi_sector;
sector = bio_sector;
chunk_sects = mddev->chunk_sectors;
bio->bi_iter.bi_sector = sector + zone->dev_start +
tmp_dev->data_offset;
- if (unlikely((bio_op(bio) == REQ_OP_DISCARD) &&
- !blk_queue_discard(bdev_get_queue(bio->bi_bdev)))) {
- /* Just ignore it */
- bio_endio(bio);
- } else {
- if (mddev->gendisk)
- trace_block_bio_remap(bdev_get_queue(bio->bi_bdev),
- bio, disk_devt(mddev->gendisk),
- bio_sector);
- mddev_check_writesame(mddev, bio);
- mddev_check_write_zeroes(mddev, bio);
- generic_make_request(bio);
- }
+ if (mddev->gendisk)
+ trace_block_bio_remap(bdev_get_queue(bio->bi_bdev),
+ bio, disk_devt(mddev->gendisk),
+ bio_sector);
+ mddev_check_writesame(mddev, bio);
+ mddev_check_write_zeroes(mddev, bio);
+ generic_make_request(bio);
}
static void raid0_status(struct seq_file *seq, struct mddev *mddev)
break;
}
continue;
- } else
+ } else {
+ if ((sectors > best_good_sectors) && (best_disk >= 0))
+ best_disk = -1;
best_good_sectors = sectors;
+ }
if (best_disk >= 0)
/* At least two disks to choose from so failfast is OK */
plug = container_of(cb, struct raid1_plug_cb, cb);
else
plug = NULL;
- spin_lock_irqsave(&conf->device_lock, flags);
if (plug) {
bio_list_add(&plug->pending, mbio);
plug->pending_cnt++;
} else {
+ spin_lock_irqsave(&conf->device_lock, flags);
bio_list_add(&conf->pending_bio_list, mbio);
conf->pending_count++;
- }
- spin_unlock_irqrestore(&conf->device_lock, flags);
- if (!plug)
+ spin_unlock_irqrestore(&conf->device_lock, flags);
md_wakeup_thread(mddev->thread);
+ }
}
r1_bio_write_done(r1_bio);
struct r1conf *conf = mddev->private;
int cnt, raid_disks;
unsigned long flags;
- int d, d2, err;
+ int d, d2;
/* Cannot change chunk_size, layout, or level */
if (mddev->chunk_sectors != mddev->new_chunk_sectors ||
return -EINVAL;
}
- if (!mddev_is_clustered(mddev)) {
- err = md_allow_write(mddev);
- if (err)
- return err;
- }
+ if (!mddev_is_clustered(mddev))
+ md_allow_write(mddev);
raid_disks = mddev->raid_disks + mddev->delta_disks;
plug = container_of(cb, struct raid10_plug_cb, cb);
else
plug = NULL;
- spin_lock_irqsave(&conf->device_lock, flags);
if (plug) {
bio_list_add(&plug->pending, mbio);
plug->pending_cnt++;
} else {
+ spin_lock_irqsave(&conf->device_lock, flags);
bio_list_add(&conf->pending_bio_list, mbio);
conf->pending_count++;
- }
- spin_unlock_irqrestore(&conf->device_lock, flags);
- if (!plug)
+ spin_unlock_irqrestore(&conf->device_lock, flags);
md_wakeup_thread(mddev->thread);
+ }
}
static void raid10_write_request(struct mddev *mddev, struct bio *bio,
#include "md.h"
#include "raid5.h"
#include "bitmap.h"
+#include "raid5-log.h"
/*
* metadata/data stored in disk with 4k size unit (a block) regardless
__r5l_set_io_unit_state(io, IO_UNIT_IO_START);
spin_unlock_irqrestore(&log->io_list_lock, flags);
+ /*
+ * In case of journal device failures, submit_bio will get error
+ * and calls endio, then active stripes will continue write
+ * process. Therefore, it is not necessary to check Faulty bit
+ * of journal device here.
+ *
+ * We can't check split_bio after current_bio is submitted. If
+ * io->split_bio is null, after current_bio is submitted, current_bio
+ * might already be completed and the io_unit is freed. We submit
+ * split_bio first to avoid the issue.
+ */
+ if (io->split_bio) {
+ if (io->has_flush)
+ io->split_bio->bi_opf |= REQ_PREFLUSH;
+ if (io->has_fua)
+ io->split_bio->bi_opf |= REQ_FUA;
+ submit_bio(io->split_bio);
+ }
+
if (io->has_flush)
io->current_bio->bi_opf |= REQ_PREFLUSH;
if (io->has_fua)
io->current_bio->bi_opf |= REQ_FUA;
submit_bio(io->current_bio);
-
- if (!io->split_bio)
- return;
-
- if (io->has_flush)
- io->split_bio->bi_opf |= REQ_PREFLUSH;
- if (io->has_fua)
- io->split_bio->bi_opf |= REQ_FUA;
- submit_bio(io->split_bio);
}
/* deferred io_unit will be dispatched here */
return;
pr_info("md/raid:%s: Disabling writeback cache for degraded array.\n",
mdname(mddev));
+
+ /* wait superblock change before suspend */
+ wait_event(mddev->sb_wait,
+ !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags));
+
mddev_suspend(mddev);
log->r5c_journal_mode = R5C_JOURNAL_MODE_WRITE_THROUGH;
mddev_resume(mddev);
* When run in degraded mode, array is set to write-through mode.
* This check helps drain pending write safely in the transition to
* write-through mode.
+ *
+ * When a stripe is syncing, the write is also handled in write
+ * through mode.
*/
- if (s->failed) {
+ if (s->failed || test_bit(STRIPE_SYNCING, &sh->state)) {
r5c_make_stripe_write_out(sh);
return -EAGAIN;
}
}
r5l_append_flush_payload(log, sh->sector);
+ /* stripe is flused to raid disks, we can do resync now */
+ if (test_bit(STRIPE_SYNC_REQUESTED, &sh->state))
+ set_bit(STRIPE_HANDLE, &sh->state);
}
int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh)
return ret;
}
-void r5c_update_on_rdev_error(struct mddev *mddev)
+void r5c_update_on_rdev_error(struct mddev *mddev, struct md_rdev *rdev)
{
struct r5conf *conf = mddev->private;
struct r5l_log *log = conf->log;
if (!log)
return;
- if (raid5_calc_degraded(conf) > 0 &&
+ if ((raid5_calc_degraded(conf) > 0 ||
+ test_bit(Journal, &rdev->flags)) &&
conf->log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_BACK)
schedule_work(&log->disable_writeback_work);
}
extern void r5c_check_stripe_cache_usage(struct r5conf *conf);
extern void r5c_check_cached_full_stripe(struct r5conf *conf);
extern struct md_sysfs_entry r5c_journal_mode;
-extern void r5c_update_on_rdev_error(struct mddev *mddev);
+extern void r5c_update_on_rdev_error(struct mddev *mddev,
+ struct md_rdev *rdev);
extern bool r5c_big_stripe_cached(struct r5conf *conf, sector_t sect);
extern struct dma_async_tx_descriptor *
static inline void lock_all_device_hash_locks_irq(struct r5conf *conf)
{
int i;
- local_irq_disable();
- spin_lock(conf->hash_locks);
+ spin_lock_irq(conf->hash_locks);
for (i = 1; i < NR_STRIPE_HASH_LOCKS; i++)
spin_lock_nest_lock(conf->hash_locks + i, conf->hash_locks);
spin_lock(&conf->device_lock);
{
int i;
spin_unlock(&conf->device_lock);
- for (i = NR_STRIPE_HASH_LOCKS; i; i--)
- spin_unlock(conf->hash_locks + i - 1);
- local_irq_enable();
+ for (i = NR_STRIPE_HASH_LOCKS - 1; i; i--)
+ spin_unlock(conf->hash_locks + i);
+ spin_unlock_irq(conf->hash_locks);
}
/* Find first data disk in a raid6 stripe */
if (test_bit(R5_InJournal, &sh->dev[i].flags))
injournal++;
/*
- * When quiesce in r5c write back, set STRIPE_HANDLE for stripes with
- * data in journal, so they are not released to cached lists
+ * In the following cases, the stripe cannot be released to cached
+ * lists. Therefore, we make the stripe write out and set
+ * STRIPE_HANDLE:
+ * 1. when quiesce in r5c write back;
+ * 2. when resync is requested fot the stripe.
*/
- if (conf->quiesce && r5c_is_writeback(conf->log) &&
- !test_bit(STRIPE_HANDLE, &sh->state) && injournal != 0) {
+ if (test_bit(STRIPE_SYNC_REQUESTED, &sh->state) ||
+ (conf->quiesce && r5c_is_writeback(conf->log) &&
+ !test_bit(STRIPE_HANDLE, &sh->state) && injournal != 0)) {
if (test_bit(STRIPE_R5C_CACHING, &sh->state))
r5c_make_stripe_write_out(sh);
set_bit(STRIPE_HANDLE, &sh->state);
static void lock_two_stripes(struct stripe_head *sh1, struct stripe_head *sh2)
{
- local_irq_disable();
if (sh1 > sh2) {
- spin_lock(&sh2->stripe_lock);
+ spin_lock_irq(&sh2->stripe_lock);
spin_lock_nested(&sh1->stripe_lock, 1);
} else {
- spin_lock(&sh1->stripe_lock);
+ spin_lock_irq(&sh1->stripe_lock);
spin_lock_nested(&sh2->stripe_lock, 1);
}
}
static void unlock_two_stripes(struct stripe_head *sh1, struct stripe_head *sh2)
{
spin_unlock(&sh1->stripe_lock);
- spin_unlock(&sh2->stripe_lock);
- local_irq_enable();
+ spin_unlock_irq(&sh2->stripe_lock);
}
/* Only freshly new full stripe normal write stripe can be added to a batch list */
struct stripe_head *osh, *nsh;
LIST_HEAD(newstripes);
struct disk_info *ndisks;
- int err;
+ int err = 0;
struct kmem_cache *sc;
int i;
int hash, cnt;
- err = md_allow_write(conf->mddev);
- if (err)
- return err;
+ md_allow_write(conf->mddev);
/* Step 1 */
sc = kmem_cache_create(conf->cache_name[1-conf->active_name],
bdevname(rdev->bdev, b),
mdname(mddev),
conf->raid_disks - mddev->degraded);
- r5c_update_on_rdev_error(mddev);
+ r5c_update_on_rdev_error(mddev, rdev);
}
/*
* When LOG_CRITICAL, stripes with injournal == 0 will be sent to
* no_space_stripes list.
*
+ * 3. during journal failure
+ * In journal failure, we try to flush all cached data to raid disks
+ * based on data in stripe cache. The array is read-only to upper
+ * layers, so we would skip all pending writes.
+ *
*/
static inline bool delay_towrite(struct r5conf *conf,
struct r5dev *dev,
if (test_bit(R5C_LOG_CRITICAL, &conf->cache_state) &&
s->injournal > 0)
return true;
+ /* case 3 above */
+ if (s->log_failed && s->injournal)
+ return true;
return false;
}
if (test_bit(STRIPE_SYNC_REQUESTED, &sh->state) && !sh->batch_head) {
spin_lock(&sh->stripe_lock);
- /* Cannot process 'sync' concurrently with 'discard' */
- if (!test_bit(STRIPE_DISCARD, &sh->state) &&
+ /*
+ * Cannot process 'sync' concurrently with 'discard'.
+ * Flush data in r5cache before 'sync'.
+ */
+ if (!test_bit(STRIPE_R5C_PARTIAL_STRIPE, &sh->state) &&
+ !test_bit(STRIPE_R5C_FULL_STRIPE, &sh->state) &&
+ !test_bit(STRIPE_DISCARD, &sh->state) &&
test_and_clear_bit(STRIPE_SYNC_REQUESTED, &sh->state)) {
set_bit(STRIPE_SYNCING, &sh->state);
clear_bit(STRIPE_INSYNC, &sh->state);
" to_write=%d failed=%d failed_num=%d,%d\n",
s.locked, s.uptodate, s.to_read, s.to_write, s.failed,
s.failed_num[0], s.failed_num[1]);
- /* check if the array has lost more than max_degraded devices and,
+ /*
+ * check if the array has lost more than max_degraded devices and,
* if so, some requests might need to be failed.
+ *
+ * When journal device failed (log_failed), we will only process
+ * the stripe if there is data need write to raid disks
*/
- if (s.failed > conf->max_degraded || s.log_failed) {
+ if (s.failed > conf->max_degraded ||
+ (s.log_failed && s.injournal == 0)) {
sh->check_state = 0;
sh->reconstruct_state = 0;
break_stripe_batch_list(sh, 0);
struct stripe_head *sh, *tmp;
struct list_head *handle_list = NULL;
struct r5worker_group *wg;
- bool second_try = !r5c_is_writeback(conf->log);
- bool try_loprio = test_bit(R5C_LOG_TIGHT, &conf->cache_state);
+ bool second_try = !r5c_is_writeback(conf->log) &&
+ !r5l_log_disk_error(conf);
+ bool try_loprio = test_bit(R5C_LOG_TIGHT, &conf->cache_state) ||
+ r5l_log_disk_error(conf);
again:
wg = NULL;
raid5_set_cache_size(struct mddev *mddev, int size)
{
struct r5conf *conf = mddev->private;
- int err;
if (size <= 16 || size > 32768)
return -EINVAL;
;
mutex_unlock(&conf->cache_size_mutex);
-
- err = md_allow_write(mddev);
- if (err)
- return err;
+ md_allow_write(mddev);
mutex_lock(&conf->cache_size_mutex);
while (size > conf->max_nr_stripes)
* neilb: there is no locking about new writes here,
* so this cannot be safe.
*/
- if (atomic_read(&conf->active_stripes)) {
+ if (atomic_read(&conf->active_stripes) ||
+ atomic_read(&conf->r5c_cached_full_stripes) ||
+ atomic_read(&conf->r5c_cached_partial_stripes)) {
return -EBUSY;
}
log_exit(conf);
pr_info("gpmc cs%i access configuration:\n", cs);
GPMC_GET_RAW_BOOL(GPMC_CS_CONFIG1, 4, 4, "time-para-granularity");
GPMC_GET_RAW(GPMC_CS_CONFIG1, 8, 9, "mux-add-data");
- GPMC_GET_RAW_MAX(GPMC_CS_CONFIG1, 12, 13,
+ GPMC_GET_RAW_SHIFT_MAX(GPMC_CS_CONFIG1, 12, 13, 1,
GPMC_CONFIG1_DEVICESIZE_MAX, "device-width");
GPMC_GET_RAW(GPMC_CS_CONFIG1, 16, 17, "wait-pin");
GPMC_GET_RAW_BOOL(GPMC_CS_CONFIG1, 21, 21, "wait-on-write");
config PCI_ENDPOINT_TEST
depends on PCI
+ select CRC32
tristate "PCI Endpoint Test driver"
---help---
Enable this configuration option to enable the host side test driver
struct mmc_pwrseq pwrseq;
bool clk_enabled;
u32 post_power_on_delay_ms;
+ u32 power_off_delay_us;
struct clk *ext_clk;
struct gpio_descs *reset_gpios;
};
mmc_pwrseq_simple_set_gpios_value(pwrseq, 1);
+ if (pwrseq->power_off_delay_us)
+ usleep_range(pwrseq->power_off_delay_us,
+ 2 * pwrseq->power_off_delay_us);
+
if (!IS_ERR(pwrseq->ext_clk) && pwrseq->clk_enabled) {
clk_disable_unprepare(pwrseq->ext_clk);
pwrseq->clk_enabled = false;
device_property_read_u32(dev, "post-power-on-delay-ms",
&pwrseq->post_power_on_delay_ms);
+ device_property_read_u32(dev, "power-off-delay-us",
+ &pwrseq->power_off_delay_us);
pwrseq->pwrseq.dev = dev;
pwrseq->pwrseq.ops = &mmc_pwrseq_simple_ops;
static void octeon_mmc_int_enable(struct cvm_mmc_host *host, u64 val)
{
writeq(val, host->base + MIO_EMM_INT(host));
- if (!host->dma_active || (host->dma_active && !host->has_ciu3))
+ if (!host->has_ciu3)
writeq(val, host->base + MIO_EMM_INT_EN(host));
}
}
host->global_pwr_gpiod = devm_gpiod_get_optional(&pdev->dev,
- "power-gpios",
+ "power",
GPIOD_OUT_HIGH);
if (IS_ERR(host->global_pwr_gpiod)) {
dev_err(&pdev->dev, "Invalid power GPIO\n");
if (ret) {
dev_err(&pdev->dev, "Error populating slots\n");
octeon_mmc_set_shared_power(host, 0);
- return ret;
+ goto error;
}
i++;
}
return 0;
+
+error:
+ for (i = 0; i < CAVIUM_MAX_MMC; i++) {
+ if (host->slot[i])
+ cvm_mmc_of_slot_remove(host->slot[i]);
+ if (host->slot_pdev[i])
+ of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL);
+ }
+ return ret;
}
static int octeon_mmc_remove(struct platform_device *pdev)
return 0;
error:
+ for (i = 0; i < CAVIUM_MAX_MMC; i++) {
+ if (host->slot[i])
+ cvm_mmc_of_slot_remove(host->slot[i]);
+ if (host->slot_pdev[i])
+ of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL);
+ }
clk_disable_unprepare(host->clk);
return ret;
}
cvm_mmc_reset_bus(slot);
if (host->global_pwr_gpiod)
host->set_shared_power(host, 0);
- else
+ else if (!IS_ERR(mmc->supply.vmmc))
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, 0);
break;
case MMC_POWER_UP:
if (host->global_pwr_gpiod)
host->set_shared_power(host, 1);
- else
+ else if (!IS_ERR(mmc->supply.vmmc))
mmc_regulator_set_ocr(mmc, mmc->supply.vmmc, ios->vdd);
break;
}
return -EINVAL;
}
- mmc->supply.vmmc = devm_regulator_get_optional(dev, "vmmc");
- if (IS_ERR(mmc->supply.vmmc)) {
- if (PTR_ERR(mmc->supply.vmmc) == -EPROBE_DEFER)
- return -EPROBE_DEFER;
- /*
- * Legacy Octeon firmware has no regulator entry, fall-back to
- * a hard-coded voltage to get a sane OCR.
- */
+ ret = mmc_regulator_get_supply(mmc);
+ if (ret == -EPROBE_DEFER)
+ return ret;
+ /*
+ * Legacy Octeon firmware has no regulator entry, fall-back to
+ * a hard-coded voltage to get a sane OCR.
+ */
+ if (IS_ERR(mmc->supply.vmmc))
mmc->ocr_avail = MMC_VDD_32_33 | MMC_VDD_33_34;
- } else {
- ret = mmc_regulator_get_ocrmask(mmc->supply.vmmc);
- if (ret > 0)
- mmc->ocr_avail = ret;
- }
/* Common MMC bindings */
ret = mmc_of_parse(mmc);
};
static const struct sdhci_pltfm_data sdhci_iproc_pltfm_data = {
- .quirks = SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK,
+ .quirks = SDHCI_QUIRK_DATA_TIMEOUT_USES_SDCLK |
+ SDHCI_QUIRK_MULTIBLOCK_READ_ACMD12,
.quirks2 = SDHCI_QUIRK2_ACMD23_BROKEN,
.ops = &sdhci_iproc_ops,
};
return ret;
}
-void xenon_clean_phy(struct sdhci_host *host)
-{
- struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
- struct xenon_priv *priv = sdhci_pltfm_priv(pltfm_host);
-
- kfree(priv->phy_params);
-}
-
static int xenon_add_phy(struct device_node *np, struct sdhci_host *host,
const char *phy_name)
{
if (ret)
return ret;
- ret = xenon_emmc_phy_parse_param_dt(host, np, priv->phy_params);
- if (ret)
- xenon_clean_phy(host);
-
- return ret;
+ return xenon_emmc_phy_parse_param_dt(host, np, priv->phy_params);
}
int xenon_phy_parse_dt(struct device_node *np, struct sdhci_host *host)
err = xenon_sdhc_prepare(host);
if (err)
- goto clean_phy_param;
+ goto err_clk;
err = sdhci_add_host(host);
if (err)
remove_sdhc:
xenon_sdhc_unprepare(host);
-clean_phy_param:
- xenon_clean_phy(host);
err_clk:
clk_disable_unprepare(pltfm_host->clk);
free_pltfm:
struct sdhci_host *host = platform_get_drvdata(pdev);
struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
- xenon_clean_phy(host);
-
sdhci_remove_host(host, 0);
xenon_sdhc_unprepare(host);
};
int xenon_phy_adj(struct sdhci_host *host, struct mmc_ios *ios);
-void xenon_clean_phy(struct sdhci_host *host);
int xenon_phy_parse_dt(struct device_node *np,
struct sdhci_host *host);
void xenon_soc_pad_ctrl(struct sdhci_host *host,
return -1;
ad_info->aggregator_id = aggregator->aggregator_identifier;
- ad_info->ports = aggregator->num_of_ports;
+ ad_info->ports = __agg_active_ports(aggregator);
ad_info->actor_key = aggregator->actor_oper_aggregator_key;
ad_info->partner_key = aggregator->partner_oper_aggregator_key;
ether_addr_copy(ad_info->partner_system,
bond_for_each_slave_rcu(bond, slave, iter) {
unsigned long trans_start = dev_trans_start(slave->dev);
+ slave->new_link = BOND_LINK_NOCHANGE;
+
if (slave->link != BOND_LINK_UP) {
if (bond_time_in_interval(bond, trans_start, 1) &&
bond_time_in_interval(bond, slave->last_rx, 1)) {
- slave->link = BOND_LINK_UP;
+ slave->new_link = BOND_LINK_UP;
slave_state_changed = 1;
/* primary_slave has no meaning in round-robin
if (!bond_time_in_interval(bond, trans_start, 2) ||
!bond_time_in_interval(bond, slave->last_rx, 2)) {
- slave->link = BOND_LINK_DOWN;
+ slave->new_link = BOND_LINK_DOWN;
slave_state_changed = 1;
if (slave->link_failure_count < UINT_MAX)
if (!rtnl_trylock())
goto re_arm;
+ bond_for_each_slave(bond, slave, iter) {
+ if (slave->new_link != BOND_LINK_NOCHANGE)
+ slave->link = slave->new_link;
+ }
+
if (slave_state_changed) {
bond_slave_state_change(bond);
if (BOND_MODE(bond) == BOND_MODE_XOR)
int arp_validate_value, fail_over_mac_value, primary_reselect_value, i;
struct bond_opt_value newval;
const struct bond_opt_value *valptr;
- int arp_all_targets_value;
+ int arp_all_targets_value = 0;
u16 ad_actor_sys_prio = 0;
u16 ad_user_port_key = 0;
- __be32 arp_target[BOND_MAX_ARP_TARGETS];
+ __be32 arp_target[BOND_MAX_ARP_TARGETS] = { 0 };
int arp_ip_count;
int bond_mode = BOND_MODE_ROUNDROBIN;
int xmit_hashtype = BOND_XMIT_POLICY_LAYER2;
arp_validate_value = 0;
}
- arp_all_targets_value = 0;
if (arp_all_targets) {
bond_opt_initstr(&newval, arp_all_targets);
valptr = bond_opt_parse(bond_opt_get(BOND_OPT_ARP_ALL_TARGETS),
mv88e6xxx_g1_stats_read(chip, reg, &low);
if (s->sizeof_stat == 8)
mv88e6xxx_g1_stats_read(chip, reg + 1, &high);
+ break;
+ default:
+ return UINT64_MAX;
}
value = (((u64)high) << 16) | low;
return value;
ret = ax_mii_init(dev);
if (ret)
- goto out_irq;
+ goto err_out;
ax_NS8390_init(dev, 0);
ret = register_netdev(dev);
if (ret)
- goto out_irq;
+ goto err_out;
netdev_info(dev, "%dbit, irq %d, %lx, MAC: %pM\n",
ei_local->word16 ? 16 : 8, dev->irq, dev->base_addr,
return 0;
- out_irq:
- /* cleanup irq */
- free_irq(dev->irq, dev);
err_out:
return ret;
}
static int hw_atl_a0_hw_offload_set(struct aq_hw_s *self,
struct aq_nic_cfg_s *aq_nic_cfg)
{
- int err = 0;
-
/* TX checksums offloads*/
tpo_ipv4header_crc_offload_en_set(self, 1);
tpo_tcp_udp_crc_offload_en_set(self, 1);
- if (err < 0)
- goto err_exit;
/* RX checksums offloads*/
rpo_ipv4header_crc_offload_en_set(self, 1);
rpo_tcp_udp_crc_offload_en_set(self, 1);
- if (err < 0)
- goto err_exit;
/* LSO offloads*/
tdm_large_send_offload_en_set(self, 0xFFFFFFFFU);
- if (err < 0)
- goto err_exit;
-
- err = aq_hw_err_from_flags(self);
-err_exit:
- return err;
+ return aq_hw_err_from_flags(self);
}
static int hw_atl_a0_hw_init_tx_path(struct aq_hw_s *self)
static int hw_atl_b0_hw_offload_set(struct aq_hw_s *self,
struct aq_nic_cfg_s *aq_nic_cfg)
{
- int err = 0;
unsigned int i;
/* TX checksums offloads*/
tpo_ipv4header_crc_offload_en_set(self, 1);
tpo_tcp_udp_crc_offload_en_set(self, 1);
- if (err < 0)
- goto err_exit;
/* RX checksums offloads*/
rpo_ipv4header_crc_offload_en_set(self, 1);
rpo_tcp_udp_crc_offload_en_set(self, 1);
- if (err < 0)
- goto err_exit;
/* LSO offloads*/
tdm_large_send_offload_en_set(self, 0xFFFFFFFFU);
- if (err < 0)
- goto err_exit;
/* LRO offloads */
{
rpo_lro_en_set(self, aq_nic_cfg->is_lro ? 0xFFFFFFFFU : 0U);
}
- err = aq_hw_err_from_flags(self);
-
-err_exit:
- return err;
+ return aq_hw_err_from_flags(self);
}
static int hw_atl_b0_hw_init_tx_path(struct aq_hw_s *self)
if (pci_set_dma_mask(pdev, DMA_BIT_MASK(32)) &&
pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32))) {
printk(KERN_ERR "atl2: No usable DMA configuration, aborting\n");
+ err = -EIO;
goto err_dma;
}
* pcibios_set_master to do the needed arch specific settings */
pci_set_master(pdev);
- err = -ENOMEM;
netdev = alloc_etherdev(sizeof(struct atl2_adapter));
- if (!netdev)
+ if (!netdev) {
+ err = -ENOMEM;
goto err_alloc_etherdev;
+ }
SET_NETDEV_DEV(netdev, &pdev->dev);
if (err)
goto err_sw_init;
- err = -EIO;
-
netdev->hw_features = NETIF_F_HW_VLAN_CTAG_RX;
netdev->features |= (NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX);
dev->min_mtu = ETH_ZLEN;
dev->max_mtu = BNXT_MAX_MTU;
- bnxt_dcb_init(bp);
-
#ifdef CONFIG_BNXT_SRIOV
init_waitqueue_head(&bp->sriov_cfg_wait);
#endif
bnxt_hwrm_func_qcfg(bp);
bnxt_hwrm_port_led_qcaps(bp);
bnxt_ethtool_init(bp);
+ bnxt_dcb_init(bp);
bnxt_set_rx_skb_mode(bp, false);
bnxt_set_tpa_flags(bp);
if ((mode & DCB_CAP_DCBX_VER_CEE) || !(mode & DCB_CAP_DCBX_VER_IEEE))
return 1;
- if ((mode & DCB_CAP_DCBX_HOST) && BNXT_VF(bp))
- return 1;
+ if (mode & DCB_CAP_DCBX_HOST) {
+ if (BNXT_VF(bp) || (bp->flags & BNXT_FLAG_FW_LLDP_AGENT))
+ return 1;
+ }
if (mode == bp->dcbx_cap)
return 0;
#define T4FW_VERSION_MAJOR 0x01
#define T4FW_VERSION_MINOR 0x10
-#define T4FW_VERSION_MICRO 0x21
+#define T4FW_VERSION_MICRO 0x2B
#define T4FW_VERSION_BUILD 0x00
#define T4FW_MIN_VERSION_MAJOR 0x01
#define T5FW_VERSION_MAJOR 0x01
#define T5FW_VERSION_MINOR 0x10
-#define T5FW_VERSION_MICRO 0x21
+#define T5FW_VERSION_MICRO 0x2B
#define T5FW_VERSION_BUILD 0x00
#define T5FW_MIN_VERSION_MAJOR 0x00
#define T6FW_VERSION_MAJOR 0x01
#define T6FW_VERSION_MINOR 0x10
-#define T6FW_VERSION_MICRO 0x21
+#define T6FW_VERSION_MICRO 0x2B
#define T6FW_VERSION_BUILD 0x00
#define T6FW_MIN_VERSION_MAJOR 0x00
struct be_adapter *adapter = netdev_priv(dev);
u8 l4_hdr = 0;
- /* The code below restricts offload features for some tunneled packets.
+ /* The code below restricts offload features for some tunneled and
+ * Q-in-Q packets.
* Offload features for normal (non tunnel) packets are unchanged.
*/
+ features = vlan_features_check(skb, features);
if (!skb->encapsulation ||
!(adapter->flags & BE_FLAGS_VXLAN_OFFLOADS))
return features;
return 0;
}
+static const struct of_device_id ftmac100_of_ids[] = {
+ { .compatible = "andestech,atmac100" },
+ { }
+};
+
static struct platform_driver ftmac100_driver = {
.probe = ftmac100_probe,
.remove = ftmac100_remove,
.driver = {
.name = DRV_NAME,
+ .of_match_table = ftmac100_of_ids
},
};
MODULE_AUTHOR("Po-Yu Chuang <ratbert@faraday-tech.com>");
MODULE_DESCRIPTION("FTMAC100 driver");
MODULE_LICENSE("GPL");
+MODULE_DEVICE_TABLE(of, ftmac100_of_ids);
{
int err, phy_reset;
bool active_high = false;
- int msec = 1;
+ int msec = 1, phy_post_delay = 0;
struct device_node *np = pdev->dev.of_node;
if (!np)
else if (!gpio_is_valid(phy_reset))
return 0;
+ err = of_property_read_u32(np, "phy-reset-post-delay", &phy_post_delay);
+ /* valid reset duration should be less than 1s */
+ if (!err && phy_post_delay > 1000)
+ return -EINVAL;
+
active_high = of_property_read_bool(np, "phy-reset-active-high");
err = devm_gpio_request_one(&pdev->dev, phy_reset,
gpio_set_value_cansleep(phy_reset, !active_high);
+ if (!phy_post_delay)
+ return 0;
+
+ if (phy_post_delay > 20)
+ msleep(phy_post_delay);
+ else
+ usleep_range(phy_post_delay * 1000,
+ phy_post_delay * 1000 + 1000);
+
return 0;
}
#else /* CONFIG_OF */
int port = 0;
if (msi_x) {
- int nreq = dev->caps.num_ports * num_online_cpus() + 1;
-
- nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
- nreq);
- if (nreq > MAX_MSIX)
- nreq = MAX_MSIX;
+ int nreq = min3(dev->caps.num_ports *
+ (int)num_online_cpus() + 1,
+ dev->caps.num_eqs - dev->caps.reserved_eqs,
+ MAX_MSIX);
entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
if (!entries)
config MLX5_CORE_EN
bool "Mellanox Technologies ConnectX-4 Ethernet support"
- depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
+ depends on NETDEVICES && ETHERNET && INET && PCI && MLX5_CORE
depends on IPV6=y || IPV6=n || MLX5_CORE=m
imply PTP_1588_CLOCK
default n
mlx5_core_warn(dev, "%s(0x%x) timeout. Will cause a leak of a command resource\n",
mlx5_command_str(msg_to_opcode(ent->in)),
msg_to_opcode(ent->in));
- mlx5_cmd_comp_handler(dev, 1UL << ent->idx);
+ mlx5_cmd_comp_handler(dev, 1UL << ent->idx, true);
}
static void cmd_work_handler(struct work_struct *work)
}
cmd->ent_arr[ent->idx] = ent;
+ set_bit(MLX5_CMD_ENT_STATE_PENDING_COMP, &ent->state);
lay = get_inst(cmd, ent->idx);
ent->lay = lay;
memset(lay, 0, sizeof(*lay));
if (ent->callback)
schedule_delayed_work(&ent->cb_timeout_work, cb_timeout);
+ /* Skip sending command to fw if internal error */
+ if (pci_channel_offline(dev->pdev) ||
+ dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) {
+ u8 status = 0;
+ u32 drv_synd;
+
+ ent->ret = mlx5_internal_err_ret_value(dev, msg_to_opcode(ent->in), &drv_synd, &status);
+ MLX5_SET(mbox_out, ent->out, status, status);
+ MLX5_SET(mbox_out, ent->out, syndrome, drv_synd);
+
+ mlx5_cmd_comp_handler(dev, 1UL << ent->idx, true);
+ return;
+ }
+
/* ring doorbell after the descriptor is valid */
mlx5_core_dbg(dev, "writing 0x%x to command doorbell\n", 1 << ent->idx);
wmb();
poll_timeout(ent);
/* make sure we read the descriptor after ownership is SW */
rmb();
- mlx5_cmd_comp_handler(dev, 1UL << ent->idx);
+ mlx5_cmd_comp_handler(dev, 1UL << ent->idx, (ent->ret == -ETIMEDOUT));
}
}
wait_for_completion(&ent->done);
} else if (!wait_for_completion_timeout(&ent->done, timeout)) {
ent->ret = -ETIMEDOUT;
- mlx5_cmd_comp_handler(dev, 1UL << ent->idx);
+ mlx5_cmd_comp_handler(dev, 1UL << ent->idx, true);
}
err = ent->ret;
}
}
-void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec)
+void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool forced)
{
struct mlx5_cmd *cmd = &dev->cmd;
struct mlx5_cmd_work_ent *ent;
struct semaphore *sem;
ent = cmd->ent_arr[i];
+
+ /* if we already completed the command, ignore it */
+ if (!test_and_clear_bit(MLX5_CMD_ENT_STATE_PENDING_COMP,
+ &ent->state)) {
+ /* only real completion can free the cmd slot */
+ if (!forced) {
+ mlx5_core_err(dev, "Command completion arrived after timeout (entry idx = %d).\n",
+ ent->idx);
+ free_ent(cmd, ent->idx);
+ }
+ continue;
+ }
+
if (ent->callback)
cancel_delayed_work(&ent->cb_timeout_work);
if (ent->page_queue)
mlx5_core_dbg(dev, "command completed. ret 0x%x, delivery status %s(0x%x)\n",
ent->ret, deliv_status_to_str(ent->status), ent->status);
}
- free_ent(cmd, ent->idx);
+
+ /* only real completion will free the entry slot */
+ if (!forced)
+ free_ent(cmd, ent->idx);
if (ent->callback) {
ds = ent->ts2 - ent->ts1;
void mlx5e_destroy_direct_tirs(struct mlx5e_priv *priv);
void mlx5e_destroy_rqt(struct mlx5e_priv *priv, struct mlx5e_rqt *rqt);
-int mlx5e_create_ttc_table(struct mlx5e_priv *priv, u32 underlay_qpn);
+int mlx5e_create_ttc_table(struct mlx5e_priv *priv);
void mlx5e_destroy_ttc_table(struct mlx5e_priv *priv);
int mlx5e_create_tis(struct mlx5_core_dev *mdev, int tc,
ptys2ethtool_supported_port(link_ksettings, eth_proto_cap);
ptys2ethtool_supported_link(supported, eth_proto_cap);
ethtool_link_ksettings_add_link_mode(link_ksettings, supported, Pause);
- ethtool_link_ksettings_add_link_mode(link_ksettings, supported, Asym_Pause);
}
static void get_advertising(u32 eth_proto_cap, u8 tx_pause,
unsigned long *advertising = link_ksettings->link_modes.advertising;
ptys2ethtool_adver_link(advertising, eth_proto_cap);
- if (tx_pause)
+ if (rx_pause)
ethtool_link_ksettings_add_link_mode(link_ksettings, advertising, Pause);
if (tx_pause ^ rx_pause)
ethtool_link_ksettings_add_link_mode(link_ksettings, advertising, Asym_Pause);
struct mlx5e_priv *priv = netdev_priv(netdev);
struct mlx5_core_dev *mdev = priv->mdev;
u32 out[MLX5_ST_SZ_DW(ptys_reg)] = {0};
+ u32 rx_pause = 0;
+ u32 tx_pause = 0;
u32 eth_proto_cap;
u32 eth_proto_admin;
u32 eth_proto_lp;
an_disable_admin = MLX5_GET(ptys_reg, out, an_disable_admin);
an_status = MLX5_GET(ptys_reg, out, an_status);
+ mlx5_query_port_pause(mdev, &rx_pause, &tx_pause);
+
ethtool_link_ksettings_zero_link_mode(link_ksettings, supported);
ethtool_link_ksettings_zero_link_mode(link_ksettings, advertising);
get_supported(eth_proto_cap, link_ksettings);
- get_advertising(eth_proto_admin, 0, 0, link_ksettings);
+ get_advertising(eth_proto_admin, tx_pause, rx_pause, link_ksettings);
get_speed_duplex(netdev, eth_proto_oper, link_ksettings);
eth_proto_oper = eth_proto_oper ? eth_proto_oper : eth_proto_cap;
mlx5e_destroy_flow_table(&ttc->ft);
}
-int mlx5e_create_ttc_table(struct mlx5e_priv *priv, u32 underlay_qpn)
+int mlx5e_create_ttc_table(struct mlx5e_priv *priv)
{
struct mlx5e_ttc_table *ttc = &priv->fs.ttc;
struct mlx5_flow_table_attr ft_attr = {};
ft_attr.max_fte = MLX5E_TTC_TABLE_SIZE;
ft_attr.level = MLX5E_TTC_FT_LEVEL;
ft_attr.prio = MLX5E_NIC_PRIO;
- ft_attr.underlay_qpn = underlay_qpn;
ft->t = mlx5_create_flow_table(priv->fs.ns, &ft_attr);
if (IS_ERR(ft->t)) {
priv->netdev->hw_features &= ~NETIF_F_NTUPLE;
}
- err = mlx5e_create_ttc_table(priv, 0);
+ err = mlx5e_create_ttc_table(priv);
if (err) {
netdev_err(priv->netdev, "Failed to create ttc table, err=%d\n",
err);
new_channels.params = priv->channels.params;
new_channels.params.num_tc = tc ? tc : 1;
- if (test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
priv->channels.params = new_channels.params;
goto out;
}
#define MLX5_IB_GRH_BYTES 40
#define MLX5_IPOIB_ENCAP_LEN 4
#define MLX5_GID_SIZE 16
+#define MLX5_IPOIB_PSEUDO_LEN 20
+#define MLX5_IPOIB_HARD_LEN (MLX5_IPOIB_PSEUDO_LEN + MLX5_IPOIB_ENCAP_LEN)
static inline void mlx5i_complete_rx_cqe(struct mlx5e_rq *rq,
struct mlx5_cqe64 *cqe,
struct sk_buff *skb)
{
struct net_device *netdev = rq->netdev;
+ char *pseudo_header;
u8 *dgid;
u8 g;
if (likely(netdev->features & NETIF_F_RXHASH))
mlx5e_skb_set_hash(cqe, skb);
+ /* 20 bytes of ipoib header and 4 for encap existing */
+ pseudo_header = skb_push(skb, MLX5_IPOIB_PSEUDO_LEN);
+ memset(pseudo_header, 0, MLX5_IPOIB_PSEUDO_LEN);
skb_reset_mac_header(skb);
- skb_pull(skb, MLX5_IPOIB_ENCAP_LEN);
+ skb_pull(skb, MLX5_IPOIB_HARD_LEN);
skb->dev = netdev;
#include <net/tc_act/tc_vlan.h>
#include <net/tc_act/tc_tunnel_key.h>
#include <net/tc_act/tc_pedit.h>
+#include <net/tc_act/tc_csum.h>
#include <net/vxlan.h>
#include <net/arp.h>
#include "en.h"
if (e->flags & MLX5_ENCAP_ENTRY_VALID)
mlx5_encap_dealloc(priv->mdev, e->encap_id);
- hlist_del_rcu(&e->encap_hlist);
+ hash_del_rcu(&e->encap_hlist);
kfree(e->encap_header);
kfree(e);
}
struct mlx5e_tc_flow_parse_attr *parse_attr)
{
struct pedit_headers *set_masks, *add_masks, *set_vals, *add_vals;
- int i, action_size, nactions, max_actions, first, last;
+ int i, action_size, nactions, max_actions, first, last, first_z;
void *s_masks_p, *a_masks_p, *vals_p;
- u32 s_mask, a_mask, val;
struct mlx5_fields *f;
u8 cmd, field_bsize;
+ u32 s_mask, a_mask;
unsigned long mask;
void *action;
for (i = 0; i < ARRAY_SIZE(fields); i++) {
f = &fields[i];
/* avoid seeing bits set from previous iterations */
- s_mask = a_mask = mask = val = 0;
+ s_mask = 0;
+ a_mask = 0;
s_masks_p = (void *)set_masks + f->offset;
a_masks_p = (void *)add_masks + f->offset;
memset(a_masks_p, 0, f->size);
}
- memcpy(&val, vals_p, f->size);
-
field_bsize = f->size * BITS_PER_BYTE;
+
+ first_z = find_first_zero_bit(&mask, field_bsize);
first = find_first_bit(&mask, field_bsize);
last = find_last_bit(&mask, field_bsize);
- if (first > 0 || last != (field_bsize - 1)) {
+ if (first > 0 || last != (field_bsize - 1) || first_z < last) {
printk(KERN_WARNING "mlx5: partial rewrite (mask %lx) is currently not offloaded\n",
mask);
return -EOPNOTSUPP;
}
if (field_bsize == 32)
- MLX5_SET(set_action_in, action, data, ntohl(val));
+ MLX5_SET(set_action_in, action, data, ntohl(*(__be32 *)vals_p));
else if (field_bsize == 16)
- MLX5_SET(set_action_in, action, data, ntohs(val));
+ MLX5_SET(set_action_in, action, data, ntohs(*(__be16 *)vals_p));
else if (field_bsize == 8)
- MLX5_SET(set_action_in, action, data, val);
+ MLX5_SET(set_action_in, action, data, *(u8 *)vals_p);
action += action_size;
nactions++;
return err;
}
+static bool csum_offload_supported(struct mlx5e_priv *priv, u32 action, u32 update_flags)
+{
+ u32 prot_flags = TCA_CSUM_UPDATE_FLAG_IPV4HDR | TCA_CSUM_UPDATE_FLAG_TCP |
+ TCA_CSUM_UPDATE_FLAG_UDP;
+
+ /* The HW recalcs checksums only if re-writing headers */
+ if (!(action & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)) {
+ netdev_warn(priv->netdev,
+ "TC csum action is only offloaded with pedit\n");
+ return false;
+ }
+
+ if (update_flags & ~prot_flags) {
+ netdev_warn(priv->netdev,
+ "can't offload TC csum action for some header/s - flags %#x\n",
+ update_flags);
+ return false;
+ }
+
+ return true;
+}
+
static int parse_tc_nic_actions(struct mlx5e_priv *priv, struct tcf_exts *exts,
struct mlx5e_tc_flow_parse_attr *parse_attr,
struct mlx5e_tc_flow *flow)
continue;
}
+ if (is_tcf_csum(a)) {
+ if (csum_offload_supported(priv, attr->action,
+ tcf_csum_update_flags(a)))
+ continue;
+
+ return -EOPNOTSUPP;
+ }
+
if (is_tcf_skbedit_mark(a)) {
u32 mark = tcf_skbedit_mark(a);
continue;
}
+ if (is_tcf_csum(a)) {
+ if (csum_offload_supported(priv, attr->action,
+ tcf_csum_update_flags(a)))
+ continue;
+
+ return -EOPNOTSUPP;
+ }
+
if (is_tcf_mirred_egress_redirect(a)) {
int ifindex = tcf_mirred_ifindex(a);
struct net_device *out_dev, *encap_dev = NULL;
break;
case MLX5_EVENT_TYPE_CMD:
- mlx5_cmd_comp_handler(dev, be32_to_cpu(eqe->data.cmd.vector));
+ mlx5_cmd_comp_handler(dev, be32_to_cpu(eqe->data.cmd.vector), false);
break;
case MLX5_EVENT_TYPE_PORT_CHANGE:
#include "eswitch.h"
int mlx5_cmd_update_root_ft(struct mlx5_core_dev *dev,
- struct mlx5_flow_table *ft)
+ struct mlx5_flow_table *ft, u32 underlay_qpn)
{
u32 in[MLX5_ST_SZ_DW(set_flow_table_root_in)] = {0};
u32 out[MLX5_ST_SZ_DW(set_flow_table_root_out)] = {0};
if ((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) &&
- ft->underlay_qpn == 0)
+ underlay_qpn == 0)
return 0;
MLX5_SET(set_flow_table_root_in, in, opcode,
MLX5_CMD_OP_SET_FLOW_TABLE_ROOT);
MLX5_SET(set_flow_table_root_in, in, table_type, ft->type);
MLX5_SET(set_flow_table_root_in, in, table_id, ft->id);
+ MLX5_SET(set_flow_table_root_in, in, underlay_qpn, underlay_qpn);
if (ft->vport) {
MLX5_SET(set_flow_table_root_in, in, vport_number, ft->vport);
MLX5_SET(set_flow_table_root_in, in, other_vport, 1);
}
- if ((MLX5_CAP_GEN(dev, port_type) == MLX5_CAP_PORT_TYPE_IB) &&
- ft->underlay_qpn != 0)
- MLX5_SET(set_flow_table_root_in, in, underlay_qpn, ft->underlay_qpn);
-
return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
}
unsigned int index);
int mlx5_cmd_update_root_ft(struct mlx5_core_dev *dev,
- struct mlx5_flow_table *ft);
+ struct mlx5_flow_table *ft,
+ u32 underlay_qpn);
int mlx5_cmd_fc_alloc(struct mlx5_core_dev *dev, u16 *id);
int mlx5_cmd_fc_free(struct mlx5_core_dev *dev, u16 id);
if (ft->level >= min_level)
return 0;
- err = mlx5_cmd_update_root_ft(root->dev, ft);
+ err = mlx5_cmd_update_root_ft(root->dev, ft, root->underlay_qpn);
if (err)
mlx5_core_warn(root->dev, "Update root flow table of id=%u failed\n",
ft->id);
goto unlock_root;
}
- ft->underlay_qpn = ft_attr->underlay_qpn;
-
tree_init_node(&ft->node, 1, del_flow_table);
log_table_sz = ft->max_fte ? ilog2(ft->max_fte) : 0;
next_ft = find_next_chained_ft(fs_prio);
new_root_ft = find_next_ft(ft);
if (new_root_ft) {
- int err = mlx5_cmd_update_root_ft(root->dev, new_root_ft);
+ int err = mlx5_cmd_update_root_ft(root->dev, new_root_ft,
+ root->underlay_qpn);
if (err) {
mlx5_core_warn(root->dev, "Update root flow table of id=%u failed\n",
mlx5_cleanup_fs(dev);
return err;
}
+
+int mlx5_fs_add_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn)
+{
+ struct mlx5_flow_root_namespace *root = dev->priv.steering->root_ns;
+
+ root->underlay_qpn = underlay_qpn;
+ return 0;
+}
+EXPORT_SYMBOL(mlx5_fs_add_rx_underlay_qpn);
+
+int mlx5_fs_remove_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn)
+{
+ struct mlx5_flow_root_namespace *root = dev->priv.steering->root_ns;
+
+ root->underlay_qpn = 0;
+ return 0;
+}
+EXPORT_SYMBOL(mlx5_fs_remove_rx_underlay_qpn);
/* FWD rules that point on this flow table */
struct list_head fwd_rules;
u32 flags;
- u32 underlay_qpn;
};
struct mlx5_fc_cache {
struct mlx5_flow_table *root_ft;
/* Should be held when chaining flow tables */
struct mutex chain_lock;
+ u32 underlay_qpn;
};
int mlx5_init_fc_stats(struct mlx5_core_dev *dev);
spin_unlock_irqrestore(&dev->cmd.alloc_lock, flags);
mlx5_core_dbg(dev, "vector 0x%llx\n", vector);
- mlx5_cmd_comp_handler(dev, vector);
+ mlx5_cmd_comp_handler(dev, vector, true);
return;
no_trig:
mlx5e_build_nic_params(mdev, &priv->channels.params, profile->max_nch(mdev));
+ /* Override RQ params as IPoIB supports only LINKED LIST RQ for now */
+ mlx5e_set_rq_type_params(mdev, &priv->channels.params, MLX5_WQ_TYPE_LINKED_LIST);
+ priv->channels.params.lro_en = false;
+
mutex_init(&priv->state_lock);
netdev->hw_features |= NETIF_F_SG;
static void mlx5i_destroy_underlay_qp(struct mlx5_core_dev *mdev, struct mlx5_core_qp *qp)
{
+ mlx5_fs_remove_rx_underlay_qpn(mdev, qp->qpn);
+
mlx5_core_destroy_qp(mdev, qp);
}
return err;
}
+ mlx5_fs_add_rx_underlay_qpn(priv->mdev, ipriv->qp.qpn);
+
err = mlx5e_create_tis(priv->mdev, 0 /* tc */, ipriv->qp.qpn, &priv->tisn[0]);
if (err) {
mlx5_core_warn(priv->mdev, "create tis failed, %d\n", err);
static int mlx5i_create_flow_steering(struct mlx5e_priv *priv)
{
- struct mlx5i_priv *ipriv = priv->ppriv;
int err;
priv->fs.ns = mlx5_get_flow_namespace(priv->mdev,
priv->netdev->hw_features &= ~NETIF_F_NTUPLE;
}
- err = mlx5e_create_ttc_table(priv, ipriv->qp.qpn);
+ err = mlx5e_create_ttc_table(priv);
if (err) {
netdev_err(priv->netdev, "Failed to create ttc table, err=%d\n",
err);
struct mlx5_priv *priv = &mdev->priv;
struct msix_entry *msix = priv->msix_arr;
int irq = msix[i + MLX5_EQ_VEC_COMP_BASE].vector;
- int err;
if (!zalloc_cpumask_var(&priv->irq_info[i].mask, GFP_KERNEL)) {
mlx5_core_warn(mdev, "zalloc_cpumask_var failed");
cpumask_set_cpu(cpumask_local_spread(i, priv->numa_node),
priv->irq_info[i].mask);
- err = irq_set_affinity_hint(irq, priv->irq_info[i].mask);
- if (err) {
- mlx5_core_warn(mdev, "irq_set_affinity_hint failed,irq 0x%.4x",
- irq);
- goto err_clear_mask;
- }
+#ifdef CONFIG_SMP
+ if (irq_set_affinity_hint(irq, priv->irq_info[i].mask))
+ mlx5_core_warn(mdev, "irq_set_affinity_hint failed, irq 0x%.4x", irq);
+#endif
return 0;
-
-err_clear_mask:
- free_cpumask_var(priv->irq_info[i].mask);
- return err;
}
static void mlx5_irq_clear_affinity_hint(struct mlx5_core_dev *mdev, int i)
entry->counter_valid = false;
entry->counter = 0;
+ entry->index = mlxsw_sp_rif_index(rif);
+
if (!counters_enabled)
return 0;
- entry->index = mlxsw_sp_rif_index(rif);
err = mlxsw_sp_rif_counter_value_get(mlxsw_sp, rif,
MLXSW_SP_RIF_COUNTER_EGRESS,
&cnt);
{
unsigned int *p_counter_index;
+ if (!mlxsw_sp_rif_counter_valid_get(rif, dir))
+ return;
+
p_counter_index = mlxsw_sp_rif_p_counter_get(rif, dir);
if (WARN_ON(!p_counter_index))
return;
err = mlxsw_sp_port_fdb_uc_op(mlxsw_sp, local_port, mac, fid,
adding, true);
if (err) {
- if (net_ratelimit())
- netdev_err(mlxsw_sp_port->dev, "Failed to set FDB entry\n");
+ dev_err_ratelimited(mlxsw_sp->bus_info->dev, "Failed to set FDB entry\n");
return;
}
err = mlxsw_sp_port_fdb_uc_lag_op(mlxsw_sp, lag_id, mac, fid, lag_vid,
adding, true);
if (err) {
- if (net_ratelimit())
- netdev_err(mlxsw_sp_port->dev, "Failed to set FDB entry\n");
+ dev_err_ratelimited(mlxsw_sp->bus_info->dev, "Failed to set FDB entry\n");
return;
}
cmd.req.arg3 = 0;
if (recv_ctx->state == NX_HOST_CTX_STATE_ACTIVE)
- netxen_issue_cmd(adapter, &cmd);
+ rcode = netxen_issue_cmd(adapter, &cmd);
if (rcode != NX_RCODE_SUCCESS)
return -EIO;
memset(&camline, 0, sizeof(union gft_cam_line_union));
qed_wr(p_hwfn, p_ptt, PRS_REG_GFT_CAM + CAM_LINE_SIZE * pf_id,
camline.cam_line_mapped.camline);
- memset(&ramline, 0, sizeof(union gft_cam_line_union));
+ memset(&ramline, 0, sizeof(ramline));
for (i = 0; i < RAM_LINE_SIZE / REG_SIZE; i++) {
u32 hw_addr = PRS_REG_GFT_PROFILE_MASK_RAM;
#define _QLCNIC_LINUX_MAJOR 5
#define _QLCNIC_LINUX_MINOR 3
-#define _QLCNIC_LINUX_SUBVERSION 65
-#define QLCNIC_LINUX_VERSIONID "5.3.65"
+#define _QLCNIC_LINUX_SUBVERSION 66
+#define QLCNIC_LINUX_VERSIONID "5.3.66"
#define QLCNIC_DRV_IDC_VER 0x01
#define QLCNIC_DRIVER_VERSION ((_QLCNIC_LINUX_MAJOR << 16) |\
(_QLCNIC_LINUX_MINOR << 8) | (_QLCNIC_LINUX_SUBVERSION))
return 0;
}
+void qlcnic_83xx_get_port_type(struct qlcnic_adapter *adapter)
+{
+ struct qlcnic_hardware_context *ahw = adapter->ahw;
+ struct qlcnic_cmd_args cmd;
+ u32 config;
+ int err;
+
+ err = qlcnic_alloc_mbx_args(&cmd, adapter, QLCNIC_CMD_GET_LINK_STATUS);
+ if (err)
+ return;
+
+ err = qlcnic_issue_cmd(adapter, &cmd);
+ if (err) {
+ dev_info(&adapter->pdev->dev,
+ "Get Link Status Command failed: 0x%x\n", err);
+ goto out;
+ } else {
+ config = cmd.rsp.arg[3];
+
+ switch (QLC_83XX_SFP_MODULE_TYPE(config)) {
+ case QLC_83XX_MODULE_FIBRE_1000BASE_SX:
+ case QLC_83XX_MODULE_FIBRE_1000BASE_LX:
+ case QLC_83XX_MODULE_FIBRE_1000BASE_CX:
+ case QLC_83XX_MODULE_TP_1000BASE_T:
+ ahw->port_type = QLCNIC_GBE;
+ break;
+ default:
+ ahw->port_type = QLCNIC_XGBE;
+ }
+ }
+out:
+ qlcnic_free_mbx_args(&cmd);
+}
+
int qlcnic_83xx_test_link(struct qlcnic_adapter *adapter)
{
u8 pci_func;
int qlcnic_83xx_set_pauseparam(struct qlcnic_adapter *,
struct ethtool_pauseparam *);
int qlcnic_83xx_test_link(struct qlcnic_adapter *);
+void qlcnic_83xx_get_port_type(struct qlcnic_adapter *adapter);
int qlcnic_83xx_reg_test(struct qlcnic_adapter *);
int qlcnic_83xx_get_regs_len(struct qlcnic_adapter *);
int qlcnic_83xx_get_registers(struct qlcnic_adapter *, u32 *);
u32 ret = 0;
struct qlcnic_adapter *adapter = netdev_priv(dev);
+ if (qlcnic_83xx_check(adapter))
+ qlcnic_83xx_get_port_type(adapter);
+
if (adapter->ahw->port_type != QLCNIC_GBE)
return -EOPNOTSUPP;
/* Allocate rx SKB if we don't have one available. */
if (!qca->rx_skb) {
- qca->rx_skb = netdev_alloc_skb(net_dev,
- net_dev->mtu + VLAN_ETH_HLEN);
+ qca->rx_skb = netdev_alloc_skb_ip_align(net_dev,
+ net_dev->mtu +
+ VLAN_ETH_HLEN);
if (!qca->rx_skb) {
netdev_dbg(net_dev, "out of RX resources\n");
qca->stats.out_of_mem++;
qca->rx_skb, qca->rx_skb->dev);
qca->rx_skb->ip_summed = CHECKSUM_UNNECESSARY;
netif_rx_ni(qca->rx_skb);
- qca->rx_skb = netdev_alloc_skb(net_dev,
+ qca->rx_skb = netdev_alloc_skb_ip_align(net_dev,
net_dev->mtu + VLAN_ETH_HLEN);
if (!qca->rx_skb) {
netdev_dbg(net_dev, "out of RX resources\n");
if (!qca->rx_buffer)
return -ENOBUFS;
- qca->rx_skb = netdev_alloc_skb(dev, qca->net_dev->mtu + VLAN_ETH_HLEN);
+ qca->rx_skb = netdev_alloc_skb_ip_align(dev, qca->net_dev->mtu +
+ VLAN_ETH_HLEN);
if (!qca->rx_skb) {
kfree(qca->rx_buffer);
netdev_info(qca->net_dev, "Failed to allocate RX sk_buff.\n");
/* MDIO bus init */
ret = sh_mdio_init(mdp, pd);
if (ret) {
- dev_err(&ndev->dev, "failed to initialise MDIO\n");
+ if (ret != -EPROBE_DEFER)
+ dev_err(&pdev->dev, "MDIO init failed: %d\n", ret);
goto out_release;
}
#include "mcdi.h"
enum {
- EFX_REV_SIENA_A0 = 0,
- EFX_REV_HUNT_A0 = 1,
+ /* Revisions 0-2 were Falcon A0, A1 and B0 respectively.
+ * They are not supported by this driver but these revision numbers
+ * form part of the ethtool API for register dumping.
+ */
+ EFX_REV_SIENA_A0 = 3,
+ EFX_REV_HUNT_A0 = 4,
};
static inline int efx_nic_rev(struct efx_nic *efx)
ep++;
} else {
seq_printf(seq, "%d [0x%x]: 0x%x 0x%x 0x%x 0x%x\n",
- i, (unsigned int)virt_to_phys(ep),
+ i, (unsigned int)virt_to_phys(p),
le32_to_cpu(p->des0), le32_to_cpu(p->des1),
le32_to_cpu(p->des2), le32_to_cpu(p->des3));
p++;
if (port) {
del_timer_sync(&port->vio.timer);
+ del_timer_sync(&port->clean_timer);
napi_disable(&port->napi);
+ unregister_netdev(port->dev);
list_del_rcu(&port->list);
synchronize_rcu();
- del_timer_sync(&port->clean_timer);
spin_lock_irqsave(&port->vp->lock, flags);
sunvnet_port_rm_txq_common(port);
spin_unlock_irqrestore(&port->vp->lock, flags);
dev_set_drvdata(&vdev->dev, NULL);
- unregister_netdev(port->dev);
free_netdev(port->dev);
}
tx_pipe->dma_channel = knav_dma_open_channel(dev,
tx_pipe->dma_chan_name, &config);
- if (IS_ERR_OR_NULL(tx_pipe->dma_channel)) {
+ if (IS_ERR(tx_pipe->dma_channel)) {
dev_err(dev, "failed opening tx chan(%s)\n",
tx_pipe->dma_chan_name);
+ ret = PTR_ERR(tx_pipe->dma_channel);
goto err;
}
netcp->rx_channel = knav_dma_open_channel(netcp->netcp_device->device,
netcp->dma_chan_name, &config);
- if (IS_ERR_OR_NULL(netcp->rx_channel)) {
+ if (IS_ERR(netcp->rx_channel)) {
dev_err(netcp->ndev_dev, "failed opening rx chan(%s\n",
netcp->dma_chan_name);
+ ret = PTR_ERR(netcp->rx_channel);
goto fail;
}
case HWTSTAMP_FILTER_NONE:
cpts_rx_enable(cpts, 0);
break;
- case HWTSTAMP_FILTER_ALL:
case HWTSTAMP_FILTER_PTP_V1_L4_EVENT:
case HWTSTAMP_FILTER_PTP_V1_L4_SYNC:
case HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ:
if (nla_put_u32(skb, IFLA_GENEVE_ID, vni))
goto nla_put_failure;
- if (ip_tunnel_info_af(info) == AF_INET) {
+ if (rtnl_dereference(geneve->sock4)) {
if (nla_put_in_addr(skb, IFLA_GENEVE_REMOTE,
info->key.u.ipv4.dst))
goto nla_put_failure;
!!(info->key.tun_flags & TUNNEL_CSUM)))
goto nla_put_failure;
+ }
+
#if IS_ENABLED(CONFIG_IPV6)
- } else {
+ if (rtnl_dereference(geneve->sock6)) {
if (nla_put_in6_addr(skb, IFLA_GENEVE_REMOTE6,
&info->key.u.ipv6.dst))
goto nla_put_failure;
if (nla_put_u8(skb, IFLA_GENEVE_UDP_ZERO_CSUM6_RX,
!geneve->use_udp6_rx_checksums))
goto nla_put_failure;
-#endif
}
+#endif
if (nla_put_u8(skb, IFLA_GENEVE_TTL, info->key.ttl) ||
nla_put_u8(skb, IFLA_GENEVE_TOS, info->key.tos) ||
/* Check if there's an existing gtpX device to configure */
dev = dev_get_by_index_rcu(net, nla_get_u32(nla[GTPA_LINK]));
- if (dev->netdev_ops == >p_netdev_ops)
+ if (dev && dev->netdev_ops == >p_netdev_ops)
gtp = netdev_priv(dev);
put_net(net);
* are "42101001.sb" or "42101002.sb"
*/
sprintf(stir421x_fw_name, "4210%4X.sb",
- self->usbdev->descriptor.bcdDevice);
+ le16_to_cpu(self->usbdev->descriptor.bcdDevice));
ret = request_firmware(&fw, stir421x_fw_name, &self->usbdev->dev);
if (ret < 0)
return ret;
*/
static struct lock_class_key macvlan_netdev_addr_lock_key;
-#define ALWAYS_ON_FEATURES \
- (NETIF_F_SG | NETIF_F_HW_CSUM | NETIF_F_GSO_SOFTWARE | NETIF_F_LLTX | \
+#define ALWAYS_ON_OFFLOADS \
+ (NETIF_F_SG | NETIF_F_HW_CSUM | NETIF_F_GSO_SOFTWARE | \
NETIF_F_GSO_ROBUST)
+#define ALWAYS_ON_FEATURES (ALWAYS_ON_OFFLOADS | NETIF_F_LLTX)
+
#define MACVLAN_FEATURES \
(NETIF_F_SG | NETIF_F_HW_CSUM | NETIF_F_HIGHDMA | NETIF_F_FRAGLIST | \
NETIF_F_GSO | NETIF_F_TSO | NETIF_F_UFO | NETIF_F_LRO | \
dev->features |= ALWAYS_ON_FEATURES;
dev->hw_features |= NETIF_F_LRO;
dev->vlan_features = lowerdev->vlan_features & MACVLAN_FEATURES;
+ dev->vlan_features |= ALWAYS_ON_OFFLOADS;
dev->gso_max_size = lowerdev->gso_max_size;
dev->gso_max_segs = lowerdev->gso_max_segs;
dev->hard_header_len = lowerdev->hard_header_len;
config MDIO_OCTEON
tristate "Octeon and some ThunderX SOCs MDIO buses"
depends on 64BIT
- depends on HAS_IOMEM
+ depends on HAS_IOMEM && OF_MDIO
select MDIO_CAVIUM
help
This module provides a driver for the Octeon and ThunderX MDIO
{
int err;
- /* The Marvell PHY has an errata which requires
- * that certain registers get written in order
- * to restart autonegotiation */
- err = phy_write(phydev, MII_BMCR, BMCR_RESET);
-
- if (err < 0)
- return err;
-
- err = phy_write(phydev, 0x1d, 0x1f);
- if (err < 0)
- return err;
-
- err = phy_write(phydev, 0x1e, 0x200c);
- if (err < 0)
- return err;
-
- err = phy_write(phydev, 0x1d, 0x5);
- if (err < 0)
- return err;
-
- err = phy_write(phydev, 0x1e, 0);
- if (err < 0)
- return err;
-
- err = phy_write(phydev, 0x1e, 0x100);
- if (err < 0)
- return err;
-
err = marvell_set_polarity(phydev, phydev->mdix_ctrl);
if (err < 0)
return err;
return 0;
}
+static int m88e1101_config_aneg(struct phy_device *phydev)
+{
+ int err;
+
+ /* This Marvell PHY has an errata which requires
+ * that certain registers get written in order
+ * to restart autonegotiation
+ */
+ err = phy_write(phydev, MII_BMCR, BMCR_RESET);
+
+ if (err < 0)
+ return err;
+
+ err = phy_write(phydev, 0x1d, 0x1f);
+ if (err < 0)
+ return err;
+
+ err = phy_write(phydev, 0x1e, 0x200c);
+ if (err < 0)
+ return err;
+
+ err = phy_write(phydev, 0x1d, 0x5);
+ if (err < 0)
+ return err;
+
+ err = phy_write(phydev, 0x1e, 0);
+ if (err < 0)
+ return err;
+
+ err = phy_write(phydev, 0x1e, 0x100);
+ if (err < 0)
+ return err;
+
+ return marvell_config_aneg(phydev);
+}
+
static int m88e1111_config_aneg(struct phy_device *phydev)
{
int err;
.flags = PHY_HAS_INTERRUPT,
.probe = marvell_probe,
.config_init = &marvell_config_init,
- .config_aneg = &marvell_config_aneg,
+ .config_aneg = &m88e1101_config_aneg,
.read_status = &genphy_read_status,
.ack_interrupt = &marvell_ack_interrupt,
.config_intr = &marvell_config_intr,
pb = devm_kzalloc(dev, sizeof(*pb), GFP_KERNEL);
if (pb == NULL) {
ret_val = -ENOMEM;
- goto err_parent_bus;
+ goto err_pb_kz;
}
-
pb->switch_data = data;
pb->switch_fn = switch_fn;
pb->current_child = -1;
cb->mii_bus = mdiobus_alloc();
if (!cb->mii_bus) {
ret_val = -ENOMEM;
+ devm_kfree(dev, cb);
of_node_put(child_bus_node);
break;
}
mdiobus_free(cb->mii_bus);
devm_kfree(dev, cb);
} else {
- of_node_get(child_bus_node);
cb->next = pb->children;
pb->children = cb;
}
return 0;
}
+ devm_kfree(dev, pb);
+err_pb_kz:
/* balance the reference of_mdio_find_bus() took */
- put_device(&pb->mii_bus->dev);
-
+ if (!mux_bus)
+ put_device(&parent_bus->dev);
err_parent_bus:
of_node_put(parent_bus_node);
return ret_val;
mutex_init(&bus->mdio_lock);
- if (bus->reset)
- bus->reset(bus);
-
/* de-assert bus level PHY GPIO resets */
if (bus->num_reset_gpios > 0) {
bus->reset_gpiod = devm_kcalloc(&bus->dev,
}
}
+ if (bus->reset)
+ bus->reset(bus);
+
for (i = 0; i < PHY_MAX_ADDR; i++) {
if ((bus->phy_mask & (1 << i)) == 0) {
struct phy_device *phydev;
return -ENODEV;
}
+ return 0;
+
+bad_desc:
+ dev_info(&dev->udev->dev, "bad CDC descriptors\n");
+ return -ENODEV;
+}
+EXPORT_SYMBOL_GPL(usbnet_generic_cdc_bind);
+
+
+/* like usbnet_generic_cdc_bind() but handles filter initialization
+ * correctly
+ */
+int usbnet_ether_cdc_bind(struct usbnet *dev, struct usb_interface *intf)
+{
+ int rv;
+
+ rv = usbnet_generic_cdc_bind(dev, intf);
+ if (rv < 0)
+ goto bail_out;
+
/* Some devices don't initialise properly. In particular
* the packet filter is not reset. There are devices that
* don't do reset all the way. So the packet filter should
*/
usbnet_cdc_update_filter(dev);
- return 0;
-
-bad_desc:
- dev_info(&dev->udev->dev, "bad CDC descriptors\n");
- return -ENODEV;
+bail_out:
+ return rv;
}
-EXPORT_SYMBOL_GPL(usbnet_generic_cdc_bind);
+EXPORT_SYMBOL_GPL(usbnet_ether_cdc_bind);
void usbnet_cdc_unbind(struct usbnet *dev, struct usb_interface *intf)
{
BUILD_BUG_ON((sizeof(((struct usbnet *)0)->data)
< sizeof(struct cdc_state)));
- status = usbnet_generic_cdc_bind(dev, intf);
+ status = usbnet_ether_cdc_bind(dev, intf);
if (status < 0)
return status;
int rd_mac_len = 0;
netdev_dbg(dev->net, "get_mac_address:\n\tusbnet VID:%0x PID:%0x\n",
- dev->udev->descriptor.idVendor,
- dev->udev->descriptor.idProduct);
+ le16_to_cpu(dev->udev->descriptor.idVendor),
+ le16_to_cpu(dev->udev->descriptor.idProduct));
memset(mac_addr, 0, sizeof(mac_addr));
rd_mac_len = control_read(dev, REQUEST_READ, 0,
{QMI_FIXED_INTF(0x1199, 0x9071, 10)}, /* Sierra Wireless MC74xx */
{QMI_FIXED_INTF(0x1199, 0x9079, 8)}, /* Sierra Wireless EM74xx */
{QMI_FIXED_INTF(0x1199, 0x9079, 10)}, /* Sierra Wireless EM74xx */
+ {QMI_FIXED_INTF(0x1199, 0x907b, 8)}, /* Sierra Wireless EM74xx */
+ {QMI_FIXED_INTF(0x1199, 0x907b, 10)}, /* Sierra Wireless EM74xx */
{QMI_FIXED_INTF(0x1bbb, 0x011e, 4)}, /* Telekom Speedstick LTE II (Alcatel One Touch L100V LTE) */
{QMI_FIXED_INTF(0x1bbb, 0x0203, 2)}, /* Alcatel L800MA */
{QMI_FIXED_INTF(0x2357, 0x0201, 4)}, /* TP-LINK HSUPA Modem MA180 */
if (ret < 0)
return ret;
- if (features & NETIF_F_HW_CSUM)
+ if (features & NETIF_F_IP_CSUM)
read_buf |= Tx_COE_EN_;
else
read_buf &= ~Tx_COE_EN_;
spin_lock_init(&pdata->mac_cr_lock);
+ /* LAN95xx devices do not alter the computed checksum of 0 to 0xffff.
+ * RFC 2460, ipv6 UDP calculated checksum yields a result of zero must
+ * be changed to 0xffff. RFC 768, ipv4 UDP computed checksum is zero,
+ * it is transmitted as all ones. The zero transmitted checksum means
+ * transmitter generated no checksum. Hence, enable csum offload only
+ * for ipv4 packets.
+ */
if (DEFAULT_TX_CSUM_ENABLE)
- dev->net->features |= NETIF_F_HW_CSUM;
+ dev->net->features |= NETIF_F_IP_CSUM;
if (DEFAULT_RX_CSUM_ENABLE)
dev->net->features |= NETIF_F_RXCSUM;
- dev->net->hw_features = NETIF_F_HW_CSUM | NETIF_F_RXCSUM;
+ dev->net->hw_features = NETIF_F_IP_CSUM | NETIF_F_RXCSUM;
smsc95xx_init_mac_address(dev);
.ndo_poll_controller = virtnet_netpoll,
#endif
.ndo_xdp = virtnet_xdp,
+ .ndo_features_check = passthru_features_check,
};
static void virtnet_config_changed_work(struct work_struct *work)
/* we need to enable NAPI, otherwise dev_close will deadlock */
for (i = 0; i < adapter->num_rx_queues; i++)
napi_enable(&adapter->rx_queue[i].napi);
+ /*
+ * Need to clear the quiesce bit to ensure that vmxnet3_close
+ * can quiesce the device properly
+ */
+ clear_bit(VMXNET3_STATE_BIT_QUIESCED, &adapter->state);
dev_close(adapter->netdev);
}
static int vrf_rcv_finish(struct net *net, struct sock *sk, struct sk_buff *skb)
{
+ kfree_skb(skb);
return 0;
}
{
struct net *net = dev_net(dev);
- if (NF_HOOK(pf, hook, net, NULL, skb, dev, NULL, vrf_rcv_finish) < 0)
+ if (nf_hook(pf, hook, net, NULL, skb, dev, NULL, vrf_rcv_finish) != 1)
skb = NULL; /* kfree_skb(skb) handled by nf code */
return skb;
xennet_disconnect_backend(info);
xennet_destroy_queues(info);
out:
- unregister_netdev(info->netdev);
- xennet_free_netdev(info->netdev);
+ device_unregister(&dev->dev);
return err;
}
}
#ifdef CONFIG_BLK_DEV_INTEGRITY
+static void nvme_prep_integrity(struct gendisk *disk, struct nvme_id_ns *id,
+ u16 bs)
+{
+ struct nvme_ns *ns = disk->private_data;
+ u16 old_ms = ns->ms;
+ u8 pi_type = 0;
+
+ ns->ms = le16_to_cpu(id->lbaf[id->flbas & NVME_NS_FLBAS_LBA_MASK].ms);
+ ns->ext = ns->ms && (id->flbas & NVME_NS_FLBAS_META_EXT);
+
+ /* PI implementation requires metadata equal t10 pi tuple size */
+ if (ns->ms == sizeof(struct t10_pi_tuple))
+ pi_type = id->dps & NVME_NS_DPS_PI_MASK;
+
+ if (blk_get_integrity(disk) &&
+ (ns->pi_type != pi_type || ns->ms != old_ms ||
+ bs != queue_logical_block_size(disk->queue) ||
+ (ns->ms && ns->ext)))
+ blk_integrity_unregister(disk);
+
+ ns->pi_type = pi_type;
+}
+
static void nvme_init_integrity(struct nvme_ns *ns)
{
struct blk_integrity integrity;
blk_queue_max_integrity_segments(ns->queue, 1);
}
#else
+static void nvme_prep_integrity(struct gendisk *disk, struct nvme_id_ns *id,
+ u16 bs)
+{
+}
static void nvme_init_integrity(struct nvme_ns *ns)
{
}
static void __nvme_revalidate_disk(struct gendisk *disk, struct nvme_id_ns *id)
{
struct nvme_ns *ns = disk->private_data;
- u8 lbaf, pi_type;
- u16 old_ms;
- unsigned short bs;
-
- old_ms = ns->ms;
- lbaf = id->flbas & NVME_NS_FLBAS_LBA_MASK;
- ns->lba_shift = id->lbaf[lbaf].ds;
- ns->ms = le16_to_cpu(id->lbaf[lbaf].ms);
- ns->ext = ns->ms && (id->flbas & NVME_NS_FLBAS_META_EXT);
+ u16 bs;
/*
* If identify namespace failed, use default 512 byte block size so
* block layer can use before failing read/write for 0 capacity.
*/
+ ns->lba_shift = id->lbaf[id->flbas & NVME_NS_FLBAS_LBA_MASK].ds;
if (ns->lba_shift == 0)
ns->lba_shift = 9;
bs = 1 << ns->lba_shift;
- /* XXX: PI implementation requires metadata equal t10 pi tuple size */
- pi_type = ns->ms == sizeof(struct t10_pi_tuple) ?
- id->dps & NVME_NS_DPS_PI_MASK : 0;
blk_mq_freeze_queue(disk->queue);
- if (blk_get_integrity(disk) && (ns->pi_type != pi_type ||
- ns->ms != old_ms ||
- bs != queue_logical_block_size(disk->queue) ||
- (ns->ms && ns->ext)))
- blk_integrity_unregister(disk);
- ns->pi_type = pi_type;
+ if (ns->ctrl->ops->flags & NVME_F_METADATA_SUPPORTED)
+ nvme_prep_integrity(disk, id, bs);
blk_queue_logical_block_size(ns->queue, bs);
-
if (ns->ms && !blk_get_integrity(disk) && !ns->ext)
nvme_init_integrity(ns);
if (ns->ms && !(ns->ms == 8 && ns->pi_type) && !blk_get_integrity(disk))
}
memcpy(ctrl->psd, id->psd, sizeof(ctrl->psd));
- if (ctrl->ops->is_fabrics) {
+ if (ctrl->ops->flags & NVME_F_FABRICS) {
ctrl->icdoff = le16_to_cpu(id->icdoff);
ctrl->ioccsz = le32_to_cpu(id->ioccsz);
ctrl->iorcsz = le32_to_cpu(id->iorcsz);
if (ns->ndev)
nvme_nvm_unregister_sysfs(ns);
del_gendisk(ns->disk);
- blk_mq_abort_requeue_list(ns->queue);
blk_cleanup_queue(ns->queue);
}
continue;
revalidate_disk(ns->disk);
blk_set_queue_dying(ns->queue);
- blk_mq_abort_requeue_list(ns->queue);
- blk_mq_start_stopped_hw_queues(ns->queue, true);
+
+ /*
+ * Forcibly start all queues to avoid having stuck requests.
+ * Note that we must ensure the queues are not stopped
+ * when the final removal happens.
+ */
+ blk_mq_start_hw_queues(ns->queue);
+
+ /* draining requests in requeue list */
+ blk_mq_kick_requeue_list(ns->queue);
}
mutex_unlock(&ctrl->namespaces_mutex);
}
#define NVMEFC_QUEUE_DELAY 3 /* ms units */
-#define NVME_FC_MAX_CONNECT_ATTEMPTS 1
-
struct nvme_fc_queue {
struct nvme_fc_ctrl *ctrl;
struct device *dev;
struct work_struct delete_work;
struct work_struct reset_work;
struct delayed_work connect_work;
- int reconnect_delay;
- int connect_attempts;
struct kref ref;
u32 flags;
complete_rq = __nvme_fc_fcpop_chk_teardowns(ctrl, op);
if (!complete_rq) {
if (unlikely(op->flags & FCOP_FLAGS_TERMIO)) {
- status = cpu_to_le16(NVME_SC_ABORT_REQ);
+ status = cpu_to_le16(NVME_SC_ABORT_REQ << 1);
if (blk_queue_dying(rq->q))
- status |= cpu_to_le16(NVME_SC_DNR);
+ status |= cpu_to_le16(NVME_SC_DNR << 1);
}
nvme_end_request(rq, status, result);
} else
dev_warn(ctrl->ctrl.device,
"NVME-FC{%d}: transport association error detected: %s\n",
ctrl->cnum, errmsg);
- dev_info(ctrl->ctrl.device,
+ dev_warn(ctrl->ctrl.device,
"NVME-FC{%d}: resetting controller\n", ctrl->cnum);
+ /* stop the queues on error, cleanup is in reset thread */
+ if (ctrl->queue_count > 1)
+ nvme_stop_queues(&ctrl->ctrl);
+
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RECONNECTING)) {
dev_err(ctrl->ctrl.device,
"NVME-FC{%d}: error_recovery: Couldn't change state "
if (!opts->nr_io_queues)
return 0;
- dev_info(ctrl->ctrl.device, "creating %d I/O queues.\n",
- opts->nr_io_queues);
-
nvme_fc_init_io_queues(ctrl);
memset(&ctrl->tag_set, 0, sizeof(ctrl->tag_set));
if (ctrl->queue_count == 1)
return 0;
- dev_info(ctrl->ctrl.device, "Recreating %d I/O queues.\n",
- opts->nr_io_queues);
-
nvme_fc_init_io_queues(ctrl);
ret = blk_mq_reinit_tagset(&ctrl->tag_set);
int ret;
bool changed;
- ctrl->connect_attempts++;
+ ++ctrl->ctrl.opts->nr_reconnects;
/*
* Create the admin queue
changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE);
WARN_ON_ONCE(!changed);
- ctrl->connect_attempts = 0;
-
- kref_get(&ctrl->ctrl.kref);
+ ctrl->ctrl.opts->nr_reconnects = 0;
if (ctrl->queue_count > 1) {
nvme_start_queues(&ctrl->ctrl);
/*
* tear down the controller
- * This will result in the last reference on the nvme ctrl to
- * expire, calling the transport nvme_fc_nvme_ctrl_freed() callback.
- * From there, the transport will tear down it's logical queues and
- * association.
+ * After the last reference on the nvme ctrl is removed,
+ * the transport nvme_fc_nvme_ctrl_freed() callback will be
+ * invoked. From there, the transport will tear down it's
+ * logical queues and association.
*/
nvme_uninit_ctrl(&ctrl->ctrl);
nvme_put_ctrl(&ctrl->ctrl);
}
-static int
-__nvme_fc_del_ctrl(struct nvme_fc_ctrl *ctrl)
+static bool
+__nvme_fc_schedule_delete_work(struct nvme_fc_ctrl *ctrl)
{
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_DELETING))
- return -EBUSY;
+ return true;
if (!queue_work(nvme_fc_wq, &ctrl->delete_work))
- return -EBUSY;
+ return true;
- return 0;
+ return false;
+}
+
+static int
+__nvme_fc_del_ctrl(struct nvme_fc_ctrl *ctrl)
+{
+ return __nvme_fc_schedule_delete_work(ctrl) ? -EBUSY : 0;
}
/*
return ret;
}
+static void
+nvme_fc_reconnect_or_delete(struct nvme_fc_ctrl *ctrl, int status)
+{
+ /* If we are resetting/deleting then do nothing */
+ if (ctrl->ctrl.state != NVME_CTRL_RECONNECTING) {
+ WARN_ON_ONCE(ctrl->ctrl.state == NVME_CTRL_NEW ||
+ ctrl->ctrl.state == NVME_CTRL_LIVE);
+ return;
+ }
+
+ dev_info(ctrl->ctrl.device,
+ "NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
+ ctrl->cnum, status);
+
+ if (nvmf_should_reconnect(&ctrl->ctrl)) {
+ dev_info(ctrl->ctrl.device,
+ "NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
+ ctrl->cnum, ctrl->ctrl.opts->reconnect_delay);
+ queue_delayed_work(nvme_fc_wq, &ctrl->connect_work,
+ ctrl->ctrl.opts->reconnect_delay * HZ);
+ } else {
+ dev_warn(ctrl->ctrl.device,
+ "NVME-FC{%d}: Max reconnect attempts (%d) "
+ "reached. Removing controller\n",
+ ctrl->cnum, ctrl->ctrl.opts->nr_reconnects);
+ WARN_ON(__nvme_fc_schedule_delete_work(ctrl));
+ }
+}
+
static void
nvme_fc_reset_ctrl_work(struct work_struct *work)
{
nvme_fc_delete_association(ctrl);
ret = nvme_fc_create_association(ctrl);
- if (ret) {
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: reset: Reconnect attempt failed (%d)\n",
- ctrl->cnum, ret);
- if (ctrl->connect_attempts >= NVME_FC_MAX_CONNECT_ATTEMPTS) {
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: Max reconnect attempts (%d) "
- "reached. Removing controller\n",
- ctrl->cnum, ctrl->connect_attempts);
-
- if (!nvme_change_ctrl_state(&ctrl->ctrl,
- NVME_CTRL_DELETING)) {
- dev_err(ctrl->ctrl.device,
- "NVME-FC{%d}: failed to change state "
- "to DELETING\n", ctrl->cnum);
- return;
- }
-
- WARN_ON(!queue_work(nvme_fc_wq, &ctrl->delete_work));
- return;
- }
-
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
- ctrl->cnum, ctrl->reconnect_delay);
- queue_delayed_work(nvme_fc_wq, &ctrl->connect_work,
- ctrl->reconnect_delay * HZ);
- } else
+ if (ret)
+ nvme_fc_reconnect_or_delete(ctrl, ret);
+ else
dev_info(ctrl->ctrl.device,
"NVME-FC{%d}: controller reset complete\n", ctrl->cnum);
}
{
struct nvme_fc_ctrl *ctrl = to_fc_ctrl(nctrl);
- dev_warn(ctrl->ctrl.device,
+ dev_info(ctrl->ctrl.device,
"NVME-FC{%d}: admin requested controller reset\n", ctrl->cnum);
if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING))
static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = {
.name = "fc",
.module = THIS_MODULE,
- .is_fabrics = true,
+ .flags = NVME_F_FABRICS,
.reg_read32 = nvmf_reg_read32,
.reg_read64 = nvmf_reg_read64,
.reg_write32 = nvmf_reg_write32,
struct nvme_fc_ctrl, connect_work);
ret = nvme_fc_create_association(ctrl);
- if (ret) {
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: Reconnect attempt failed (%d)\n",
- ctrl->cnum, ret);
- if (ctrl->connect_attempts >= NVME_FC_MAX_CONNECT_ATTEMPTS) {
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: Max reconnect attempts (%d) "
- "reached. Removing controller\n",
- ctrl->cnum, ctrl->connect_attempts);
-
- if (!nvme_change_ctrl_state(&ctrl->ctrl,
- NVME_CTRL_DELETING)) {
- dev_err(ctrl->ctrl.device,
- "NVME-FC{%d}: failed to change state "
- "to DELETING\n", ctrl->cnum);
- return;
- }
-
- WARN_ON(!queue_work(nvme_fc_wq, &ctrl->delete_work));
- return;
- }
-
- dev_warn(ctrl->ctrl.device,
- "NVME-FC{%d}: Reconnect attempt in %d seconds.\n",
- ctrl->cnum, ctrl->reconnect_delay);
- queue_delayed_work(nvme_fc_wq, &ctrl->connect_work,
- ctrl->reconnect_delay * HZ);
- } else
+ if (ret)
+ nvme_fc_reconnect_or_delete(ctrl, ret);
+ else
dev_info(ctrl->ctrl.device,
"NVME-FC{%d}: controller reconnect complete\n",
ctrl->cnum);
unsigned long flags;
int ret, idx;
+ if (!(rport->remoteport.port_role &
+ (FC_PORT_ROLE_NVME_DISCOVERY | FC_PORT_ROLE_NVME_TARGET))) {
+ ret = -EBADR;
+ goto out_fail;
+ }
+
ctrl = kzalloc(sizeof(*ctrl), GFP_KERNEL);
if (!ctrl) {
ret = -ENOMEM;
INIT_WORK(&ctrl->delete_work, nvme_fc_delete_ctrl_work);
INIT_WORK(&ctrl->reset_work, nvme_fc_reset_ctrl_work);
INIT_DELAYED_WORK(&ctrl->connect_work, nvme_fc_connect_ctrl_work);
- ctrl->reconnect_delay = opts->reconnect_delay;
spin_lock_init(&ctrl->lock);
/* io queue count */
ctrl->ctrl.opts = NULL;
/* initiate nvme ctrl ref counting teardown */
nvme_uninit_ctrl(&ctrl->ctrl);
- nvme_put_ctrl(&ctrl->ctrl);
/* as we're past the point where we transition to the ref
* counting teardown path, if we return a bad pointer here,
return ERR_PTR(ret);
}
+ kref_get(&ctrl->ctrl.kref);
+
dev_info(ctrl->ctrl.device,
"NVME-FC{%d}: new ctrl: NQN \"%s\"\n",
ctrl->cnum, ctrl->ctrl.opts->subsysnqn);
static struct nvmf_transport_ops nvme_fc_transport = {
.name = "fc",
.required_opts = NVMF_OPT_TRADDR | NVMF_OPT_HOST_TRADDR,
- .allowed_opts = NVMF_OPT_RECONNECT_DELAY,
+ .allowed_opts = NVMF_OPT_RECONNECT_DELAY | NVMF_OPT_CTRL_LOSS_TMO,
.create_ctrl = nvme_fc_create_ctrl,
};
struct nvme_ctrl_ops {
const char *name;
struct module *module;
- bool is_fabrics;
+ unsigned int flags;
+#define NVME_F_FABRICS (1 << 0)
+#define NVME_F_METADATA_SUPPORTED (1 << 1)
int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val);
int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val);
int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val);
c.dbbuf.prp2 = cpu_to_le64(dev->dbbuf_eis_dma_addr);
if (nvme_submit_sync_cmd(dev->ctrl.admin_q, &c, NULL, 0)) {
- dev_warn(dev->dev, "unable to set dbbuf\n");
+ dev_warn(dev->ctrl.device, "unable to set dbbuf\n");
/* Free memory and continue on */
nvme_dbbuf_dma_free(dev);
}
result = pci_read_config_word(to_pci_dev(dev->dev), PCI_STATUS,
&pci_status);
if (result == PCIBIOS_SUCCESSFUL)
- dev_warn(dev->dev,
+ dev_warn(dev->ctrl.device,
"controller is down; will reset: CSTS=0x%x, PCI_STATUS=0x%hx\n",
csts, pci_status);
else
- dev_warn(dev->dev,
+ dev_warn(dev->ctrl.device,
"controller is down; will reset: CSTS=0x%x, PCI_STATUS read failed (%d)\n",
csts, result);
}
if (dev->cmb) {
iounmap(dev->cmb);
dev->cmb = NULL;
+ if (dev->cmbsz) {
+ sysfs_remove_file_from_group(&dev->ctrl.device->kobj,
+ &dev_attr_cmb.attr, NULL);
+ dev->cmbsz = 0;
+ }
}
}
*/
if (pdev->vendor == PCI_VENDOR_ID_APPLE && pdev->device == 0x2001) {
dev->q_depth = 2;
- dev_warn(dev->dev, "detected Apple NVMe controller, set "
- "queue depth=%u to work around controller resets\n",
+ dev_warn(dev->ctrl.device, "detected Apple NVMe controller, "
+ "set queue depth=%u to work around controller resets\n",
dev->q_depth);
}
if (dev->cmbsz) {
if (sysfs_add_file_to_group(&dev->ctrl.device->kobj,
&dev_attr_cmb.attr, NULL))
- dev_warn(dev->dev,
+ dev_warn(dev->ctrl.device,
"failed to add sysfs attribute for CMB\n");
}
}
{
struct pci_dev *pdev = to_pci_dev(dev->dev);
+ nvme_release_cmb(dev);
pci_free_irq_vectors(pdev);
if (pci_is_enabled(pdev)) {
static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = {
.name = "pcie",
.module = THIS_MODULE,
+ .flags = NVME_F_METADATA_SUPPORTED,
.reg_read32 = nvme_pci_reg_read32,
.reg_write32 = nvme_pci_reg_write32,
.reg_read64 = nvme_pci_reg_read64,
nvme_dev_disable(dev, true);
nvme_dev_remove_admin(dev);
nvme_free_queues(dev, 0);
- nvme_release_cmb(dev);
nvme_release_prp_pools(dev);
nvme_dev_unmap(dev);
nvme_put_ctrl(&dev->ctrl);
{ PCI_VDEVICE(INTEL, 0x0a54),
.driver_data = NVME_QUIRK_STRIPE_SIZE |
NVME_QUIRK_DEALLOCATE_ZEROES, },
+ { PCI_VDEVICE(INTEL, 0xf1a5), /* Intel 600P/P3100 */
+ .driver_data = NVME_QUIRK_NO_DEEPEST_PS },
{ PCI_VDEVICE(INTEL, 0x5845), /* Qemu emulated controller */
.driver_data = NVME_QUIRK_IDENTIFY_CNS, },
{ PCI_DEVICE(0x1c58, 0x0003), /* HGST adapter */
nvme_rdma_wr_error(cq, wc, "SEND");
}
+static inline int nvme_rdma_queue_sig_limit(struct nvme_rdma_queue *queue)
+{
+ int sig_limit;
+
+ /*
+ * We signal completion every queue depth/2 and also handle the
+ * degenerated case of a device with queue_depth=1, where we
+ * would need to signal every message.
+ */
+ sig_limit = max(queue->queue_size / 2, 1);
+ return (++queue->sig_count % sig_limit) == 0;
+}
+
static int nvme_rdma_post_send(struct nvme_rdma_queue *queue,
struct nvme_rdma_qe *qe, struct ib_sge *sge, u32 num_sge,
struct ib_send_wr *first, bool flush)
* Would have been way to obvious to handle this in hardware or
* at least the RDMA stack..
*
- * This messy and racy code sniplet is copy and pasted from the iSER
- * initiator, and the magic '32' comes from there as well.
- *
* Always signal the flushes. The magic request used for the flush
* sequencer is not allocated in our driver's tagset and it's
* triggered to be freed by blk_cleanup_queue(). So we need to
* embedded in request's payload, is not freed when __ib_process_cq()
* calls wr_cqe->done().
*/
- if ((++queue->sig_count % 32) == 0 || flush)
+ if (nvme_rdma_queue_sig_limit(queue) || flush)
wr.send_flags |= IB_SEND_SIGNALED;
if (first)
static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = {
.name = "rdma",
.module = THIS_MODULE,
- .is_fabrics = true,
+ .flags = NVME_F_FABRICS,
.reg_read32 = nvmf_reg_read32,
.reg_read64 = nvmf_reg_read64,
.reg_write32 = nvmf_reg_write32,
}
EXPORT_SYMBOL_GPL(nvmet_req_init);
+void nvmet_req_uninit(struct nvmet_req *req)
+{
+ percpu_ref_put(&req->sq->ref);
+}
+EXPORT_SYMBOL_GPL(nvmet_req_uninit);
+
static inline bool nvmet_cc_en(u32 cc)
{
return cc & 0x1;
{
int cpu, idx, cnt;
- if (!(tgtport->ops->target_features &
- NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED) ||
- tgtport->ops->max_hw_queues == 1)
+ if (tgtport->ops->max_hw_queues == 1)
return WORK_CPU_UNBOUND;
/* Simple cpu selection based on qid modulo active cpu count */
.dma_boundary = FCLOOP_DMABOUND_4G,
/* optional features */
.target_features = NVMET_FCTGTFEAT_CMD_IN_ISR |
- NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED |
NVMET_FCTGTFEAT_OPDONE_IN_ISR,
/* sizes of additional private data for data structures */
.target_priv_sz = sizeof(struct fcloop_tport),
static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
.name = "loop",
.module = THIS_MODULE,
- .is_fabrics = true,
+ .flags = NVME_F_FABRICS,
.reg_read32 = nvmf_reg_read32,
.reg_read64 = nvmf_reg_read64,
.reg_write32 = nvmf_reg_write32,
bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
struct nvmet_sq *sq, struct nvmet_fabrics_ops *ops);
+void nvmet_req_uninit(struct nvmet_req *req);
void nvmet_req_complete(struct nvmet_req *req, u16 status);
void nvmet_cq_setup(struct nvmet_ctrl *ctrl, struct nvmet_cq *cq, u16 qid,
rsp->n_rdma = 0;
if (unlikely(wc->status != IB_WC_SUCCESS)) {
+ nvmet_req_uninit(&rsp->req);
nvmet_rdma_release_rsp(rsp);
if (wc->status != IB_WC_WR_FLUSH_ERR) {
pr_info("RDMA READ for CQE 0x%p failed with status %s (%d).\n",
/* Allocate memory for the expanded device tree */
mem = dt_alloc(size + 4, __alignof__(struct device_node));
+ if (!mem)
+ return NULL;
+
memset(mem, 0, size);
*(__be32 *)(mem + size) = cpu_to_be32(0xdeadbeef);
const struct of_device_id *i;
for (i = __reservedmem_of_table; i < &__rmem_of_table_sentinel; i++) {
- int const (*initfn)(struct reserved_mem *rmem) = i->data;
+ reservedmem_of_init_fn initfn = i->data;
const char *compat = i->compatible;
if (!of_flat_dt_is_compatible(rmem->fdt_node, compat))
arch_initcall_sync(of_platform_default_populate_init);
#endif
-static int of_platform_device_destroy(struct device *dev, void *data)
+int of_platform_device_destroy(struct device *dev, void *data)
{
/* Do not touch devices not populated from the device tree */
if (!dev->of_node || !of_node_check_flag(dev->of_node, OF_POPULATED))
of_node_clear_flag(dev->of_node, OF_POPULATED_BUS);
return 0;
}
+EXPORT_SYMBOL_GPL(of_platform_device_destroy);
/**
* of_platform_depopulate() - Remove devices populated from device tree
static int imx6q_pcie_abort_handler(unsigned long addr,
unsigned int fsr, struct pt_regs *regs)
{
- return 0;
+ unsigned long pc = instruction_pointer(regs);
+ unsigned long instr = *(unsigned long *)pc;
+ int reg = (instr >> 12) & 15;
+
+ /*
+ * If the instruction being executed was a read,
+ * make it look like it read all-ones.
+ */
+ if ((instr & 0x0c100000) == 0x04100000) {
+ unsigned long val;
+
+ if (instr & 0x00400000)
+ val = 255;
+ else
+ val = -1;
+
+ regs->uregs[reg] = val;
+ regs->ARM_pc += 4;
+ return 0;
+ }
+
+ if ((instr & 0x0e100090) == 0x00100090) {
+ regs->uregs[reg] = -1;
+ regs->ARM_pc += 4;
+ return 0;
+ }
+
+ return 1;
}
static void imx6_pcie_assert_core_reset(struct imx6_pcie *imx6_pcie)
* we can install the handler here without risking it
* accessing some uninitialized driver state.
*/
- hook_fault_code(16 + 6, imx6q_pcie_abort_handler, SIGBUS, 0,
- "imprecise external abort");
+ hook_fault_code(8, imx6q_pcie_abort_handler, SIGBUS, 0,
+ "external abort on non-linefetch");
return platform_driver_register(&imx6_pcie_driver);
}
config PCI_ENDPOINT
bool "PCI Endpoint Support"
+ depends on HAS_DMA
help
Enable this configuration option to support configurable PCI
endpoint. This should be enabled if the platform has a PCI
if (!pm_runtime_suspended(dev)
|| pci_target_state(pci_dev) != pci_dev->current_state
- || platform_pci_need_resume(pci_dev))
+ || platform_pci_need_resume(pci_dev)
+ || (pci_dev->dev_flags & PCI_DEV_FLAGS_NEEDS_RESUME))
return false;
/*
cdev = &stdev->cdev;
cdev_init(cdev, &switchtec_fops);
cdev->owner = THIS_MODULE;
- cdev->kobj.parent = &dev->kobj;
return stdev;
stdev->mmio_sys_info = stdev->mmio + SWITCHTEC_GAS_SYS_INFO_OFFSET;
stdev->mmio_flash_info = stdev->mmio + SWITCHTEC_GAS_FLASH_INFO_OFFSET;
stdev->mmio_ntb = stdev->mmio + SWITCHTEC_GAS_NTB_OFFSET;
- stdev->partition = ioread8(&stdev->mmio_ntb->partition_id);
+ stdev->partition = ioread8(&stdev->mmio_sys_info->partition_id);
stdev->partition_count = ioread8(&stdev->mmio_ntb->partition_count);
stdev->mmio_part_cfg_all = stdev->mmio + SWITCHTEC_GAS_PART_CFG_OFFSET;
stdev->mmio_part_cfg = &stdev->mmio_part_cfg_all[stdev->partition];
stdev->mmio_pff_csr = stdev->mmio + SWITCHTEC_GAS_PFF_CSR_OFFSET;
+ if (stdev->partition_count < 1)
+ stdev->partition_count = 1;
+
init_pff(stdev);
pci_set_drvdata(pdev, stdev);
SWITCHTEC_EVENT_EN_IRQ,
&stdev->mmio_part_cfg->mrpc_comp_hdr);
- rc = cdev_add(&stdev->cdev, stdev->dev.devt, 1);
- if (rc)
- goto err_put;
-
- rc = device_add(&stdev->dev);
+ rc = cdev_device_add(&stdev->cdev, &stdev->dev);
if (rc)
goto err_devadd;
return 0;
err_devadd:
- cdev_del(&stdev->cdev);
stdev_kill(stdev);
err_put:
ida_simple_remove(&switchtec_minor_ida, MINOR(stdev->dev.devt));
pci_set_drvdata(pdev, NULL);
- device_del(&stdev->dev);
- cdev_del(&stdev->cdev);
+ cdev_device_del(&stdev->cdev, &stdev->dev);
ida_simple_remove(&switchtec_minor_ida, MINOR(stdev->dev.devt));
dev_info(&stdev->dev, "unregistered.\n");
power_zone->id = result;
idr_init(&power_zone->idr);
+ result = -ENOMEM;
power_zone->name = kstrdup(name, GFP_KERNEL);
if (!power_zone->name)
goto err_name_alloc;
}
spin_unlock_irqrestore(&rtc_lock, flags);
- pm_wakeup_event(dev, 0);
+ pm_wakeup_hard_event(dev);
acpi_clear_event(ACPI_EVENT_RTC);
acpi_disable_event(ACPI_EVENT_RTC, 0);
return ACPI_INTERRUPT_HANDLED;
static void __ccwgroup_remove_symlinks(struct ccwgroup_device *gdev)
{
int i;
- char str[8];
+ char str[16];
for (i = 0; i < gdev->count; i++) {
sprintf(str, "cdev%d", i);
static int __ccwgroup_create_symlinks(struct ccwgroup_device *gdev)
{
- char str[8];
+ char str[16];
int i, rc;
for (i = 0; i < gdev->count; i++) {
#include "qdio.h"
/* that gives us 15 characters in the text event views */
-#define QDIO_DBF_LEN 16
+#define QDIO_DBF_LEN 32
extern debug_info_t *qdio_dbf_setup;
extern debug_info_t *qdio_dbf_error;
};
struct qeth_discipline {
+ const struct device_type *devtype;
void (*start_poll)(struct ccw_device *, int, unsigned long);
qdio_handler_t *input_handler;
qdio_handler_t *output_handler;
extern struct qeth_discipline qeth_l3_discipline;
extern const struct attribute_group *qeth_generic_attr_groups[];
extern const struct attribute_group *qeth_osn_attr_groups[];
+extern const struct attribute_group qeth_device_attr_group;
+extern const struct attribute_group qeth_device_blkt_group;
+extern const struct device_type qeth_generic_devtype;
extern struct workqueue_struct *qeth_wq;
int qeth_card_hw_is_reachable(struct qeth_card *);
card->discipline = NULL;
}
-static const struct device_type qeth_generic_devtype = {
+const struct device_type qeth_generic_devtype = {
.name = "qeth_generic",
.groups = qeth_generic_attr_groups,
};
+EXPORT_SYMBOL_GPL(qeth_generic_devtype);
+
static const struct device_type qeth_osn_devtype = {
.name = "qeth_osn",
.groups = qeth_osn_attr_groups,
goto err_card;
}
- if (card->info.type == QETH_CARD_TYPE_OSN)
- gdev->dev.type = &qeth_osn_devtype;
- else
- gdev->dev.type = &qeth_generic_devtype;
-
switch (card->info.type) {
case QETH_CARD_TYPE_OSN:
case QETH_CARD_TYPE_OSM:
rc = qeth_core_load_discipline(card, QETH_DISCIPLINE_LAYER2);
if (rc)
goto err_card;
+
+ gdev->dev.type = (card->info.type != QETH_CARD_TYPE_OSN)
+ ? card->discipline->devtype
+ : &qeth_osn_devtype;
rc = card->discipline->setup(card->gdev);
if (rc)
goto err_disc;
- case QETH_CARD_TYPE_OSD:
- case QETH_CARD_TYPE_OSX:
+ break;
default:
+ gdev->dev.type = &qeth_generic_devtype;
break;
}
if (rc)
goto err;
rc = card->discipline->setup(card->gdev);
- if (rc)
+ if (rc) {
+ qeth_core_free_discipline(card);
goto err;
+ }
}
rc = card->discipline->set_online(gdev);
err:
if (card->options.layer2 == newdis)
goto out;
- else {
- card->info.mac_bits = 0;
- if (card->discipline) {
- card->discipline->remove(card->gdev);
- qeth_core_free_discipline(card);
- }
+ if (card->info.type == QETH_CARD_TYPE_OSM) {
+ /* fixed layer, can't switch */
+ rc = -EOPNOTSUPP;
+ goto out;
+ }
+
+ card->info.mac_bits = 0;
+ if (card->discipline) {
+ card->discipline->remove(card->gdev);
+ qeth_core_free_discipline(card);
}
rc = qeth_core_load_discipline(card, newdis);
goto out;
rc = card->discipline->setup(card->gdev);
+ if (rc)
+ qeth_core_free_discipline(card);
out:
mutex_unlock(&card->discipline_mutex);
return rc ? rc : count;
&dev_attr_inter_jumbo.attr,
NULL,
};
-static struct attribute_group qeth_device_blkt_group = {
+const struct attribute_group qeth_device_blkt_group = {
.name = "blkt",
.attrs = qeth_blkt_device_attrs,
};
+EXPORT_SYMBOL_GPL(qeth_device_blkt_group);
static struct attribute *qeth_device_attrs[] = {
&dev_attr_state.attr,
&dev_attr_switch_attrs.attr,
NULL,
};
-static struct attribute_group qeth_device_attr_group = {
+const struct attribute_group qeth_device_attr_group = {
.attrs = qeth_device_attrs,
};
+EXPORT_SYMBOL_GPL(qeth_device_attr_group);
const struct attribute_group *qeth_generic_attr_groups[] = {
&qeth_device_attr_group,
#include "qeth_core.h"
+extern const struct attribute_group *qeth_l2_attr_groups[];
+
int qeth_l2_create_device_attributes(struct device *);
void qeth_l2_remove_device_attributes(struct device *);
void qeth_l2_setup_bridgeport_attrs(struct qeth_card *card);
return 0;
}
+static const struct device_type qeth_l2_devtype = {
+ .name = "qeth_layer2",
+ .groups = qeth_l2_attr_groups,
+};
+
static int qeth_l2_probe_device(struct ccwgroup_device *gdev)
{
struct qeth_card *card = dev_get_drvdata(&gdev->dev);
+ int rc;
- qeth_l2_create_device_attributes(&gdev->dev);
+ if (gdev->dev.type == &qeth_generic_devtype) {
+ rc = qeth_l2_create_device_attributes(&gdev->dev);
+ if (rc)
+ return rc;
+ }
INIT_LIST_HEAD(&card->vid_list);
hash_init(card->mac_htable);
card->options.layer2 = 1;
{
struct qeth_card *card = dev_get_drvdata(&cgdev->dev);
- qeth_l2_remove_device_attributes(&cgdev->dev);
+ if (cgdev->dev.type == &qeth_generic_devtype)
+ qeth_l2_remove_device_attributes(&cgdev->dev);
qeth_set_allowed_threads(card, 0, 1);
wait_event(card->wait_q, qeth_threads_running(card, 0xffffffff) == 0);
case QETH_CARD_TYPE_OSN:
card->dev = alloc_netdev(0, "osn%d", NET_NAME_UNKNOWN,
ether_setup);
- card->dev->flags |= IFF_NOARP;
break;
default:
card->dev = alloc_etherdev(0);
card->dev->min_mtu = 64;
card->dev->max_mtu = ETH_MAX_MTU;
card->dev->netdev_ops = &qeth_l2_netdev_ops;
- card->dev->ethtool_ops =
- (card->info.type != QETH_CARD_TYPE_OSN) ?
- &qeth_l2_ethtool_ops : &qeth_l2_osn_ops;
+ if (card->info.type == QETH_CARD_TYPE_OSN) {
+ card->dev->ethtool_ops = &qeth_l2_osn_ops;
+ card->dev->flags |= IFF_NOARP;
+ } else {
+ card->dev->ethtool_ops = &qeth_l2_ethtool_ops;
+ }
card->dev->features |= NETIF_F_HW_VLAN_CTAG_FILTER;
if (card->info.type == QETH_CARD_TYPE_OSD && !card->info.guestlan) {
card->dev->hw_features = NETIF_F_SG;
}
struct qeth_discipline qeth_l2_discipline = {
+ .devtype = &qeth_l2_devtype,
.start_poll = qeth_qdio_start_poll,
.input_handler = (qdio_handler_t *) qeth_qdio_input_handler,
.output_handler = (qdio_handler_t *) qeth_qdio_output_handler,
} else
qeth_bridgeport_an_set(card, 0);
}
+
+const struct attribute_group *qeth_l2_attr_groups[] = {
+ &qeth_device_attr_group,
+ &qeth_device_blkt_group,
+ /* l2 specific, see l2_{create,remove}_device_attributes(): */
+ &qeth_l2_bridgeport_attr_group,
+ NULL,
+};
static int qeth_l3_probe_device(struct ccwgroup_device *gdev)
{
struct qeth_card *card = dev_get_drvdata(&gdev->dev);
+ int rc;
- qeth_l3_create_device_attributes(&gdev->dev);
+ rc = qeth_l3_create_device_attributes(&gdev->dev);
+ if (rc)
+ return rc;
+ hash_init(card->ip_htable);
+ hash_init(card->ip_mc_htable);
card->options.layer2 = 0;
card->info.hwtrap = 0;
return 0;
}
struct qeth_discipline qeth_l3_discipline = {
+ .devtype = &qeth_generic_devtype,
.start_poll = qeth_qdio_start_poll,
.input_handler = (qdio_handler_t *) qeth_qdio_input_handler,
.output_handler = (qdio_handler_t *) qeth_qdio_output_handler,
} __packed;
struct virtio_feature_desc {
- __u32 features;
+ __le32 features;
__u8 index;
} __packed;
goto bye;
}
- mempool_free(mbp, hw->mb_mempool);
if (finicsum != cfcsum) {
csio_warn(hw,
"Config File checksum mismatch: csum=%#x, computed=%#x\n",
rv = csio_hw_validate_caps(hw, mbp);
if (rv != 0)
goto bye;
+
+ mempool_free(mbp, hw->mb_mempool);
+ mbp = NULL;
+
/*
* Note that we're operating with parameters
* not supplied by the driver, rather than from hard-wired
config CXLFLASH
tristate "Support for IBM CAPI Flash"
depends on PCI && SCSI && CXL && EEH
+ select IRQ_POLL
default m
help
Allows CAPI Accelerated IO to Flash
* can_queue. Eventually we will hit the point where we run
* on all reserved structs.
*/
-static void fc_fcp_can_queue_ramp_down(struct fc_lport *lport)
+static bool fc_fcp_can_queue_ramp_down(struct fc_lport *lport)
{
struct fc_fcp_internal *si = fc_get_scsi_internal(lport);
unsigned long flags;
int can_queue;
+ bool changed = false;
spin_lock_irqsave(lport->host->host_lock, flags);
if (!can_queue)
can_queue = 1;
lport->host->can_queue = can_queue;
+ changed = true;
unlock:
spin_unlock_irqrestore(lport->host->host_lock, flags);
+ return changed;
}
/*
if (!fc_fcp_lport_queue_ready(lport)) {
if (lport->qfull) {
- fc_fcp_can_queue_ramp_down(lport);
- shost_printk(KERN_ERR, lport->host,
- "libfc: queue full, "
- "reducing can_queue to %d.\n",
- lport->host->can_queue);
+ if (fc_fcp_can_queue_ramp_down(lport))
+ shost_printk(KERN_ERR, lport->host,
+ "libfc: queue full, "
+ "reducing can_queue to %d.\n",
+ lport->host->can_queue);
}
rc = SCSI_MLQUEUE_HOST_BUSY;
goto out;
fp = fc_frame_alloc(lport, sizeof(*rtv));
if (!fp) {
rjt_data.reason = ELS_RJT_UNAB;
- rjt_data.reason = ELS_EXPL_INSUF_RES;
+ rjt_data.explan = ELS_EXPL_INSUF_RES;
fc_seq_els_rsp_send(in_fp, ELS_LS_RJT, &rjt_data);
goto drop;
}
uint32_t buffer_tag; /* used for tagged queue ring */
};
+struct lpfc_nvmet_ctxbuf {
+ struct list_head list;
+ struct lpfc_nvmet_rcv_ctx *context;
+ struct lpfc_iocbq *iocbq;
+ struct lpfc_sglq *sglq;
+};
+
struct lpfc_dma_pool {
struct lpfc_dmabuf *elements;
uint32_t max_count;
struct lpfc_dmabuf dbuf;
uint16_t total_size;
uint16_t bytes_recv;
- void *context;
- struct lpfc_iocbq *iocbq;
- struct lpfc_sglq *sglq;
+ uint16_t idx;
struct lpfc_queue *hrq; /* ptr to associated Header RQ */
struct lpfc_queue *drq; /* ptr to associated Data RQ */
};
/* INIT_LINK mailbox command */
#define LS_NPIV_FAB_SUPPORTED 0x2 /* Fabric supports NPIV */
#define LS_IGNORE_ERATT 0x4 /* intr handler should ignore ERATT */
+#define LS_MDS_LINK_DOWN 0x8 /* MDS Diagnostics Link Down */
+#define LS_MDS_LOOPBACK 0x16 /* MDS Diagnostics Link Up (Loopback) */
uint32_t hba_flag; /* hba generic flags */
#define HBA_ERATT_HANDLED 0x1 /* This flag is set when eratt handled */
uint32_t cfg_nvme_oas;
uint32_t cfg_nvme_io_channel;
uint32_t cfg_nvmet_mrq;
- uint32_t cfg_nvmet_mrq_post;
uint32_t cfg_enable_nvmet;
uint32_t cfg_nvme_enable_fb;
uint32_t cfg_nvmet_fb_size;
struct pci_pool *lpfc_mbuf_pool;
struct pci_pool *lpfc_hrb_pool; /* header receive buffer pool */
struct pci_pool *lpfc_drb_pool; /* data receive buffer pool */
+ struct pci_pool *lpfc_nvmet_drb_pool; /* data receive buffer pool */
struct pci_pool *lpfc_hbq_pool; /* SLI3 hbq buffer pool */
struct pci_pool *txrdy_payload_pool;
struct lpfc_dma_pool lpfc_mbuf_safety_pool;
static inline struct lpfc_sli_ring *
lpfc_phba_elsring(struct lpfc_hba *phba)
{
- if (phba->sli_rev == LPFC_SLI_REV4)
- return phba->sli4_hba.els_wq->pring;
+ if (phba->sli_rev == LPFC_SLI_REV4) {
+ if (phba->sli4_hba.els_wq)
+ return phba->sli4_hba.els_wq->pring;
+ else
+ return NULL;
+ }
return &phba->sli.sli3_ring[LPFC_ELS_RING];
}
#define LPFC_MIN_DEVLOSS_TMO 1
#define LPFC_MAX_DEVLOSS_TMO 255
-#define LPFC_DEF_MRQ_POST 256
-#define LPFC_MIN_MRQ_POST 32
-#define LPFC_MAX_MRQ_POST 512
+#define LPFC_DEF_MRQ_POST 512
+#define LPFC_MIN_MRQ_POST 512
+#define LPFC_MAX_MRQ_POST 2048
/*
* Write key size should be multiple of 4. If write key is changed
atomic_read(&tgtp->xmt_ls_rsp_error));
len += snprintf(buf+len, PAGE_SIZE-len,
- "FCP: Rcv %08x Drop %08x\n",
+ "FCP: Rcv %08x Release %08x Drop %08x\n",
atomic_read(&tgtp->rcv_fcp_cmd_in),
+ atomic_read(&tgtp->xmt_fcp_release),
atomic_read(&tgtp->rcv_fcp_cmd_drop));
if (atomic_read(&tgtp->rcv_fcp_cmd_in) !=
}
len += snprintf(buf+len, PAGE_SIZE-len,
- "FCP Rsp: RD %08x rsp %08x WR %08x rsp %08x\n",
+ "FCP Rsp: RD %08x rsp %08x WR %08x rsp %08x "
+ "drop %08x\n",
atomic_read(&tgtp->xmt_fcp_read),
atomic_read(&tgtp->xmt_fcp_read_rsp),
atomic_read(&tgtp->xmt_fcp_write),
- atomic_read(&tgtp->xmt_fcp_rsp));
-
- len += snprintf(buf+len, PAGE_SIZE-len,
- "FCP Rsp: abort %08x drop %08x\n",
- atomic_read(&tgtp->xmt_fcp_abort),
+ atomic_read(&tgtp->xmt_fcp_rsp),
atomic_read(&tgtp->xmt_fcp_drop));
len += snprintf(buf+len, PAGE_SIZE-len,
atomic_read(&tgtp->xmt_fcp_rsp_drop));
len += snprintf(buf+len, PAGE_SIZE-len,
- "ABORT: Xmt %08x Err %08x Cmpl %08x",
+ "ABORT: Xmt %08x Cmpl %08x\n",
+ atomic_read(&tgtp->xmt_fcp_abort),
+ atomic_read(&tgtp->xmt_fcp_abort_cmpl));
+
+ len += snprintf(buf + len, PAGE_SIZE - len,
+ "ABORT: Sol %08x Usol %08x Err %08x Cmpl %08x",
+ atomic_read(&tgtp->xmt_abort_sol),
+ atomic_read(&tgtp->xmt_abort_unsol),
atomic_read(&tgtp->xmt_abort_rsp),
- atomic_read(&tgtp->xmt_abort_rsp_error),
- atomic_read(&tgtp->xmt_abort_cmpl));
+ atomic_read(&tgtp->xmt_abort_rsp_error));
+
+ len += snprintf(buf + len, PAGE_SIZE - len,
+ "IO_CTX: %08x outstanding %08x total %x",
+ phba->sli4_hba.nvmet_ctx_cnt,
+ phba->sli4_hba.nvmet_io_wait_cnt,
+ phba->sli4_hba.nvmet_io_wait_total);
len += snprintf(buf+len, PAGE_SIZE-len, "\n");
return len;
1, 1, 16,
"Specify number of RQ pairs for processing NVMET cmds");
-/*
- * lpfc_nvmet_mrq_post: Specify number buffers to post on every MRQ
- *
- */
-LPFC_ATTR_R(nvmet_mrq_post, LPFC_DEF_MRQ_POST,
- LPFC_MIN_MRQ_POST, LPFC_MAX_MRQ_POST,
- "Specify number of buffers to post on every MRQ");
-
/*
* lpfc_enable_fc4_type: Defines what FC4 types are supported.
* Supported Values: 1 - register just FCP
&dev_attr_lpfc_suppress_rsp,
&dev_attr_lpfc_nvme_io_channel,
&dev_attr_lpfc_nvmet_mrq,
- &dev_attr_lpfc_nvmet_mrq_post,
&dev_attr_lpfc_nvme_enable_fb,
&dev_attr_lpfc_nvmet_fb_size,
&dev_attr_lpfc_enable_bg,
lpfc_enable_fc4_type_init(phba, lpfc_enable_fc4_type);
lpfc_nvmet_mrq_init(phba, lpfc_nvmet_mrq);
- lpfc_nvmet_mrq_post_init(phba, lpfc_nvmet_mrq_post);
/* Initialize first burst. Target vs Initiator are different. */
lpfc_nvme_enable_fb_init(phba, lpfc_nvme_enable_fb);
/* Not NVME Target mode. Turn off Target parameters. */
phba->nvmet_support = 0;
phba->cfg_nvmet_mrq = 0;
- phba->cfg_nvmet_mrq_post = 0;
phba->cfg_nvmet_fb_size = 0;
}
void lpfc_cancel_all_vport_retry_delay_timer(struct lpfc_hba *);
void lpfc_retry_pport_discovery(struct lpfc_hba *);
void lpfc_release_rpi(struct lpfc_hba *, struct lpfc_vport *, uint16_t);
+int lpfc_init_iocb_list(struct lpfc_hba *phba, int cnt);
+void lpfc_free_iocb_list(struct lpfc_hba *phba);
+int lpfc_post_rq_buffer(struct lpfc_hba *phba, struct lpfc_queue *hrq,
+ struct lpfc_queue *drq, int count, int idx);
void lpfc_mbx_cmpl_local_config_link(struct lpfc_hba *, LPFC_MBOXQ_t *);
void lpfc_mbx_cmpl_reg_login(struct lpfc_hba *, LPFC_MBOXQ_t *);
void lpfc_sli4_rb_free(struct lpfc_hba *, struct hbq_dmabuf *);
struct rqb_dmabuf *lpfc_sli4_nvmet_alloc(struct lpfc_hba *phba);
void lpfc_sli4_nvmet_free(struct lpfc_hba *phba, struct rqb_dmabuf *dmab);
-void lpfc_nvmet_rq_post(struct lpfc_hba *phba, struct lpfc_nvmet_rcv_ctx *ctxp,
- struct lpfc_dmabuf *mp);
+void lpfc_nvmet_ctxbuf_post(struct lpfc_hba *phba,
+ struct lpfc_nvmet_ctxbuf *ctxp);
int lpfc_nvmet_rcv_unsol_abort(struct lpfc_vport *vport,
struct fc_frame_header *fc_hdr);
void lpfc_sli4_build_dflt_fcf_record(struct lpfc_hba *, struct fcf_record *,
uint16_t);
int lpfc_sli4_rq_put(struct lpfc_queue *hq, struct lpfc_queue *dq,
struct lpfc_rqe *hrqe, struct lpfc_rqe *drqe);
-int lpfc_post_rq_buffer(struct lpfc_hba *phba, struct lpfc_queue *hq,
- struct lpfc_queue *dq, int count);
int lpfc_free_rq_buffer(struct lpfc_hba *phba, struct lpfc_queue *hq);
void lpfc_unregister_fcf(struct lpfc_hba *);
void lpfc_unregister_fcf_rescan(struct lpfc_hba *);
void lpfc_sli4_clear_fcf_rr_bmask(struct lpfc_hba *);
int lpfc_mem_alloc(struct lpfc_hba *, int align);
+int lpfc_nvmet_mem_alloc(struct lpfc_hba *phba);
int lpfc_mem_alloc_active_rrq_pool_s4(struct lpfc_hba *);
void lpfc_mem_free(struct lpfc_hba *);
void lpfc_mem_free_all(struct lpfc_hba *);
void lpfc_reset_barrier(struct lpfc_hba *);
int lpfc_sli_brdready(struct lpfc_hba *, uint32_t);
int lpfc_sli_brdkill(struct lpfc_hba *);
+int lpfc_sli_chipset_init(struct lpfc_hba *phba);
int lpfc_sli_brdreset(struct lpfc_hba *);
int lpfc_sli_brdrestart(struct lpfc_hba *);
int lpfc_sli_hba_setup(struct lpfc_hba *);
NLP_EVT_DEVICE_RECOVERY);
spin_lock_irq(shost->host_lock);
ndlp->nlp_flag &= ~NLP_NVMET_RECOV;
- spin_lock_irq(shost->host_lock);
+ spin_unlock_irq(shost->host_lock);
}
}
ae->un.AttrTypes[3] = 0x02; /* Type 1 - ELS */
ae->un.AttrTypes[2] = 0x01; /* Type 8 - FCP */
+ ae->un.AttrTypes[6] = 0x01; /* Type 40 - NVME */
ae->un.AttrTypes[7] = 0x01; /* Type 32 - CT */
size = FOURBYTES + 32;
ad->AttrLen = cpu_to_be16(size);
atomic_read(&tgtp->xmt_fcp_write),
atomic_read(&tgtp->xmt_fcp_rsp));
- len += snprintf(buf + len, size - len,
- "FCP Rsp: abort %08x drop %08x\n",
- atomic_read(&tgtp->xmt_fcp_abort),
- atomic_read(&tgtp->xmt_fcp_drop));
-
len += snprintf(buf + len, size - len,
"FCP Rsp Cmpl: %08x err %08x drop %08x\n",
atomic_read(&tgtp->xmt_fcp_rsp_cmpl),
atomic_read(&tgtp->xmt_fcp_rsp_drop));
len += snprintf(buf + len, size - len,
- "ABORT: Xmt %08x Err %08x Cmpl %08x",
+ "ABORT: Xmt %08x Cmpl %08x\n",
+ atomic_read(&tgtp->xmt_fcp_abort),
+ atomic_read(&tgtp->xmt_fcp_abort_cmpl));
+
+ len += snprintf(buf + len, size - len,
+ "ABORT: Sol %08x Usol %08x Err %08x Cmpl %08x",
+ atomic_read(&tgtp->xmt_abort_sol),
+ atomic_read(&tgtp->xmt_abort_unsol),
atomic_read(&tgtp->xmt_abort_rsp),
- atomic_read(&tgtp->xmt_abort_rsp_error),
- atomic_read(&tgtp->xmt_abort_cmpl));
+ atomic_read(&tgtp->xmt_abort_rsp_error));
len += snprintf(buf + len, size - len, "\n");
}
spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
}
+
+ len += snprintf(buf + len, size - len,
+ "IO_CTX: %08x outstanding %08x total %08x\n",
+ phba->sli4_hba.nvmet_ctx_cnt,
+ phba->sli4_hba.nvmet_io_wait_cnt,
+ phba->sli4_hba.nvmet_io_wait_total);
} else {
if (!(phba->cfg_enable_fc4_type & LPFC_ENABLE_NVME))
return len;
atomic_set(&tgtp->rcv_ls_req_out, 0);
atomic_set(&tgtp->rcv_ls_req_drop, 0);
atomic_set(&tgtp->xmt_ls_abort, 0);
+ atomic_set(&tgtp->xmt_ls_abort_cmpl, 0);
atomic_set(&tgtp->xmt_ls_rsp, 0);
atomic_set(&tgtp->xmt_ls_drop, 0);
atomic_set(&tgtp->xmt_ls_rsp_error, 0);
atomic_set(&tgtp->rcv_fcp_cmd_in, 0);
atomic_set(&tgtp->rcv_fcp_cmd_out, 0);
atomic_set(&tgtp->rcv_fcp_cmd_drop, 0);
- atomic_set(&tgtp->xmt_fcp_abort, 0);
atomic_set(&tgtp->xmt_fcp_drop, 0);
atomic_set(&tgtp->xmt_fcp_read_rsp, 0);
atomic_set(&tgtp->xmt_fcp_read, 0);
atomic_set(&tgtp->xmt_fcp_write, 0);
atomic_set(&tgtp->xmt_fcp_rsp, 0);
+ atomic_set(&tgtp->xmt_fcp_release, 0);
atomic_set(&tgtp->xmt_fcp_rsp_cmpl, 0);
atomic_set(&tgtp->xmt_fcp_rsp_error, 0);
atomic_set(&tgtp->xmt_fcp_rsp_drop, 0);
+ atomic_set(&tgtp->xmt_fcp_abort, 0);
+ atomic_set(&tgtp->xmt_fcp_abort_cmpl, 0);
+ atomic_set(&tgtp->xmt_abort_sol, 0);
+ atomic_set(&tgtp->xmt_abort_unsol, 0);
atomic_set(&tgtp->xmt_abort_rsp, 0);
atomic_set(&tgtp->xmt_abort_rsp_error, 0);
- atomic_set(&tgtp->xmt_abort_cmpl, 0);
}
return nbytes;
}
qp->assoc_qid, qp->q_cnt_1,
(unsigned long long)qp->q_cnt_4);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len,
- "\t\tWQID[%02d], QE-CNT[%04d], QE-SIZE[%04d], "
- "HOST-IDX[%04d], PORT-IDX[%04d]",
+ "\t\tWQID[%02d], QE-CNT[%04d], QE-SZ[%04d], "
+ "HST-IDX[%04d], PRT-IDX[%04d], PST[%03d]",
qp->queue_id, qp->entry_count,
qp->entry_size, qp->host_index,
- qp->hba_index);
+ qp->hba_index, qp->entry_repost);
len += snprintf(pbuffer + len,
LPFC_QUE_INFO_GET_BUF_SIZE - len, "\n");
return len;
qp->assoc_qid, qp->q_cnt_1, qp->q_cnt_2,
qp->q_cnt_3, (unsigned long long)qp->q_cnt_4);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len,
- "\tCQID[%02d], QE-CNT[%04d], QE-SIZE[%04d], "
- "HOST-IDX[%04d], PORT-IDX[%04d]",
+ "\tCQID[%02d], QE-CNT[%04d], QE-SZ[%04d], "
+ "HST-IDX[%04d], PRT-IDX[%04d], PST[%03d]",
qp->queue_id, qp->entry_count,
qp->entry_size, qp->host_index,
- qp->hba_index);
+ qp->hba_index, qp->entry_repost);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len, "\n");
"\t\t%s RQ info: ", rqtype);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len,
"AssocCQID[%02d]: RQ-STAT[nopost:x%x nobuf:x%x "
- "trunc:x%x rcv:x%llx]\n",
+ "posted:x%x rcv:x%llx]\n",
qp->assoc_qid, qp->q_cnt_1, qp->q_cnt_2,
qp->q_cnt_3, (unsigned long long)qp->q_cnt_4);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len,
- "\t\tHQID[%02d], QE-CNT[%04d], QE-SIZE[%04d], "
- "HOST-IDX[%04d], PORT-IDX[%04d]\n",
+ "\t\tHQID[%02d], QE-CNT[%04d], QE-SZ[%04d], "
+ "HST-IDX[%04d], PRT-IDX[%04d], PST[%03d]\n",
qp->queue_id, qp->entry_count, qp->entry_size,
- qp->host_index, qp->hba_index);
+ qp->host_index, qp->hba_index, qp->entry_repost);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len,
- "\t\tDQID[%02d], QE-CNT[%04d], QE-SIZE[%04d], "
- "HOST-IDX[%04d], PORT-IDX[%04d]\n",
+ "\t\tDQID[%02d], QE-CNT[%04d], QE-SZ[%04d], "
+ "HST-IDX[%04d], PRT-IDX[%04d], PST[%03d]\n",
datqp->queue_id, datqp->entry_count,
datqp->entry_size, datqp->host_index,
- datqp->hba_index);
+ datqp->hba_index, datqp->entry_repost);
return len;
}
eqtype, qp->q_cnt_1, qp->q_cnt_2, qp->q_cnt_3,
(unsigned long long)qp->q_cnt_4);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len,
- "EQID[%02d], QE-CNT[%04d], QE-SIZE[%04d], "
- "HOST-IDX[%04d], PORT-IDX[%04d]",
+ "EQID[%02d], QE-CNT[%04d], QE-SZ[%04d], "
+ "HST-IDX[%04d], PRT-IDX[%04d], PST[%03d]",
qp->queue_id, qp->entry_count, qp->entry_size,
- qp->host_index, qp->hba_index);
+ qp->host_index, qp->hba_index, qp->entry_repost);
len += snprintf(pbuffer + len, LPFC_QUE_INFO_GET_BUF_SIZE - len, "\n");
return len;
atomic_dec(&lpfc_debugfs_hba_count);
}
- debugfs_remove(lpfc_debugfs_root); /* lpfc */
- lpfc_debugfs_root = NULL;
+ if (atomic_read(&lpfc_debugfs_hba_count) == 0) {
+ debugfs_remove(lpfc_debugfs_root); /* lpfc */
+ lpfc_debugfs_root = NULL;
+ }
}
#endif
return;
#define NLP_FCP_INITIATOR 0x10 /* entry is an FCP Initiator */
#define NLP_NVME_TARGET 0x20 /* entry is a NVME Target */
#define NLP_NVME_INITIATOR 0x40 /* entry is a NVME Initiator */
+#define NLP_NVME_DISCOVERY 0x80 /* entry has NVME disc srvc */
uint16_t nlp_fc4_type; /* FC types node supports. */
/* Assigned from GID_FF, only
irsp->ulpStatus, irsp->un.ulpWord[4],
irsp->ulpTimeout);
+
+ /* If this is not a loop open failure, bail out */
+ if (!(irsp->ulpStatus == IOSTAT_LOCAL_REJECT &&
+ ((irsp->un.ulpWord[4] & IOERR_PARAM_MASK) ==
+ IOERR_LOOP_OPEN_FAILURE)))
+ goto flogifail;
+
/* FLOGI failed, so there is no fabric */
spin_lock_irq(shost->host_lock);
vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP);
if (irsp->ulpStatus) {
/* Check for retry */
+ ndlp->fc4_prli_sent--;
if (lpfc_els_retry(phba, cmdiocb, rspiocb)) {
/* ELS command is being retried */
- ndlp->fc4_prli_sent--;
goto out;
}
+
/* PRLI failed */
lpfc_printf_vlog(vport, KERN_ERR, LOG_ELS,
- "2754 PRLI failure DID:%06X Status:x%x/x%x\n",
+ "2754 PRLI failure DID:%06X Status:x%x/x%x, "
+ "data: x%x\n",
ndlp->nlp_DID, irsp->ulpStatus,
- irsp->un.ulpWord[4]);
+ irsp->un.ulpWord[4], ndlp->fc4_prli_sent);
+
/* Do not call DSM for lpfc_els_abort'ed ELS cmds */
if (lpfc_error_lost_link(irsp))
goto out;
*/
spin_lock_irq(&phba->hbalock);
pring = lpfc_phba_elsring(phba);
+
+ /* Bail out if we've no ELS wq, like in PCI error recovery case. */
+ if (unlikely(!pring)) {
+ spin_unlock_irq(&phba->hbalock);
+ return;
+ }
+
if (phba->sli_rev == LPFC_SLI_REV4)
spin_lock(&pring->ring_lock);
lpfc_do_scr_ns_plogi(phba, vport);
goto out;
fdisc_failed:
- if (vport->fc_vport->vport_state != FC_VPORT_NO_FABRIC_RSCS)
+ if (vport->fc_vport &&
+ (vport->fc_vport->vport_state != FC_VPORT_NO_FABRIC_RSCS))
lpfc_vport_set_state(vport, FC_VPORT_FAILED);
/* Cancel discovery timer */
lpfc_can_disctmo(vport);
pring = lpfc_phba_elsring(phba);
status = (ha_copy & (HA_RXMASK << (4*LPFC_ELS_RING)));
status >>= (4*LPFC_ELS_RING);
- if ((status & HA_RXMASK) ||
- (pring->flag & LPFC_DEFERRED_RING_EVENT) ||
- (phba->hba_flag & HBA_SP_QUEUE_EVT)) {
+ if (pring && (status & HA_RXMASK ||
+ pring->flag & LPFC_DEFERRED_RING_EVENT ||
+ phba->hba_flag & HBA_SP_QUEUE_EVT)) {
if (pring->flag & LPFC_STOP_IOCB_EVENT) {
pring->flag |= LPFC_DEFERRED_RING_EVENT;
/* Set the lpfc data pending flag */
set_bit(LPFC_DATA_READY, &phba->data_flags);
} else {
- if (phba->link_state >= LPFC_LINK_UP) {
+ if (phba->link_state >= LPFC_LINK_UP ||
+ phba->link_flag & LS_MDS_LOOPBACK) {
pring->flag &= ~LPFC_DEFERRED_RING_EVENT;
lpfc_sli_handle_slow_ring_event(phba, pring,
(status &
#define LPFC_HDR_BUF_SIZE 128
#define LPFC_DATA_BUF_SIZE 2048
+#define LPFC_NVMET_DATA_BUF_SIZE 128
struct rq_context {
uint32_t word0;
#define lpfc_rq_context_rqe_count_SHIFT 16 /* Version 0 Only */
};
#define TXRDY_PAYLOAD_LEN 12
+#define CMD_SEND_FRAME 0xE1
+
+struct send_frame_wqe {
+ struct ulp_bde64 bde; /* words 0-2 */
+ uint32_t frame_len; /* word 3 */
+ uint32_t fc_hdr_wd0; /* word 4 */
+ uint32_t fc_hdr_wd1; /* word 5 */
+ struct wqe_common wqe_com; /* words 6-11 */
+ uint32_t fc_hdr_wd2; /* word 12 */
+ uint32_t fc_hdr_wd3; /* word 13 */
+ uint32_t fc_hdr_wd4; /* word 14 */
+ uint32_t fc_hdr_wd5; /* word 15 */
+};
union lpfc_wqe {
uint32_t words[16];
struct fcp_trsp64_wqe fcp_trsp;
struct fcp_tsend64_wqe fcp_tsend;
struct fcp_treceive64_wqe fcp_treceive;
-
+ struct send_frame_wqe send_frame;
};
union lpfc_wqe128 {
list_for_each_entry_safe(ctxp, ctxp_next, &nvmet_aborts, list) {
ctxp->flag &= ~(LPFC_NVMET_XBUSY | LPFC_NVMET_ABORT_OP);
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ lpfc_nvmet_ctxbuf_post(phba, ctxp->ctxbuf);
}
}
{
struct lpfc_sglq *sglq_entry = NULL, *sglq_entry_next = NULL;
uint16_t i, lxri, xri_cnt, els_xri_cnt;
- uint16_t nvmet_xri_cnt, tot_cnt;
+ uint16_t nvmet_xri_cnt;
LIST_HEAD(nvmet_sgl_list);
int rc;
* update on pci function's nvmet xri-sgl list
*/
els_xri_cnt = lpfc_sli4_get_els_iocb_cnt(phba);
- nvmet_xri_cnt = phba->cfg_nvmet_mrq * phba->cfg_nvmet_mrq_post;
- tot_cnt = phba->sli4_hba.max_cfg_param.max_xri - els_xri_cnt;
- if (nvmet_xri_cnt > tot_cnt) {
- phba->cfg_nvmet_mrq_post = tot_cnt / phba->cfg_nvmet_mrq;
- nvmet_xri_cnt = phba->cfg_nvmet_mrq * phba->cfg_nvmet_mrq_post;
- lpfc_printf_log(phba, KERN_INFO, LOG_SLI,
- "6301 NVMET post-sgl count changed to %d\n",
- phba->cfg_nvmet_mrq_post);
- }
+
+ /* For NVMET, ALL remaining XRIs are dedicated for IO processing */
+ nvmet_xri_cnt = phba->sli4_hba.max_cfg_param.max_xri - els_xri_cnt;
if (nvmet_xri_cnt > phba->sli4_hba.nvmet_xri_cnt) {
/* els xri-sgl expanded */
LPFC_MBOXQ_t *mboxq;
MAILBOX_t *mb;
+ if (phba->sli_rev < LPFC_SLI_REV4) {
+ /* Reset the port first */
+ lpfc_sli_brdrestart(phba);
+ rc = lpfc_sli_chipset_init(phba);
+ if (rc)
+ return (uint64_t)-1;
+ }
mboxq = (LPFC_MBOXQ_t *) mempool_alloc(phba->mbox_mem_pool,
GFP_KERNEL);
pmb->vport = phba->pport;
if (phba->sli4_hba.link_state.status != LPFC_FC_LA_TYPE_LINK_UP) {
+ phba->link_flag &= ~(LS_MDS_LINK_DOWN | LS_MDS_LOOPBACK);
+
+ switch (phba->sli4_hba.link_state.status) {
+ case LPFC_FC_LA_TYPE_MDS_LINK_DOWN:
+ phba->link_flag |= LS_MDS_LINK_DOWN;
+ break;
+ case LPFC_FC_LA_TYPE_MDS_LOOPBACK:
+ phba->link_flag |= LS_MDS_LOOPBACK;
+ break;
+ default:
+ break;
+ }
+
/* Parse and translate status field */
mb = &pmb->u.mb;
mb->mbxStatus = lpfc_sli4_parse_latt_fault(phba,
spin_lock_init(&phba->sli4_hba.abts_nvme_buf_list_lock);
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_nvme_buf_list);
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_abts_nvmet_ctx_list);
+ INIT_LIST_HEAD(&phba->sli4_hba.lpfc_nvmet_ctx_list);
+ INIT_LIST_HEAD(&phba->sli4_hba.lpfc_nvmet_io_wait_list);
+
/* Fast-path XRI aborted CQ Event work queue list */
INIT_LIST_HEAD(&phba->sli4_hba.sp_nvme_xri_aborted_work_queue);
}
/* This abort list used by worker thread */
spin_lock_init(&phba->sli4_hba.sgl_list_lock);
spin_lock_init(&phba->sli4_hba.nvmet_io_lock);
+ spin_lock_init(&phba->sli4_hba.nvmet_io_wait_lock);
/*
* Initialize driver internal slow-path work queues
for (i = 0; i < lpfc_enable_nvmet_cnt; i++) {
if (wwn == lpfc_enable_nvmet[i]) {
#if (IS_ENABLED(CONFIG_NVME_TARGET_FC))
+ if (lpfc_nvmet_mem_alloc(phba))
+ break;
+
+ phba->nvmet_support = 1; /* a match */
+
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"6017 NVME Target %016llx\n",
wwn);
- phba->nvmet_support = 1; /* a match */
#else
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"6021 Can't enable NVME Target."
" NVME_TARGET_FC infrastructure"
" is not in kernel\n");
#endif
+ break;
}
}
}
*
* This routine is invoked to free the driver's IOCB list and memory.
**/
-static void
+void
lpfc_free_iocb_list(struct lpfc_hba *phba)
{
struct lpfc_iocbq *iocbq_entry = NULL, *iocbq_next = NULL;
* 0 - successful
* other values - error
**/
-static int
+int
lpfc_init_iocb_list(struct lpfc_hba *phba, int iocb_count)
{
struct lpfc_iocbq *iocbq_entry = NULL;
uint16_t rpi_limit, curr_rpi_range;
struct lpfc_dmabuf *dmabuf;
struct lpfc_rpi_hdr *rpi_hdr;
- uint32_t rpi_count;
/*
* If the SLI4 port supports extents, posting the rpi header isn't
return NULL;
/* The limit on the logical index is just the max_rpi count. */
- rpi_limit = phba->sli4_hba.max_cfg_param.rpi_base +
- phba->sli4_hba.max_cfg_param.max_rpi - 1;
+ rpi_limit = phba->sli4_hba.max_cfg_param.max_rpi;
spin_lock_irq(&phba->hbalock);
/*
curr_rpi_range = phba->sli4_hba.next_rpi;
spin_unlock_irq(&phba->hbalock);
- /*
- * The port has a limited number of rpis. The increment here
- * is LPFC_RPI_HDR_COUNT - 1 to account for the starting value
- * and to allow the full max_rpi range per port.
- */
- if ((curr_rpi_range + (LPFC_RPI_HDR_COUNT - 1)) > rpi_limit)
- rpi_count = rpi_limit - curr_rpi_range;
- else
- rpi_count = LPFC_RPI_HDR_COUNT;
-
- if (!rpi_count)
+ /* Reached full RPI range */
+ if (curr_rpi_range == rpi_limit)
return NULL;
+
/*
* First allocate the protocol header region for the port. The
* port expects a 4KB DMA-mapped memory region that is 4K aligned.
/* The rpi_hdr stores the logical index only. */
rpi_hdr->start_rpi = curr_rpi_range;
+ rpi_hdr->next_rpi = phba->sli4_hba.next_rpi + LPFC_RPI_HDR_COUNT;
list_add_tail(&rpi_hdr->list, &phba->sli4_hba.lpfc_rpi_hdr_list);
- /*
- * The next_rpi stores the next logical module-64 rpi value used
- * to post physical rpis in subsequent rpi postings.
- */
- phba->sli4_hba.next_rpi += rpi_count;
spin_unlock_irq(&phba->hbalock);
return rpi_hdr;
/* Create NVMET Receive Queue for header */
qdesc = lpfc_sli4_queue_alloc(phba,
phba->sli4_hba.rq_esize,
- phba->sli4_hba.rq_ecount);
+ LPFC_NVMET_RQE_DEF_COUNT);
if (!qdesc) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"3146 Failed allocate "
/* Create NVMET Receive Queue for data */
qdesc = lpfc_sli4_queue_alloc(phba,
phba->sli4_hba.rq_esize,
- phba->sli4_hba.rq_ecount);
+ LPFC_NVMET_RQE_DEF_COUNT);
if (!qdesc) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"3156 Failed allocate "
INIT_LIST_HEAD(&phba->sli4_hba.lpfc_wq_list);
}
-int
-lpfc_post_rq_buffer(struct lpfc_hba *phba, struct lpfc_queue *hrq,
- struct lpfc_queue *drq, int count)
-{
- int rc, i;
- struct lpfc_rqe hrqe;
- struct lpfc_rqe drqe;
- struct lpfc_rqb *rqbp;
- struct rqb_dmabuf *rqb_buffer;
- LIST_HEAD(rqb_buf_list);
-
- rqbp = hrq->rqbp;
- for (i = 0; i < count; i++) {
- rqb_buffer = (rqbp->rqb_alloc_buffer)(phba);
- if (!rqb_buffer)
- break;
- rqb_buffer->hrq = hrq;
- rqb_buffer->drq = drq;
- list_add_tail(&rqb_buffer->hbuf.list, &rqb_buf_list);
- }
- while (!list_empty(&rqb_buf_list)) {
- list_remove_head(&rqb_buf_list, rqb_buffer, struct rqb_dmabuf,
- hbuf.list);
-
- hrqe.address_lo = putPaddrLow(rqb_buffer->hbuf.phys);
- hrqe.address_hi = putPaddrHigh(rqb_buffer->hbuf.phys);
- drqe.address_lo = putPaddrLow(rqb_buffer->dbuf.phys);
- drqe.address_hi = putPaddrHigh(rqb_buffer->dbuf.phys);
- rc = lpfc_sli4_rq_put(hrq, drq, &hrqe, &drqe);
- if (rc < 0) {
- (rqbp->rqb_free_buffer)(phba, rqb_buffer);
- } else {
- list_add_tail(&rqb_buffer->hbuf.list,
- &rqbp->rqb_buffer_list);
- rqbp->buffer_count++;
- }
- }
- return 1;
-}
-
int
lpfc_free_rq_buffer(struct lpfc_hba *phba, struct lpfc_queue *rq)
{
goto out_destroy;
}
- lpfc_rq_adjust_repost(phba, phba->sli4_hba.hdr_rq, LPFC_ELS_HBQ);
- lpfc_rq_adjust_repost(phba, phba->sli4_hba.dat_rq, LPFC_ELS_HBQ);
-
rc = lpfc_rq_create(phba, phba->sli4_hba.hdr_rq, phba->sli4_hba.dat_rq,
phba->sli4_hba.els_cq, LPFC_USOL);
if (rc) {
lpfc_wq_destroy(phba, phba->sli4_hba.nvmels_wq);
/* Unset ELS work queue */
- if (phba->sli4_hba.els_cq)
+ if (phba->sli4_hba.els_wq)
lpfc_wq_destroy(phba, phba->sli4_hba.els_wq);
/* Unset unsolicited receive queue */
struct lpfc_hba *phba;
struct lpfc_vport *vport = NULL;
struct Scsi_Host *shost = NULL;
- int error, cnt;
+ int error;
uint32_t cfg_mode, intr_mode;
/* Allocate memory for HBA structure */
goto out_unset_pci_mem_s4;
}
- cnt = phba->cfg_iocb_cnt * 1024;
- if (phba->nvmet_support)
- cnt += phba->cfg_nvmet_mrq_post * phba->cfg_nvmet_mrq;
-
- /* Initialize and populate the iocb list per host */
- lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
- "2821 initialize iocb list %d total %d\n",
- phba->cfg_iocb_cnt, cnt);
- error = lpfc_init_iocb_list(phba, cnt);
-
- if (error) {
- lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
- "1413 Failed to initialize iocb list.\n");
- goto out_unset_driver_resource_s4;
- }
-
INIT_LIST_HEAD(&phba->active_rrq_list);
INIT_LIST_HEAD(&phba->fcf.fcf_pri_list);
if (error) {
lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"1414 Failed to set up driver resource.\n");
- goto out_free_iocb_list;
+ goto out_unset_driver_resource_s4;
}
/* Get the default values for Model Name and Description */
lpfc_destroy_shost(phba);
out_unset_driver_resource:
lpfc_unset_driver_resource_phase2(phba);
-out_free_iocb_list:
- lpfc_free_iocb_list(phba);
out_unset_driver_resource_s4:
lpfc_sli4_driver_resource_unset(phba);
out_unset_pci_mem_s4:
return -ENOMEM;
}
+int
+lpfc_nvmet_mem_alloc(struct lpfc_hba *phba)
+{
+ phba->lpfc_nvmet_drb_pool =
+ pci_pool_create("lpfc_nvmet_drb_pool",
+ phba->pcidev, LPFC_NVMET_DATA_BUF_SIZE,
+ SGL_ALIGN_SZ, 0);
+ if (!phba->lpfc_nvmet_drb_pool) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
+ "6024 Can't enable NVME Target - no memory\n");
+ return -ENOMEM;
+ }
+ return 0;
+}
+
/**
* lpfc_mem_free - Frees memory allocated by lpfc_mem_alloc
* @phba: HBA to free memory for
/* Free HBQ pools */
lpfc_sli_hbqbuf_free_all(phba);
+ if (phba->lpfc_nvmet_drb_pool)
+ pci_pool_destroy(phba->lpfc_nvmet_drb_pool);
+ phba->lpfc_nvmet_drb_pool = NULL;
if (phba->lpfc_drb_pool)
pci_pool_destroy(phba->lpfc_drb_pool);
phba->lpfc_drb_pool = NULL;
lpfc_sli4_nvmet_alloc(struct lpfc_hba *phba)
{
struct rqb_dmabuf *dma_buf;
- struct lpfc_iocbq *nvmewqe;
- union lpfc_wqe128 *wqe;
dma_buf = kzalloc(sizeof(struct rqb_dmabuf), GFP_KERNEL);
if (!dma_buf)
kfree(dma_buf);
return NULL;
}
- dma_buf->dbuf.virt = pci_pool_alloc(phba->lpfc_drb_pool, GFP_KERNEL,
- &dma_buf->dbuf.phys);
+ dma_buf->dbuf.virt = pci_pool_alloc(phba->lpfc_nvmet_drb_pool,
+ GFP_KERNEL, &dma_buf->dbuf.phys);
if (!dma_buf->dbuf.virt) {
pci_pool_free(phba->lpfc_hrb_pool, dma_buf->hbuf.virt,
dma_buf->hbuf.phys);
kfree(dma_buf);
return NULL;
}
- dma_buf->total_size = LPFC_DATA_BUF_SIZE;
-
- dma_buf->context = kzalloc(sizeof(struct lpfc_nvmet_rcv_ctx),
- GFP_KERNEL);
- if (!dma_buf->context) {
- pci_pool_free(phba->lpfc_drb_pool, dma_buf->dbuf.virt,
- dma_buf->dbuf.phys);
- pci_pool_free(phba->lpfc_hrb_pool, dma_buf->hbuf.virt,
- dma_buf->hbuf.phys);
- kfree(dma_buf);
- return NULL;
- }
-
- dma_buf->iocbq = lpfc_sli_get_iocbq(phba);
- if (!dma_buf->iocbq) {
- kfree(dma_buf->context);
- pci_pool_free(phba->lpfc_drb_pool, dma_buf->dbuf.virt,
- dma_buf->dbuf.phys);
- pci_pool_free(phba->lpfc_hrb_pool, dma_buf->hbuf.virt,
- dma_buf->hbuf.phys);
- kfree(dma_buf);
- lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
- "2621 Ran out of nvmet iocb/WQEs\n");
- return NULL;
- }
- dma_buf->iocbq->iocb_flag = LPFC_IO_NVMET;
- nvmewqe = dma_buf->iocbq;
- wqe = (union lpfc_wqe128 *)&nvmewqe->wqe;
- /* Initialize WQE */
- memset(wqe, 0, sizeof(union lpfc_wqe));
- /* Word 7 */
- bf_set(wqe_ct, &wqe->generic.wqe_com, SLI4_CT_RPI);
- bf_set(wqe_class, &wqe->generic.wqe_com, CLASS3);
- bf_set(wqe_pu, &wqe->generic.wqe_com, 1);
- /* Word 10 */
- bf_set(wqe_nvme, &wqe->fcp_tsend.wqe_com, 1);
- bf_set(wqe_ebde_cnt, &wqe->generic.wqe_com, 0);
- bf_set(wqe_qosd, &wqe->generic.wqe_com, 0);
-
- dma_buf->iocbq->context1 = NULL;
- spin_lock(&phba->sli4_hba.sgl_list_lock);
- dma_buf->sglq = __lpfc_sli_get_nvmet_sglq(phba, dma_buf->iocbq);
- spin_unlock(&phba->sli4_hba.sgl_list_lock);
- if (!dma_buf->sglq) {
- lpfc_sli_release_iocbq(phba, dma_buf->iocbq);
- kfree(dma_buf->context);
- pci_pool_free(phba->lpfc_drb_pool, dma_buf->dbuf.virt,
- dma_buf->dbuf.phys);
- pci_pool_free(phba->lpfc_hrb_pool, dma_buf->hbuf.virt,
- dma_buf->hbuf.phys);
- kfree(dma_buf);
- lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
- "6132 Ran out of nvmet XRIs\n");
- return NULL;
- }
+ dma_buf->total_size = LPFC_NVMET_DATA_BUF_SIZE;
return dma_buf;
}
void
lpfc_sli4_nvmet_free(struct lpfc_hba *phba, struct rqb_dmabuf *dmab)
{
- unsigned long flags;
-
- __lpfc_clear_active_sglq(phba, dmab->sglq->sli4_lxritag);
- dmab->sglq->state = SGL_FREED;
- dmab->sglq->ndlp = NULL;
-
- spin_lock_irqsave(&phba->sli4_hba.sgl_list_lock, flags);
- list_add_tail(&dmab->sglq->list, &phba->sli4_hba.lpfc_nvmet_sgl_list);
- spin_unlock_irqrestore(&phba->sli4_hba.sgl_list_lock, flags);
-
- lpfc_sli_release_iocbq(phba, dmab->iocbq);
- kfree(dmab->context);
pci_pool_free(phba->lpfc_hrb_pool, dmab->hbuf.virt, dmab->hbuf.phys);
- pci_pool_free(phba->lpfc_drb_pool, dmab->dbuf.virt, dmab->dbuf.phys);
+ pci_pool_free(phba->lpfc_nvmet_drb_pool,
+ dmab->dbuf.virt, dmab->dbuf.phys);
kfree(dmab);
}
rc = lpfc_sli4_rq_put(rqb_entry->hrq, rqb_entry->drq, &hrqe, &drqe);
if (rc < 0) {
(rqbp->rqb_free_buffer)(phba, rqb_entry);
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
+ "6409 Cannot post to RQ %d: %x %x\n",
+ rqb_entry->hrq->queue_id,
+ rqb_entry->hrq->host_index,
+ rqb_entry->hrq->hba_index);
} else {
list_add_tail(&rqb_entry->hbuf.list, &rqbp->rqb_buffer_list);
rqbp->buffer_count++;
/* Target driver cannot solicit NVME FB. */
if (bf_get_be32(prli_tgt, nvpr)) {
+ /* Complete the nvme target roles. The transport
+ * needs to know if the rport is capable of
+ * discovery in addition to its role.
+ */
ndlp->nlp_type |= NLP_NVME_TARGET;
+ if (bf_get_be32(prli_disc, nvpr))
+ ndlp->nlp_type |= NLP_NVME_DISCOVERY;
if ((bf_get_be32(prli_fba, nvpr) == 1) &&
(bf_get_be32(prli_fb_sz, nvpr) > 0) &&
(phba->cfg_nvme_enable_fb) &&
}
/**
- * lpfc_nvmet_rq_post - Repost a NVMET RQ DMA buffer and clean up context
+ * lpfc_nvmet_ctxbuf_post - Repost a NVMET RQ DMA buffer and clean up context
* @phba: HBA buffer is associated with
* @ctxp: context to clean up
* @mp: Buffer to free
* Returns: None
**/
void
-lpfc_nvmet_rq_post(struct lpfc_hba *phba, struct lpfc_nvmet_rcv_ctx *ctxp,
- struct lpfc_dmabuf *mp)
+lpfc_nvmet_ctxbuf_post(struct lpfc_hba *phba, struct lpfc_nvmet_ctxbuf *ctx_buf)
{
- if (ctxp) {
- if (ctxp->flag)
- lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
- "6314 rq_post ctx xri x%x flag x%x\n",
- ctxp->oxid, ctxp->flag);
-
- if (ctxp->txrdy) {
- pci_pool_free(phba->txrdy_payload_pool, ctxp->txrdy,
- ctxp->txrdy_phys);
- ctxp->txrdy = NULL;
- ctxp->txrdy_phys = 0;
+#if (IS_ENABLED(CONFIG_NVME_TARGET_FC))
+ struct lpfc_nvmet_rcv_ctx *ctxp = ctx_buf->context;
+ struct lpfc_nvmet_tgtport *tgtp;
+ struct fc_frame_header *fc_hdr;
+ struct rqb_dmabuf *nvmebuf;
+ struct lpfc_dmabuf *hbufp;
+ uint32_t *payload;
+ uint32_t size, oxid, sid, rc;
+ unsigned long iflag;
+
+ if (ctxp->txrdy) {
+ pci_pool_free(phba->txrdy_payload_pool, ctxp->txrdy,
+ ctxp->txrdy_phys);
+ ctxp->txrdy = NULL;
+ ctxp->txrdy_phys = 0;
+ }
+ ctxp->state = LPFC_NVMET_STE_FREE;
+
+ spin_lock_irqsave(&phba->sli4_hba.nvmet_io_wait_lock, iflag);
+ if (phba->sli4_hba.nvmet_io_wait_cnt) {
+ hbufp = &nvmebuf->hbuf;
+ list_remove_head(&phba->sli4_hba.lpfc_nvmet_io_wait_list,
+ nvmebuf, struct rqb_dmabuf,
+ hbuf.list);
+ phba->sli4_hba.nvmet_io_wait_cnt--;
+ spin_unlock_irqrestore(&phba->sli4_hba.nvmet_io_wait_lock,
+ iflag);
+
+ fc_hdr = (struct fc_frame_header *)(nvmebuf->hbuf.virt);
+ oxid = be16_to_cpu(fc_hdr->fh_ox_id);
+ tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
+ payload = (uint32_t *)(nvmebuf->dbuf.virt);
+ size = nvmebuf->bytes_recv;
+ sid = sli4_sid_from_fc_hdr(fc_hdr);
+
+ ctxp = (struct lpfc_nvmet_rcv_ctx *)ctx_buf->context;
+ memset(ctxp, 0, sizeof(ctxp->ctx));
+ ctxp->wqeq = NULL;
+ ctxp->txrdy = NULL;
+ ctxp->offset = 0;
+ ctxp->phba = phba;
+ ctxp->size = size;
+ ctxp->oxid = oxid;
+ ctxp->sid = sid;
+ ctxp->state = LPFC_NVMET_STE_RCV;
+ ctxp->entry_cnt = 1;
+ ctxp->flag = 0;
+ ctxp->ctxbuf = ctx_buf;
+ spin_lock_init(&ctxp->ctxlock);
+
+#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
+ if (phba->ktime_on) {
+ ctxp->ts_cmd_nvme = ktime_get_ns();
+ ctxp->ts_isr_cmd = ctxp->ts_cmd_nvme;
+ ctxp->ts_nvme_data = 0;
+ ctxp->ts_data_wqput = 0;
+ ctxp->ts_isr_data = 0;
+ ctxp->ts_data_nvme = 0;
+ ctxp->ts_nvme_status = 0;
+ ctxp->ts_status_wqput = 0;
+ ctxp->ts_isr_status = 0;
+ ctxp->ts_status_nvme = 0;
}
- ctxp->state = LPFC_NVMET_STE_FREE;
+#endif
+ atomic_inc(&tgtp->rcv_fcp_cmd_in);
+ /*
+ * The calling sequence should be:
+ * nvmet_fc_rcv_fcp_req->lpfc_nvmet_xmt_fcp_op/cmp- req->done
+ * lpfc_nvmet_xmt_fcp_op_cmp should free the allocated ctxp.
+ * When we return from nvmet_fc_rcv_fcp_req, all relevant info
+ * the NVME command / FC header is stored.
+ * A buffer has already been reposted for this IO, so just free
+ * the nvmebuf.
+ */
+ rc = nvmet_fc_rcv_fcp_req(phba->targetport, &ctxp->ctx.fcp_req,
+ payload, size);
+
+ /* Process FCP command */
+ if (rc == 0) {
+ atomic_inc(&tgtp->rcv_fcp_cmd_out);
+ nvmebuf->hrq->rqbp->rqb_free_buffer(phba, nvmebuf);
+ return;
+ }
+
+ atomic_inc(&tgtp->rcv_fcp_cmd_drop);
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
+ "2582 FCP Drop IO x%x: err x%x: x%x x%x x%x\n",
+ ctxp->oxid, rc,
+ atomic_read(&tgtp->rcv_fcp_cmd_in),
+ atomic_read(&tgtp->rcv_fcp_cmd_out),
+ atomic_read(&tgtp->xmt_fcp_release));
+
+ lpfc_nvmet_defer_release(phba, ctxp);
+ lpfc_nvmet_unsol_fcp_issue_abort(phba, ctxp, sid, oxid);
+ nvmebuf->hrq->rqbp->rqb_free_buffer(phba, nvmebuf);
+ return;
}
- lpfc_rq_buf_free(phba, mp);
+ spin_unlock_irqrestore(&phba->sli4_hba.nvmet_io_wait_lock, iflag);
+
+ spin_lock_irqsave(&phba->sli4_hba.nvmet_io_lock, iflag);
+ list_add_tail(&ctx_buf->list,
+ &phba->sli4_hba.lpfc_nvmet_ctx_list);
+ phba->sli4_hba.nvmet_ctx_cnt++;
+ spin_unlock_irqrestore(&phba->sli4_hba.nvmet_io_lock, iflag);
+#endif
}
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
"6150 LS Drop IO x%x: Prep\n",
ctxp->oxid);
lpfc_in_buf_free(phba, &nvmebuf->dbuf);
+ atomic_inc(&nvmep->xmt_ls_abort);
lpfc_nvmet_unsol_ls_issue_abort(phba, ctxp,
ctxp->sid, ctxp->oxid);
return -ENOMEM;
lpfc_nlp_put(nvmewqeq->context1);
lpfc_in_buf_free(phba, &nvmebuf->dbuf);
+ atomic_inc(&nvmep->xmt_ls_abort);
lpfc_nvmet_unsol_ls_issue_abort(phba, ctxp, ctxp->sid, ctxp->oxid);
return -ENXIO;
}
lpfc_nvmeio_data(phba, "NVMET FCP CMND: xri x%x op x%x len x%x\n",
ctxp->oxid, rsp->op, rsp->rsplen);
+ ctxp->flag |= LPFC_NVMET_IO_INP;
rc = lpfc_sli4_issue_wqe(phba, LPFC_FCP_RING, nvmewqeq);
if (rc == WQE_SUCCESS) {
- ctxp->flag |= LPFC_NVMET_IO_INP;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
if (!phba->ktime_on)
return 0;
lpfc_nvmet_xmt_fcp_release(struct nvmet_fc_target_port *tgtport,
struct nvmefc_tgt_fcp_req *rsp)
{
+ struct lpfc_nvmet_tgtport *lpfc_nvmep = tgtport->private;
struct lpfc_nvmet_rcv_ctx *ctxp =
container_of(rsp, struct lpfc_nvmet_rcv_ctx, ctx.fcp_req);
struct lpfc_hba *phba = ctxp->phba;
lpfc_nvmeio_data(phba, "NVMET FCP FREE: xri x%x ste %d\n", ctxp->oxid,
ctxp->state, 0);
+ atomic_inc(&lpfc_nvmep->xmt_fcp_release);
+
if (aborting)
return;
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ lpfc_nvmet_ctxbuf_post(phba, ctxp->ctxbuf);
}
static struct nvmet_fc_target_template lpfc_tgttemplate = {
.target_priv_sz = sizeof(struct lpfc_nvmet_tgtport),
};
+void
+lpfc_nvmet_cleanup_io_context(struct lpfc_hba *phba)
+{
+ struct lpfc_nvmet_ctxbuf *ctx_buf, *next_ctx_buf;
+ unsigned long flags;
+
+ list_for_each_entry_safe(
+ ctx_buf, next_ctx_buf,
+ &phba->sli4_hba.lpfc_nvmet_ctx_list, list) {
+ spin_lock_irqsave(
+ &phba->sli4_hba.abts_nvme_buf_list_lock, flags);
+ list_del_init(&ctx_buf->list);
+ spin_unlock_irqrestore(
+ &phba->sli4_hba.abts_nvme_buf_list_lock, flags);
+ __lpfc_clear_active_sglq(phba,
+ ctx_buf->sglq->sli4_lxritag);
+ ctx_buf->sglq->state = SGL_FREED;
+ ctx_buf->sglq->ndlp = NULL;
+
+ spin_lock_irqsave(&phba->sli4_hba.sgl_list_lock, flags);
+ list_add_tail(&ctx_buf->sglq->list,
+ &phba->sli4_hba.lpfc_nvmet_sgl_list);
+ spin_unlock_irqrestore(&phba->sli4_hba.sgl_list_lock,
+ flags);
+
+ lpfc_sli_release_iocbq(phba, ctx_buf->iocbq);
+ kfree(ctx_buf->context);
+ }
+}
+
+int
+lpfc_nvmet_setup_io_context(struct lpfc_hba *phba)
+{
+ struct lpfc_nvmet_ctxbuf *ctx_buf;
+ struct lpfc_iocbq *nvmewqe;
+ union lpfc_wqe128 *wqe;
+ int i;
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME,
+ "6403 Allocate NVMET resources for %d XRIs\n",
+ phba->sli4_hba.nvmet_xri_cnt);
+
+ /* For all nvmet xris, allocate resources needed to process a
+ * received command on a per xri basis.
+ */
+ for (i = 0; i < phba->sli4_hba.nvmet_xri_cnt; i++) {
+ ctx_buf = kzalloc(sizeof(*ctx_buf), GFP_KERNEL);
+ if (!ctx_buf) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
+ "6404 Ran out of memory for NVMET\n");
+ return -ENOMEM;
+ }
+
+ ctx_buf->context = kzalloc(sizeof(*ctx_buf->context),
+ GFP_KERNEL);
+ if (!ctx_buf->context) {
+ kfree(ctx_buf);
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
+ "6405 Ran out of NVMET "
+ "context memory\n");
+ return -ENOMEM;
+ }
+ ctx_buf->context->ctxbuf = ctx_buf;
+
+ ctx_buf->iocbq = lpfc_sli_get_iocbq(phba);
+ if (!ctx_buf->iocbq) {
+ kfree(ctx_buf->context);
+ kfree(ctx_buf);
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
+ "6406 Ran out of NVMET iocb/WQEs\n");
+ return -ENOMEM;
+ }
+ ctx_buf->iocbq->iocb_flag = LPFC_IO_NVMET;
+ nvmewqe = ctx_buf->iocbq;
+ wqe = (union lpfc_wqe128 *)&nvmewqe->wqe;
+ /* Initialize WQE */
+ memset(wqe, 0, sizeof(union lpfc_wqe));
+ /* Word 7 */
+ bf_set(wqe_ct, &wqe->generic.wqe_com, SLI4_CT_RPI);
+ bf_set(wqe_class, &wqe->generic.wqe_com, CLASS3);
+ bf_set(wqe_pu, &wqe->generic.wqe_com, 1);
+ /* Word 10 */
+ bf_set(wqe_nvme, &wqe->fcp_tsend.wqe_com, 1);
+ bf_set(wqe_ebde_cnt, &wqe->generic.wqe_com, 0);
+ bf_set(wqe_qosd, &wqe->generic.wqe_com, 0);
+
+ ctx_buf->iocbq->context1 = NULL;
+ spin_lock(&phba->sli4_hba.sgl_list_lock);
+ ctx_buf->sglq = __lpfc_sli_get_nvmet_sglq(phba, ctx_buf->iocbq);
+ spin_unlock(&phba->sli4_hba.sgl_list_lock);
+ if (!ctx_buf->sglq) {
+ lpfc_sli_release_iocbq(phba, ctx_buf->iocbq);
+ kfree(ctx_buf->context);
+ kfree(ctx_buf);
+ lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
+ "6407 Ran out of NVMET XRIs\n");
+ return -ENOMEM;
+ }
+ spin_lock(&phba->sli4_hba.nvmet_io_lock);
+ list_add_tail(&ctx_buf->list,
+ &phba->sli4_hba.lpfc_nvmet_ctx_list);
+ spin_unlock(&phba->sli4_hba.nvmet_io_lock);
+ }
+ phba->sli4_hba.nvmet_ctx_cnt = phba->sli4_hba.nvmet_xri_cnt;
+ return 0;
+}
+
int
lpfc_nvmet_create_targetport(struct lpfc_hba *phba)
{
struct lpfc_vport *vport = phba->pport;
struct lpfc_nvmet_tgtport *tgtp;
struct nvmet_fc_port_info pinfo;
- int error = 0;
+ int error;
if (phba->targetport)
return 0;
+ error = lpfc_nvmet_setup_io_context(phba);
+ if (error)
+ return error;
+
memset(&pinfo, 0, sizeof(struct nvmet_fc_port_info));
pinfo.node_name = wwn_to_u64(vport->fc_nodename.u.wwn);
pinfo.port_name = wwn_to_u64(vport->fc_portname.u.wwn);
lpfc_tgttemplate.max_sgl_segments = phba->cfg_nvme_seg_cnt + 1;
lpfc_tgttemplate.max_hw_queues = phba->cfg_nvme_io_channel;
lpfc_tgttemplate.target_features = NVMET_FCTGTFEAT_READDATA_RSP |
- NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED |
NVMET_FCTGTFEAT_CMD_IN_ISR |
NVMET_FCTGTFEAT_OPDONE_IN_ISR;
&phba->pcidev->dev,
&phba->targetport);
#else
- error = -ENOMEM;
+ error = -ENOENT;
#endif
if (error) {
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_DISC,
"6025 Cannot register NVME targetport "
"x%x\n", error);
phba->targetport = NULL;
+
+ lpfc_nvmet_cleanup_io_context(phba);
+
} else {
tgtp = (struct lpfc_nvmet_tgtport *)
phba->targetport->private;
atomic_set(&tgtp->rcv_ls_req_out, 0);
atomic_set(&tgtp->rcv_ls_req_drop, 0);
atomic_set(&tgtp->xmt_ls_abort, 0);
+ atomic_set(&tgtp->xmt_ls_abort_cmpl, 0);
atomic_set(&tgtp->xmt_ls_rsp, 0);
atomic_set(&tgtp->xmt_ls_drop, 0);
atomic_set(&tgtp->xmt_ls_rsp_error, 0);
atomic_set(&tgtp->rcv_fcp_cmd_in, 0);
atomic_set(&tgtp->rcv_fcp_cmd_out, 0);
atomic_set(&tgtp->rcv_fcp_cmd_drop, 0);
- atomic_set(&tgtp->xmt_fcp_abort, 0);
atomic_set(&tgtp->xmt_fcp_drop, 0);
atomic_set(&tgtp->xmt_fcp_read_rsp, 0);
atomic_set(&tgtp->xmt_fcp_read, 0);
atomic_set(&tgtp->xmt_fcp_write, 0);
atomic_set(&tgtp->xmt_fcp_rsp, 0);
+ atomic_set(&tgtp->xmt_fcp_release, 0);
atomic_set(&tgtp->xmt_fcp_rsp_cmpl, 0);
atomic_set(&tgtp->xmt_fcp_rsp_error, 0);
atomic_set(&tgtp->xmt_fcp_rsp_drop, 0);
+ atomic_set(&tgtp->xmt_fcp_abort, 0);
+ atomic_set(&tgtp->xmt_fcp_abort_cmpl, 0);
+ atomic_set(&tgtp->xmt_abort_unsol, 0);
+ atomic_set(&tgtp->xmt_abort_sol, 0);
atomic_set(&tgtp->xmt_abort_rsp, 0);
atomic_set(&tgtp->xmt_abort_rsp_error, 0);
- atomic_set(&tgtp->xmt_abort_cmpl, 0);
}
return error;
}
list_for_each_entry_safe(ctxp, next_ctxp,
&phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
list) {
- if (ctxp->rqb_buffer->sglq->sli4_xritag != xri)
+ if (ctxp->ctxbuf->sglq->sli4_xritag != xri)
continue;
/* Check if we already received a free context call
(ndlp->nlp_state == NLP_STE_UNMAPPED_NODE ||
ndlp->nlp_state == NLP_STE_MAPPED_NODE)) {
lpfc_set_rrq_active(phba, ndlp,
- ctxp->rqb_buffer->sglq->sli4_lxritag,
+ ctxp->ctxbuf->sglq->sli4_lxritag,
rxid, 1);
lpfc_sli4_abts_err_handler(phba, ndlp, axri);
}
"6318 XB aborted %x flg x%x (%x)\n",
ctxp->oxid, ctxp->flag, released);
if (released)
- lpfc_nvmet_rq_post(phba, ctxp,
- &ctxp->rqb_buffer->hbuf);
+ lpfc_nvmet_ctxbuf_post(phba, ctxp->ctxbuf);
+
if (rrq_empty)
lpfc_worker_wake_up(phba);
return;
list_for_each_entry_safe(ctxp, next_ctxp,
&phba->sli4_hba.lpfc_abts_nvmet_ctx_list,
list) {
- if (ctxp->rqb_buffer->sglq->sli4_xritag != xri)
+ if (ctxp->ctxbuf->sglq->sli4_xritag != xri)
continue;
spin_unlock(&phba->sli4_hba.abts_nvme_buf_list_lock);
init_completion(&tgtp->tport_unreg_done);
nvmet_fc_unregister_targetport(phba->targetport);
wait_for_completion_timeout(&tgtp->tport_unreg_done, 5);
+ lpfc_nvmet_cleanup_io_context(phba);
}
phba->targetport = NULL;
#endif
oxid = 0;
size = 0;
sid = 0;
+ ctxp = NULL;
goto dropit;
}
struct lpfc_nvmet_rcv_ctx *ctxp;
struct lpfc_nvmet_tgtport *tgtp;
struct fc_frame_header *fc_hdr;
+ struct lpfc_nvmet_ctxbuf *ctx_buf;
uint32_t *payload;
- uint32_t size, oxid, sid, rc;
+ uint32_t size, oxid, sid, rc, qno;
+ unsigned long iflag;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
uint32_t id;
#endif
+ ctx_buf = NULL;
if (!nvmebuf || !phba->targetport) {
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
- "6157 FCP Drop IO\n");
+ "6157 NVMET FCP Drop IO\n");
oxid = 0;
size = 0;
sid = 0;
+ ctxp = NULL;
goto dropit;
}
+ spin_lock_irqsave(&phba->sli4_hba.nvmet_io_lock, iflag);
+ if (phba->sli4_hba.nvmet_ctx_cnt) {
+ list_remove_head(&phba->sli4_hba.lpfc_nvmet_ctx_list,
+ ctx_buf, struct lpfc_nvmet_ctxbuf, list);
+ phba->sli4_hba.nvmet_ctx_cnt--;
+ }
+ spin_unlock_irqrestore(&phba->sli4_hba.nvmet_io_lock, iflag);
- tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
- payload = (uint32_t *)(nvmebuf->dbuf.virt);
fc_hdr = (struct fc_frame_header *)(nvmebuf->hbuf.virt);
- size = nvmebuf->bytes_recv;
oxid = be16_to_cpu(fc_hdr->fh_ox_id);
- sid = sli4_sid_from_fc_hdr(fc_hdr);
+ size = nvmebuf->bytes_recv;
- ctxp = (struct lpfc_nvmet_rcv_ctx *)nvmebuf->context;
- if (ctxp == NULL) {
- atomic_inc(&tgtp->rcv_fcp_cmd_drop);
- lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
- "6158 FCP Drop IO x%x: Alloc\n",
- oxid);
- lpfc_nvmet_rq_post(phba, NULL, &nvmebuf->hbuf);
- /* Cannot send ABTS without context */
+#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
+ if (phba->cpucheck_on & LPFC_CHECK_NVMET_RCV) {
+ id = smp_processor_id();
+ if (id < LPFC_CHECK_CPU_CNT)
+ phba->cpucheck_rcv_io[id]++;
+ }
+#endif
+
+ lpfc_nvmeio_data(phba, "NVMET FCP RCV: xri x%x sz %d CPU %02x\n",
+ oxid, size, smp_processor_id());
+
+ if (!ctx_buf) {
+ /* Queue this NVME IO to process later */
+ spin_lock_irqsave(&phba->sli4_hba.nvmet_io_wait_lock, iflag);
+ list_add_tail(&nvmebuf->hbuf.list,
+ &phba->sli4_hba.lpfc_nvmet_io_wait_list);
+ phba->sli4_hba.nvmet_io_wait_cnt++;
+ phba->sli4_hba.nvmet_io_wait_total++;
+ spin_unlock_irqrestore(&phba->sli4_hba.nvmet_io_wait_lock,
+ iflag);
+
+ /* Post a brand new DMA buffer to RQ */
+ qno = nvmebuf->idx;
+ lpfc_post_rq_buffer(
+ phba, phba->sli4_hba.nvmet_mrq_hdr[qno],
+ phba->sli4_hba.nvmet_mrq_data[qno], 1, qno);
return;
}
+
+ tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
+ payload = (uint32_t *)(nvmebuf->dbuf.virt);
+ sid = sli4_sid_from_fc_hdr(fc_hdr);
+
+ ctxp = (struct lpfc_nvmet_rcv_ctx *)ctx_buf->context;
memset(ctxp, 0, sizeof(ctxp->ctx));
ctxp->wqeq = NULL;
ctxp->txrdy = NULL;
ctxp->oxid = oxid;
ctxp->sid = sid;
ctxp->state = LPFC_NVMET_STE_RCV;
- ctxp->rqb_buffer = nvmebuf;
ctxp->entry_cnt = 1;
ctxp->flag = 0;
+ ctxp->ctxbuf = ctx_buf;
spin_lock_init(&ctxp->ctxlock);
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
ctxp->ts_isr_status = 0;
ctxp->ts_status_nvme = 0;
}
-
- if (phba->cpucheck_on & LPFC_CHECK_NVMET_RCV) {
- id = smp_processor_id();
- if (id < LPFC_CHECK_CPU_CNT)
- phba->cpucheck_rcv_io[id]++;
- }
#endif
- lpfc_nvmeio_data(phba, "NVMET FCP RCV: xri x%x sz %d CPU %02x\n",
- oxid, size, smp_processor_id());
-
atomic_inc(&tgtp->rcv_fcp_cmd_in);
/*
* The calling sequence should be:
* nvmet_fc_rcv_fcp_req -> lpfc_nvmet_xmt_fcp_op/cmp -> req->done
* lpfc_nvmet_xmt_fcp_op_cmp should free the allocated ctxp.
+ * When we return from nvmet_fc_rcv_fcp_req, all relevant info in
+ * the NVME command / FC header is stored, so we are free to repost
+ * the buffer.
*/
rc = nvmet_fc_rcv_fcp_req(phba->targetport, &ctxp->ctx.fcp_req,
payload, size);
/* Process FCP command */
if (rc == 0) {
atomic_inc(&tgtp->rcv_fcp_cmd_out);
+ lpfc_rq_buf_free(phba, &nvmebuf->hbuf); /* repost */
return;
}
atomic_inc(&tgtp->rcv_fcp_cmd_drop);
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
- "6159 FCP Drop IO x%x: err x%x\n",
- ctxp->oxid, rc);
+ "6159 FCP Drop IO x%x: err x%x: x%x x%x x%x\n",
+ ctxp->oxid, rc,
+ atomic_read(&tgtp->rcv_fcp_cmd_in),
+ atomic_read(&tgtp->rcv_fcp_cmd_out),
+ atomic_read(&tgtp->xmt_fcp_release));
dropit:
lpfc_nvmeio_data(phba, "NVMET FCP DROP: xri x%x sz %d from %06x\n",
oxid, size, sid);
if (oxid) {
+ lpfc_nvmet_defer_release(phba, ctxp);
lpfc_nvmet_unsol_fcp_issue_abort(phba, ctxp, sid, oxid);
+ lpfc_rq_buf_free(phba, &nvmebuf->hbuf); /* repost */
return;
}
- if (nvmebuf) {
- nvmebuf->iocbq->hba_wqidx = 0;
- /* We assume a rcv'ed cmd ALWAYs fits into 1 buffer */
- lpfc_nvmet_rq_post(phba, NULL, &nvmebuf->hbuf);
- }
+ if (ctx_buf)
+ lpfc_nvmet_ctxbuf_post(phba, ctx_buf);
+
+ if (nvmebuf)
+ lpfc_rq_buf_free(phba, &nvmebuf->hbuf); /* repost */
#endif
}
uint64_t isr_timestamp)
{
if (phba->nvmet_support == 0) {
- lpfc_nvmet_rq_post(phba, NULL, &nvmebuf->hbuf);
+ lpfc_rq_buf_free(phba, &nvmebuf->hbuf);
return;
}
lpfc_nvmet_unsol_fcp_buffer(phba, pring, nvmebuf,
nvmewqe = ctxp->wqeq;
if (nvmewqe == NULL) {
/* Allocate buffer for command wqe */
- nvmewqe = ctxp->rqb_buffer->iocbq;
+ nvmewqe = ctxp->ctxbuf->iocbq;
if (nvmewqe == NULL) {
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_IOERR,
"6110 lpfc_nvmet_prep_fcp_wqe: No "
return NULL;
}
- sgl = (struct sli4_sge *)ctxp->rqb_buffer->sglq->sgl;
+ sgl = (struct sli4_sge *)ctxp->ctxbuf->sglq->sgl;
switch (rsp->op) {
case NVMET_FCOP_READDATA:
case NVMET_FCOP_READDATA_RSP:
result = wcqe->parameter;
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
- atomic_inc(&tgtp->xmt_abort_cmpl);
+ if (ctxp->flag & LPFC_NVMET_ABORT_OP)
+ atomic_inc(&tgtp->xmt_fcp_abort_cmpl);
ctxp->state = LPFC_NVMET_STE_DONE;
}
ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+ atomic_inc(&tgtp->xmt_abort_rsp);
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
"6165 ABORT cmpl: xri x%x flg x%x (%d) "
wcqe->word0, wcqe->total_data_placed,
result, wcqe->word3);
+ cmdwqe->context2 = NULL;
+ cmdwqe->context3 = NULL;
/*
* if transport has released ctx, then can reuse it. Otherwise,
* will be recycled by transport release call.
*/
if (released)
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
+ lpfc_nvmet_ctxbuf_post(phba, ctxp->ctxbuf);
- cmdwqe->context2 = NULL;
- cmdwqe->context3 = NULL;
+ /* This is the iocbq for the abort, not the command */
lpfc_sli_release_iocbq(phba, cmdwqe);
/* Since iaab/iaar are NOT set, there is no work left.
result = wcqe->parameter;
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
- atomic_inc(&tgtp->xmt_abort_cmpl);
+ if (ctxp->flag & LPFC_NVMET_ABORT_OP)
+ atomic_inc(&tgtp->xmt_fcp_abort_cmpl);
if (!ctxp) {
/* if context is clear, related io alrady complete */
}
ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
spin_unlock_irqrestore(&ctxp->ctxlock, flags);
+ atomic_inc(&tgtp->xmt_abort_rsp);
lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
"6316 ABTS cmpl xri x%x flg x%x (%x) "
ctxp->oxid, ctxp->flag, released,
wcqe->word0, wcqe->total_data_placed,
result, wcqe->word3);
+
+ cmdwqe->context2 = NULL;
+ cmdwqe->context3 = NULL;
/*
* if transport has released ctx, then can reuse it. Otherwise,
* will be recycled by transport release call.
*/
if (released)
- lpfc_nvmet_rq_post(phba, ctxp, &ctxp->rqb_buffer->hbuf);
-
- cmdwqe->context2 = NULL;
- cmdwqe->context3 = NULL;
+ lpfc_nvmet_ctxbuf_post(phba, ctxp->ctxbuf);
/* Since iaab/iaar are NOT set, there is no work left.
* For LPFC_NVMET_XBUSY, lpfc_sli4_nvmet_xri_aborted
result = wcqe->parameter;
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
- atomic_inc(&tgtp->xmt_abort_cmpl);
+ atomic_inc(&tgtp->xmt_ls_abort_cmpl);
lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
"6083 Abort cmpl: ctx %p WCQE: %08x %08x %08x %08x\n",
sid, xri, ctxp->wqeq->sli4_xritag);
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
- if (!ctxp->wqeq) {
- ctxp->wqeq = ctxp->rqb_buffer->iocbq;
- ctxp->wqeq->hba_wqidx = 0;
- }
ndlp = lpfc_findnode_did(phba->pport, sid);
if (!ndlp || !NLP_CHK_NODE_ACT(ndlp) ||
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
if (!ctxp->wqeq) {
- ctxp->wqeq = ctxp->rqb_buffer->iocbq;
+ ctxp->wqeq = ctxp->ctxbuf->iocbq;
ctxp->wqeq->hba_wqidx = 0;
}
/* Issue ABTS for this WQE based on iotag */
ctxp->abort_wqeq = lpfc_sli_get_iocbq(phba);
if (!ctxp->abort_wqeq) {
+ atomic_inc(&tgtp->xmt_abort_rsp_error);
lpfc_printf_log(phba, KERN_WARNING, LOG_NVME_ABTS,
"6161 ABORT failed: No wqeqs: "
"xri: x%x\n", ctxp->oxid);
/* driver queued commands are in process of being flushed */
if (phba->hba_flag & HBA_NVME_IOQ_FLUSH) {
spin_unlock_irqrestore(&phba->hbalock, flags);
+ atomic_inc(&tgtp->xmt_abort_rsp_error);
lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
"6163 Driver in reset cleanup - flushing "
"NVME Req now. hba_flag x%x oxid x%x\n",
/* Outstanding abort is in progress */
if (abts_wqeq->iocb_flag & LPFC_DRIVER_ABORTED) {
spin_unlock_irqrestore(&phba->hbalock, flags);
+ atomic_inc(&tgtp->xmt_abort_rsp_error);
lpfc_printf_log(phba, KERN_ERR, LOG_NVME,
"6164 Outstanding NVME I/O Abort Request "
"still pending on oxid x%x\n",
abts_wqeq->context2 = ctxp;
rc = lpfc_sli4_issue_wqe(phba, LPFC_FCP_RING, abts_wqeq);
spin_unlock_irqrestore(&phba->hbalock, flags);
- if (rc == WQE_SUCCESS)
+ if (rc == WQE_SUCCESS) {
+ atomic_inc(&tgtp->xmt_abort_sol);
return 0;
+ }
+ atomic_inc(&tgtp->xmt_abort_rsp_error);
ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
lpfc_sli_release_iocbq(phba, abts_wqeq);
lpfc_printf_log(phba, KERN_ERR, LOG_NVME_ABTS,
tgtp = (struct lpfc_nvmet_tgtport *)phba->targetport->private;
if (!ctxp->wqeq) {
- ctxp->wqeq = ctxp->rqb_buffer->iocbq;
+ ctxp->wqeq = ctxp->ctxbuf->iocbq;
ctxp->wqeq->hba_wqidx = 0;
}
rc = lpfc_sli4_issue_wqe(phba, LPFC_FCP_RING, abts_wqeq);
spin_unlock_irqrestore(&phba->hbalock, flags);
if (rc == WQE_SUCCESS) {
- atomic_inc(&tgtp->xmt_abort_rsp);
return 0;
}
aerr:
+ atomic_inc(&tgtp->xmt_abort_rsp_error);
ctxp->flag &= ~LPFC_NVMET_ABORT_OP;
atomic_inc(&tgtp->xmt_abort_rsp_error);
lpfc_printf_log(phba, KERN_WARNING, LOG_NVME_ABTS,
}
abts_wqeq = ctxp->wqeq;
wqe_abts = &abts_wqeq->wqe;
+
lpfc_nvmet_unsol_issue_abort(phba, ctxp, sid, xri);
spin_lock_irqsave(&phba->hbalock, flags);
rc = lpfc_sli4_issue_wqe(phba, LPFC_ELS_RING, abts_wqeq);
spin_unlock_irqrestore(&phba->hbalock, flags);
if (rc == WQE_SUCCESS) {
- atomic_inc(&tgtp->xmt_abort_rsp);
+ atomic_inc(&tgtp->xmt_abort_unsol);
return 0;
}
********************************************************************/
#define LPFC_NVMET_DEFAULT_SEGS (64 + 1) /* 256K IOs */
+#define LPFC_NVMET_RQE_DEF_COUNT 512
#define LPFC_NVMET_SUCCESS_LEN 12
/* Used for NVME Target */
atomic_t rcv_ls_req_out;
atomic_t rcv_ls_req_drop;
atomic_t xmt_ls_abort;
+ atomic_t xmt_ls_abort_cmpl;
/* Stats counters - lpfc_nvmet_xmt_ls_rsp */
atomic_t xmt_ls_rsp;
atomic_t rcv_fcp_cmd_in;
atomic_t rcv_fcp_cmd_out;
atomic_t rcv_fcp_cmd_drop;
+ atomic_t xmt_fcp_release;
/* Stats counters - lpfc_nvmet_xmt_fcp_op */
- atomic_t xmt_fcp_abort;
atomic_t xmt_fcp_drop;
atomic_t xmt_fcp_read_rsp;
atomic_t xmt_fcp_read;
atomic_t xmt_fcp_rsp_drop;
- /* Stats counters - lpfc_nvmet_unsol_issue_abort */
+ /* Stats counters - lpfc_nvmet_xmt_fcp_abort */
+ atomic_t xmt_fcp_abort;
+ atomic_t xmt_fcp_abort_cmpl;
+ atomic_t xmt_abort_sol;
+ atomic_t xmt_abort_unsol;
atomic_t xmt_abort_rsp;
atomic_t xmt_abort_rsp_error;
-
- /* Stats counters - lpfc_nvmet_xmt_abort_cmp */
- atomic_t xmt_abort_cmpl;
};
struct lpfc_nvmet_rcv_ctx {
#define LPFC_NVMET_CTX_RLS 0x8 /* ctx free requested */
#define LPFC_NVMET_ABTS_RCV 0x10 /* ABTS received on exchange */
struct rqb_dmabuf *rqb_buffer;
+ struct lpfc_nvmet_ctxbuf *ctxbuf;
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
uint64_t ts_isr_cmd;
struct lpfc_iocbq *);
static void lpfc_sli4_send_seq_to_ulp(struct lpfc_vport *,
struct hbq_dmabuf *);
+static void lpfc_sli4_handle_mds_loopback(struct lpfc_vport *vport,
+ struct hbq_dmabuf *dmabuf);
static int lpfc_sli4_fp_handle_cqe(struct lpfc_hba *, struct lpfc_queue *,
struct lpfc_cqe *);
static int lpfc_sli4_post_sgl_list(struct lpfc_hba *, struct list_head *,
if (unlikely(!hq) || unlikely(!dq))
return -ENOMEM;
put_index = hq->host_index;
- temp_hrqe = hq->qe[hq->host_index].rqe;
+ temp_hrqe = hq->qe[put_index].rqe;
temp_drqe = dq->qe[dq->host_index].rqe;
if (hq->type != LPFC_HRQ || dq->type != LPFC_DRQ)
return -EINVAL;
- if (hq->host_index != dq->host_index)
+ if (put_index != dq->host_index)
return -EINVAL;
/* If the host has not yet processed the next entry then we are done */
- if (((hq->host_index + 1) % hq->entry_count) == hq->hba_index)
+ if (((put_index + 1) % hq->entry_count) == hq->hba_index)
return -EBUSY;
lpfc_sli_pcimem_bcopy(hrqe, temp_hrqe, hq->entry_size);
lpfc_sli_pcimem_bcopy(drqe, temp_drqe, dq->entry_size);
/* Update the host index to point to the next slot */
- hq->host_index = ((hq->host_index + 1) % hq->entry_count);
+ hq->host_index = ((put_index + 1) % hq->entry_count);
dq->host_index = ((dq->host_index + 1) % dq->entry_count);
+ hq->RQ_buf_posted++;
/* Ring The Header Receive Queue Doorbell */
if (!(hq->host_index % hq->entry_repost)) {
/* Reset HBA */
lpfc_printf_log(phba, KERN_INFO, LOG_SLI,
"0325 Reset HBA Data: x%x x%x\n",
- phba->pport->port_state, psli->sli_flag);
+ (phba->pport) ? phba->pport->port_state : 0,
+ psli->sli_flag);
/* perform board reset */
phba->fc_eventTag = 0;
phba->link_events = 0;
- phba->pport->fc_myDID = 0;
- phba->pport->fc_prevDID = 0;
+ if (phba->pport) {
+ phba->pport->fc_myDID = 0;
+ phba->pport->fc_prevDID = 0;
+ }
/* Turn off parity checking and serr during the physical reset */
pci_read_config_word(phba->pcidev, PCI_COMMAND, &cfg_value);
/* Restart HBA */
lpfc_printf_log(phba, KERN_INFO, LOG_SLI,
"0337 Restart HBA Data: x%x x%x\n",
- phba->pport->port_state, psli->sli_flag);
+ (phba->pport) ? phba->pport->port_state : 0,
+ psli->sli_flag);
word0 = 0;
mb = (MAILBOX_t *) &word0;
readl(to_slim); /* flush */
/* Only skip post after fc_ffinit is completed */
- if (phba->pport->port_state)
+ if (phba->pport && phba->pport->port_state)
word0 = 1; /* This is really setting up word1 */
else
word0 = 0; /* This is really setting up word1 */
readl(to_slim); /* flush */
lpfc_sli_brdreset(phba);
- phba->pport->stopped = 0;
+ if (phba->pport)
+ phba->pport->stopped = 0;
phba->link_state = LPFC_INIT_START;
phba->hba_flag = 0;
spin_unlock_irq(&phba->hbalock);
* iteration, the function will restart the HBA again. The function returns
* zero if HBA successfully restarted else returns negative error code.
**/
-static int
+int
lpfc_sli_chipset_init(struct lpfc_hba *phba)
{
uint32_t status, i = 0;
bf_set(lpfc_mbx_set_feature_mds,
&mbox->u.mqe.un.set_feature, 1);
bf_set(lpfc_mbx_set_feature_mds_deep_loopbk,
- &mbox->u.mqe.un.set_feature, 0);
+ &mbox->u.mqe.un.set_feature, 1);
mbox->u.mqe.un.set_feature.feature = LPFC_SET_MDS_DIAGS;
mbox->u.mqe.un.set_feature.param_len = 8;
break;
(phba->hba_flag & HBA_FCOE_MODE) ? "FCoE" : "FC");
}
+int
+lpfc_post_rq_buffer(struct lpfc_hba *phba, struct lpfc_queue *hrq,
+ struct lpfc_queue *drq, int count, int idx)
+{
+ int rc, i;
+ struct lpfc_rqe hrqe;
+ struct lpfc_rqe drqe;
+ struct lpfc_rqb *rqbp;
+ struct rqb_dmabuf *rqb_buffer;
+ LIST_HEAD(rqb_buf_list);
+
+ rqbp = hrq->rqbp;
+ for (i = 0; i < count; i++) {
+ /* IF RQ is already full, don't bother */
+ if (rqbp->buffer_count + i >= rqbp->entry_count - 1)
+ break;
+ rqb_buffer = rqbp->rqb_alloc_buffer(phba);
+ if (!rqb_buffer)
+ break;
+ rqb_buffer->hrq = hrq;
+ rqb_buffer->drq = drq;
+ rqb_buffer->idx = idx;
+ list_add_tail(&rqb_buffer->hbuf.list, &rqb_buf_list);
+ }
+ while (!list_empty(&rqb_buf_list)) {
+ list_remove_head(&rqb_buf_list, rqb_buffer, struct rqb_dmabuf,
+ hbuf.list);
+
+ hrqe.address_lo = putPaddrLow(rqb_buffer->hbuf.phys);
+ hrqe.address_hi = putPaddrHigh(rqb_buffer->hbuf.phys);
+ drqe.address_lo = putPaddrLow(rqb_buffer->dbuf.phys);
+ drqe.address_hi = putPaddrHigh(rqb_buffer->dbuf.phys);
+ rc = lpfc_sli4_rq_put(hrq, drq, &hrqe, &drqe);
+ if (rc < 0) {
+ rqbp->rqb_free_buffer(phba, rqb_buffer);
+ } else {
+ list_add_tail(&rqb_buffer->hbuf.list,
+ &rqbp->rqb_buffer_list);
+ rqbp->buffer_count++;
+ }
+ }
+ return 1;
+}
+
/**
* lpfc_sli4_hba_setup - SLI4 device initialization PCI function
* @phba: Pointer to HBA context object.
int
lpfc_sli4_hba_setup(struct lpfc_hba *phba)
{
- int rc, i;
+ int rc, i, cnt;
LPFC_MBOXQ_t *mboxq;
struct lpfc_mqe *mqe;
uint8_t *vpd;
goto out_destroy_queue;
}
phba->sli4_hba.nvmet_xri_cnt = rc;
+
+ cnt = phba->cfg_iocb_cnt * 1024;
+ /* We need 1 iocbq for every SGL, for IO processing */
+ cnt += phba->sli4_hba.nvmet_xri_cnt;
+ /* Initialize and populate the iocb list per host */
+ lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
+ "2821 initialize iocb list %d total %d\n",
+ phba->cfg_iocb_cnt, cnt);
+ rc = lpfc_init_iocb_list(phba, cnt);
+ if (rc) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
+ "1413 Failed to init iocb list.\n");
+ goto out_destroy_queue;
+ }
+
lpfc_nvmet_create_targetport(phba);
} else {
/* update host scsi xri-sgl sizes and mappings */
"and mapping: %d\n", rc);
goto out_destroy_queue;
}
+
+ cnt = phba->cfg_iocb_cnt * 1024;
+ /* Initialize and populate the iocb list per host */
+ lpfc_printf_log(phba, KERN_INFO, LOG_INIT,
+ "2820 initialize iocb list %d total %d\n",
+ phba->cfg_iocb_cnt, cnt);
+ rc = lpfc_init_iocb_list(phba, cnt);
+ if (rc) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
+ "6301 Failed to init iocb list.\n");
+ goto out_destroy_queue;
+ }
}
if (phba->nvmet_support && phba->cfg_nvmet_mrq) {
-
/* Post initial buffers to all RQs created */
for (i = 0; i < phba->cfg_nvmet_mrq; i++) {
rqbp = phba->sli4_hba.nvmet_mrq_hdr[i]->rqbp;
INIT_LIST_HEAD(&rqbp->rqb_buffer_list);
rqbp->rqb_alloc_buffer = lpfc_sli4_nvmet_alloc;
rqbp->rqb_free_buffer = lpfc_sli4_nvmet_free;
- rqbp->entry_count = 256;
+ rqbp->entry_count = LPFC_NVMET_RQE_DEF_COUNT;
rqbp->buffer_count = 0;
- /* Divide by 4 and round down to multiple of 16 */
- rc = (phba->cfg_nvmet_mrq_post >> 2) & 0xfff8;
- phba->sli4_hba.nvmet_mrq_hdr[i]->entry_repost = rc;
- phba->sli4_hba.nvmet_mrq_data[i]->entry_repost = rc;
-
lpfc_post_rq_buffer(
phba, phba->sli4_hba.nvmet_mrq_hdr[i],
phba->sli4_hba.nvmet_mrq_data[i],
- phba->cfg_nvmet_mrq_post);
+ LPFC_NVMET_RQE_DEF_COUNT, i);
}
}
/* Unset all the queues set up in this routine when error out */
lpfc_sli4_queue_unset(phba);
out_destroy_queue:
+ lpfc_free_iocb_list(phba);
lpfc_sli4_queue_destroy(phba);
out_stop_timers:
lpfc_stop_hba_timers(phba);
memset(wqe, 0, sizeof(union lpfc_wqe128));
/* Some of the fields are in the right position already */
memcpy(wqe, &iocbq->iocb, sizeof(union lpfc_wqe));
- wqe->generic.wqe_com.word7 = 0; /* The ct field has moved so reset */
- wqe->generic.wqe_com.word10 = 0;
+ if (iocbq->iocb.ulpCommand != CMD_SEND_FRAME) {
+ /* The ct field has moved so reset */
+ wqe->generic.wqe_com.word7 = 0;
+ wqe->generic.wqe_com.word10 = 0;
+ }
abort_tag = (uint32_t) iocbq->iotag;
xritag = iocbq->sli4_xritag;
}
break;
+ case CMD_SEND_FRAME:
+ bf_set(wqe_xri_tag, &wqe->generic.wqe_com, xritag);
+ bf_set(wqe_reqtag, &wqe->generic.wqe_com, iocbq->iotag);
+ return 0;
case CMD_XRI_ABORTED_CX:
case CMD_CREATE_XRI_CR: /* Do we expect to use this? */
case CMD_IOCB_FCP_IBIDIR64_CR: /* bidirectional xfer */
struct fc_frame_header *fc_hdr;
struct lpfc_queue *hrq = phba->sli4_hba.hdr_rq;
struct lpfc_queue *drq = phba->sli4_hba.dat_rq;
+ struct lpfc_nvmet_tgtport *tgtp;
struct hbq_dmabuf *dma_buf;
uint32_t status, rq_id;
unsigned long iflags;
case FC_STATUS_RQ_BUF_LEN_EXCEEDED:
lpfc_printf_log(phba, KERN_ERR, LOG_SLI,
"2537 Receive Frame Truncated!!\n");
- hrq->RQ_buf_trunc++;
case FC_STATUS_RQ_SUCCESS:
lpfc_sli4_rq_release(hrq, drq);
spin_lock_irqsave(&phba->hbalock, iflags);
goto out;
}
hrq->RQ_rcv_buf++;
+ hrq->RQ_buf_posted--;
memcpy(&dma_buf->cq_event.cqe.rcqe_cmpl, rcqe, sizeof(*rcqe));
/* If a NVME LS event (type 0x28), treat it as Fast path */
spin_unlock_irqrestore(&phba->hbalock, iflags);
workposted = true;
break;
- case FC_STATUS_INSUFF_BUF_NEED_BUF:
case FC_STATUS_INSUFF_BUF_FRM_DISC:
+ if (phba->nvmet_support) {
+ tgtp = phba->targetport->private;
+ lpfc_printf_log(phba, KERN_ERR, LOG_SLI | LOG_NVME,
+ "6402 RQE Error x%x, posted %d err_cnt "
+ "%d: %x %x %x\n",
+ status, hrq->RQ_buf_posted,
+ hrq->RQ_no_posted_buf,
+ atomic_read(&tgtp->rcv_fcp_cmd_in),
+ atomic_read(&tgtp->rcv_fcp_cmd_out),
+ atomic_read(&tgtp->xmt_fcp_release));
+ }
+ /* fallthrough */
+
+ case FC_STATUS_INSUFF_BUF_NEED_BUF:
hrq->RQ_no_posted_buf++;
/* Post more buffers if possible */
spin_lock_irqsave(&phba->hbalock, iflags);
while ((cqe = lpfc_sli4_cq_get(cq))) {
workposted |= lpfc_sli4_sp_handle_mcqe(phba, cqe);
if (!(++ecount % cq->entry_repost))
- lpfc_sli4_cq_release(cq, LPFC_QUEUE_NOARM);
+ break;
cq->CQ_mbox++;
}
break;
workposted |= lpfc_sli4_sp_handle_cqe(phba, cq,
cqe);
if (!(++ecount % cq->entry_repost))
- lpfc_sli4_cq_release(cq, LPFC_QUEUE_NOARM);
+ break;
}
/* Track the max number of CQEs processed in 1 EQ */
struct lpfc_queue *drq;
struct rqb_dmabuf *dma_buf;
struct fc_frame_header *fc_hdr;
+ struct lpfc_nvmet_tgtport *tgtp;
uint32_t status, rq_id;
unsigned long iflags;
uint32_t fctl, idx;
case FC_STATUS_RQ_BUF_LEN_EXCEEDED:
lpfc_printf_log(phba, KERN_ERR, LOG_SLI,
"6126 Receive Frame Truncated!!\n");
- hrq->RQ_buf_trunc++;
- break;
case FC_STATUS_RQ_SUCCESS:
lpfc_sli4_rq_release(hrq, drq);
spin_lock_irqsave(&phba->hbalock, iflags);
}
spin_unlock_irqrestore(&phba->hbalock, iflags);
hrq->RQ_rcv_buf++;
+ hrq->RQ_buf_posted--;
fc_hdr = (struct fc_frame_header *)dma_buf->hbuf.virt;
/* Just some basic sanity checks on FCP Command frame */
drop:
lpfc_in_buf_free(phba, &dma_buf->dbuf);
break;
- case FC_STATUS_INSUFF_BUF_NEED_BUF:
case FC_STATUS_INSUFF_BUF_FRM_DISC:
+ if (phba->nvmet_support) {
+ tgtp = phba->targetport->private;
+ lpfc_printf_log(phba, KERN_ERR, LOG_SLI | LOG_NVME,
+ "6401 RQE Error x%x, posted %d err_cnt "
+ "%d: %x %x %x\n",
+ status, hrq->RQ_buf_posted,
+ hrq->RQ_no_posted_buf,
+ atomic_read(&tgtp->rcv_fcp_cmd_in),
+ atomic_read(&tgtp->rcv_fcp_cmd_out),
+ atomic_read(&tgtp->xmt_fcp_release));
+ }
+ /* fallthrough */
+
+ case FC_STATUS_INSUFF_BUF_NEED_BUF:
hrq->RQ_no_posted_buf++;
/* Post more buffers if possible */
- spin_lock_irqsave(&phba->hbalock, iflags);
- phba->hba_flag |= HBA_POST_RECEIVE_BUFFER;
- spin_unlock_irqrestore(&phba->hbalock, iflags);
- workposted = true;
break;
}
out:
while ((cqe = lpfc_sli4_cq_get(cq))) {
workposted |= lpfc_sli4_fp_handle_cqe(phba, cq, cqe);
if (!(++ecount % cq->entry_repost))
- lpfc_sli4_cq_release(cq, LPFC_QUEUE_NOARM);
+ break;
}
/* Track the max number of CQEs processed in 1 EQ */
while ((cqe = lpfc_sli4_cq_get(cq))) {
workposted |= lpfc_sli4_fp_handle_cqe(phba, cq, cqe);
if (!(++ecount % cq->entry_repost))
- lpfc_sli4_cq_release(cq, LPFC_QUEUE_NOARM);
+ break;
}
/* Track the max number of CQEs processed in 1 EQ */
while ((eqe = lpfc_sli4_eq_get(eq))) {
lpfc_sli4_fof_handle_eqe(phba, eqe);
if (!(++ecount % eq->entry_repost))
- lpfc_sli4_eq_release(eq, LPFC_QUEUE_NOARM);
+ break;
eq->EQ_processed++;
}
lpfc_sli4_hba_handle_eqe(phba, eqe, hba_eqidx);
if (!(++ecount % fpeq->entry_repost))
- lpfc_sli4_eq_release(fpeq, LPFC_QUEUE_NOARM);
+ break;
fpeq->EQ_processed++;
}
}
queue->entry_size = entry_size;
queue->entry_count = entry_count;
-
- /*
- * entry_repost is calculated based on the number of entries in the
- * queue. This works out except for RQs. If buffers are NOT initially
- * posted for every RQE, entry_repost should be adjusted accordingly.
- */
- queue->entry_repost = (entry_count >> 3);
- if (queue->entry_repost < LPFC_QUEUE_MIN_REPOST)
- queue->entry_repost = LPFC_QUEUE_MIN_REPOST;
queue->phba = phba;
+ /* entry_repost will be set during q creation */
+
return queue;
out_fail:
lpfc_sli4_queue_free(queue);
status = -ENXIO;
eq->host_index = 0;
eq->hba_index = 0;
+ eq->entry_repost = LPFC_EQ_REPOST;
mempool_free(mbox, phba->mbox_mem_pool);
return status;
default:
lpfc_printf_log(phba, KERN_ERR, LOG_SLI,
"0361 Unsupported CQ count: "
- "entry cnt %d sz %d pg cnt %d repost %d\n",
+ "entry cnt %d sz %d pg cnt %d\n",
cq->entry_count, cq->entry_size,
- cq->page_count, cq->entry_repost);
+ cq->page_count);
if (cq->entry_count < 256) {
status = -EINVAL;
goto out;
cq->assoc_qid = eq->queue_id;
cq->host_index = 0;
cq->hba_index = 0;
+ cq->entry_repost = LPFC_CQ_REPOST;
out:
mempool_free(mbox, phba->mbox_mem_pool);
cq->assoc_qid = eq->queue_id;
cq->host_index = 0;
cq->hba_index = 0;
+ cq->entry_repost = LPFC_CQ_REPOST;
rc = 0;
list_for_each_entry(dmabuf, &cq->page_list, list) {
mq->subtype = subtype;
mq->host_index = 0;
mq->hba_index = 0;
+ mq->entry_repost = LPFC_MQ_REPOST;
/* link the mq onto the parent cq child list */
list_add_tail(&mq->list, &cq->child_list);
return status;
}
-/**
- * lpfc_rq_adjust_repost - Adjust entry_repost for an RQ
- * @phba: HBA structure that indicates port to create a queue on.
- * @rq: The queue structure to use for the receive queue.
- * @qno: The associated HBQ number
- *
- *
- * For SLI4 we need to adjust the RQ repost value based on
- * the number of buffers that are initially posted to the RQ.
- */
-void
-lpfc_rq_adjust_repost(struct lpfc_hba *phba, struct lpfc_queue *rq, int qno)
-{
- uint32_t cnt;
-
- /* sanity check on queue memory */
- if (!rq)
- return;
- cnt = lpfc_hbq_defs[qno]->entry_count;
-
- /* Recalc repost for RQs based on buffers initially posted */
- cnt = (cnt >> 3);
- if (cnt < LPFC_QUEUE_MIN_REPOST)
- cnt = LPFC_QUEUE_MIN_REPOST;
-
- rq->entry_repost = cnt;
-}
-
/**
* lpfc_rq_create - Create a Receive Queue on the HBA
* @phba: HBA structure that indicates port to create a queue on.
hrq->subtype = subtype;
hrq->host_index = 0;
hrq->hba_index = 0;
+ hrq->entry_repost = LPFC_RQ_REPOST;
/* now create the data queue */
lpfc_sli4_config(phba, mbox, LPFC_MBOX_SUBSYSTEM_FCOE,
if (phba->sli4_hba.pc_sli4_params.rqv == LPFC_Q_CREATE_VERSION_1) {
bf_set(lpfc_rq_context_rqe_count_1,
&rq_create->u.request.context, hrq->entry_count);
- rq_create->u.request.context.buffer_size = LPFC_DATA_BUF_SIZE;
+ if (subtype == LPFC_NVMET)
+ rq_create->u.request.context.buffer_size =
+ LPFC_NVMET_DATA_BUF_SIZE;
+ else
+ rq_create->u.request.context.buffer_size =
+ LPFC_DATA_BUF_SIZE;
bf_set(lpfc_rq_context_rqe_size, &rq_create->u.request.context,
LPFC_RQE_SIZE_8);
bf_set(lpfc_rq_context_page_size, &rq_create->u.request.context,
LPFC_RQ_RING_SIZE_4096);
break;
}
- bf_set(lpfc_rq_context_buf_size, &rq_create->u.request.context,
- LPFC_DATA_BUF_SIZE);
+ if (subtype == LPFC_NVMET)
+ bf_set(lpfc_rq_context_buf_size,
+ &rq_create->u.request.context,
+ LPFC_NVMET_DATA_BUF_SIZE);
+ else
+ bf_set(lpfc_rq_context_buf_size,
+ &rq_create->u.request.context,
+ LPFC_DATA_BUF_SIZE);
}
bf_set(lpfc_rq_context_cq_id, &rq_create->u.request.context,
cq->queue_id);
drq->subtype = subtype;
drq->host_index = 0;
drq->hba_index = 0;
+ drq->entry_repost = LPFC_RQ_REPOST;
/* link the header and data RQs onto the parent cq child list */
list_add_tail(&hrq->list, &cq->child_list);
cq->queue_id);
bf_set(lpfc_rq_context_data_size,
&rq_create->u.request.context,
- LPFC_DATA_BUF_SIZE);
+ LPFC_NVMET_DATA_BUF_SIZE);
bf_set(lpfc_rq_context_hdr_size,
&rq_create->u.request.context,
LPFC_HDR_BUF_SIZE);
hrq->subtype = subtype;
hrq->host_index = 0;
hrq->hba_index = 0;
+ hrq->entry_repost = LPFC_RQ_REPOST;
drq->db_format = LPFC_DB_RING_FORMAT;
drq->db_regaddr = phba->sli4_hba.RQDBregaddr;
drq->subtype = subtype;
drq->host_index = 0;
drq->hba_index = 0;
+ drq->entry_repost = LPFC_RQ_REPOST;
list_add_tail(&hrq->list, &cq->child_list);
list_add_tail(&drq->list, &cq->child_list);
struct fc_vft_header *fc_vft_hdr;
uint32_t *header = (uint32_t *) fc_hdr;
+#define FC_RCTL_MDS_DIAGS 0xF4
+
switch (fc_hdr->fh_r_ctl) {
case FC_RCTL_DD_UNCAT: /* uncategorized information */
case FC_RCTL_DD_SOL_DATA: /* solicited data */
case FC_RCTL_F_BSY: /* fabric busy to data frame */
case FC_RCTL_F_BSYL: /* fabric busy to link control frame */
case FC_RCTL_LCR: /* link credit reset */
+ case FC_RCTL_MDS_DIAGS: /* MDS Diagnostics */
case FC_RCTL_END: /* end */
break;
case FC_RCTL_VFTH: /* Virtual Fabric tagging Header */
default:
goto drop;
}
+
+#define FC_TYPE_VENDOR_UNIQUE 0xFF
+
switch (fc_hdr->fh_type) {
case FC_TYPE_BLS:
case FC_TYPE_ELS:
case FC_TYPE_FCP:
case FC_TYPE_CT:
case FC_TYPE_NVME:
+ case FC_TYPE_VENDOR_UNIQUE:
break;
case FC_TYPE_IP:
case FC_TYPE_ILS:
lpfc_printf_log(phba, KERN_INFO, LOG_ELS,
"2538 Received frame rctl:%s (x%x), type:%s (x%x), "
"frame Data:%08x %08x %08x %08x %08x %08x %08x\n",
+ (fc_hdr->fh_r_ctl == FC_RCTL_MDS_DIAGS) ? "MDS Diags" :
lpfc_rctl_names[fc_hdr->fh_r_ctl], fc_hdr->fh_r_ctl,
- lpfc_type_names[fc_hdr->fh_type], fc_hdr->fh_type,
- be32_to_cpu(header[0]), be32_to_cpu(header[1]),
- be32_to_cpu(header[2]), be32_to_cpu(header[3]),
- be32_to_cpu(header[4]), be32_to_cpu(header[5]),
- be32_to_cpu(header[6]));
+ (fc_hdr->fh_type == FC_TYPE_VENDOR_UNIQUE) ?
+ "Vendor Unique" : lpfc_type_names[fc_hdr->fh_type],
+ fc_hdr->fh_type, be32_to_cpu(header[0]),
+ be32_to_cpu(header[1]), be32_to_cpu(header[2]),
+ be32_to_cpu(header[3]), be32_to_cpu(header[4]),
+ be32_to_cpu(header[5]), be32_to_cpu(header[6]));
return 0;
drop:
lpfc_printf_log(phba, KERN_WARNING, LOG_ELS,
lpfc_sli_release_iocbq(phba, iocbq);
}
+static void
+lpfc_sli4_mds_loopback_cmpl(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
+ struct lpfc_iocbq *rspiocb)
+{
+ struct lpfc_dmabuf *pcmd = cmdiocb->context2;
+
+ if (pcmd && pcmd->virt)
+ pci_pool_free(phba->lpfc_drb_pool, pcmd->virt, pcmd->phys);
+ kfree(pcmd);
+ lpfc_sli_release_iocbq(phba, cmdiocb);
+}
+
+static void
+lpfc_sli4_handle_mds_loopback(struct lpfc_vport *vport,
+ struct hbq_dmabuf *dmabuf)
+{
+ struct fc_frame_header *fc_hdr;
+ struct lpfc_hba *phba = vport->phba;
+ struct lpfc_iocbq *iocbq = NULL;
+ union lpfc_wqe *wqe;
+ struct lpfc_dmabuf *pcmd = NULL;
+ uint32_t frame_len;
+ int rc;
+
+ fc_hdr = (struct fc_frame_header *)dmabuf->hbuf.virt;
+ frame_len = bf_get(lpfc_rcqe_length, &dmabuf->cq_event.cqe.rcqe_cmpl);
+
+ /* Send the received frame back */
+ iocbq = lpfc_sli_get_iocbq(phba);
+ if (!iocbq)
+ goto exit;
+
+ /* Allocate buffer for command payload */
+ pcmd = kmalloc(sizeof(struct lpfc_dmabuf), GFP_KERNEL);
+ if (pcmd)
+ pcmd->virt = pci_pool_alloc(phba->lpfc_drb_pool, GFP_KERNEL,
+ &pcmd->phys);
+ if (!pcmd || !pcmd->virt)
+ goto exit;
+
+ INIT_LIST_HEAD(&pcmd->list);
+
+ /* copyin the payload */
+ memcpy(pcmd->virt, dmabuf->dbuf.virt, frame_len);
+
+ /* fill in BDE's for command */
+ iocbq->iocb.un.xseq64.bdl.addrHigh = putPaddrHigh(pcmd->phys);
+ iocbq->iocb.un.xseq64.bdl.addrLow = putPaddrLow(pcmd->phys);
+ iocbq->iocb.un.xseq64.bdl.bdeFlags = BUFF_TYPE_BDE_64;
+ iocbq->iocb.un.xseq64.bdl.bdeSize = frame_len;
+
+ iocbq->context2 = pcmd;
+ iocbq->vport = vport;
+ iocbq->iocb_flag &= ~LPFC_FIP_ELS_ID_MASK;
+ iocbq->iocb_flag |= LPFC_USE_FCPWQIDX;
+
+ /*
+ * Setup rest of the iocb as though it were a WQE
+ * Build the SEND_FRAME WQE
+ */
+ wqe = (union lpfc_wqe *)&iocbq->iocb;
+
+ wqe->send_frame.frame_len = frame_len;
+ wqe->send_frame.fc_hdr_wd0 = be32_to_cpu(*((uint32_t *)fc_hdr));
+ wqe->send_frame.fc_hdr_wd1 = be32_to_cpu(*((uint32_t *)fc_hdr + 1));
+ wqe->send_frame.fc_hdr_wd2 = be32_to_cpu(*((uint32_t *)fc_hdr + 2));
+ wqe->send_frame.fc_hdr_wd3 = be32_to_cpu(*((uint32_t *)fc_hdr + 3));
+ wqe->send_frame.fc_hdr_wd4 = be32_to_cpu(*((uint32_t *)fc_hdr + 4));
+ wqe->send_frame.fc_hdr_wd5 = be32_to_cpu(*((uint32_t *)fc_hdr + 5));
+
+ iocbq->iocb.ulpCommand = CMD_SEND_FRAME;
+ iocbq->iocb.ulpLe = 1;
+ iocbq->iocb_cmpl = lpfc_sli4_mds_loopback_cmpl;
+ rc = lpfc_sli_issue_iocb(phba, LPFC_ELS_RING, iocbq, 0);
+ if (rc == IOCB_ERROR)
+ goto exit;
+
+ lpfc_in_buf_free(phba, &dmabuf->dbuf);
+ return;
+
+exit:
+ lpfc_printf_log(phba, KERN_WARNING, LOG_SLI,
+ "2023 Unable to process MDS loopback frame\n");
+ if (pcmd && pcmd->virt)
+ pci_pool_free(phba->lpfc_drb_pool, pcmd->virt, pcmd->phys);
+ kfree(pcmd);
+ lpfc_sli_release_iocbq(phba, iocbq);
+ lpfc_in_buf_free(phba, &dmabuf->dbuf);
+}
+
/**
* lpfc_sli4_handle_received_buffer - Handle received buffers from firmware
* @phba: Pointer to HBA context object.
fcfi = bf_get(lpfc_rcqe_fcf_id,
&dmabuf->cq_event.cqe.rcqe_cmpl);
+ if (fc_hdr->fh_r_ctl == 0xF4 && fc_hdr->fh_type == 0xFF) {
+ vport = phba->pport;
+ /* Handle MDS Loopback frames */
+ lpfc_sli4_handle_mds_loopback(vport, dmabuf);
+ return;
+ }
+
/* d_id this frame is directed to */
did = sli4_did_from_fc_hdr(fc_hdr);
"status x%x add_status x%x, mbx status x%x\n",
shdr_status, shdr_add_status, rc);
rc = -ENXIO;
+ } else {
+ /*
+ * The next_rpi stores the next logical module-64 rpi value used
+ * to post physical rpis in subsequent rpi postings.
+ */
+ spin_lock_irq(&phba->hbalock);
+ phba->sli4_hba.next_rpi = rpi_page->next_rpi;
+ spin_unlock_irq(&phba->hbalock);
}
return rc;
}
spin_lock_irqsave(&pring->ring_lock, iflags);
ctxp = pwqe->context2;
- sglq = ctxp->rqb_buffer->sglq;
+ sglq = ctxp->ctxbuf->sglq;
if (pwqe->sli4_xritag == NO_XRI) {
pwqe->sli4_lxritag = sglq->sli4_lxritag;
pwqe->sli4_xritag = sglq->sli4_xritag;
#define LPFC_XRI_EXCH_BUSY_WAIT_TMO 10000
#define LPFC_XRI_EXCH_BUSY_WAIT_T1 10
#define LPFC_XRI_EXCH_BUSY_WAIT_T2 30000
-#define LPFC_RELEASE_NOTIFICATION_INTERVAL 32
#define LPFC_RPI_LOW_WATER_MARK 10
#define LPFC_UNREG_FCF 1
uint32_t entry_count; /* Number of entries to support on the queue */
uint32_t entry_size; /* Size of each queue entry. */
uint32_t entry_repost; /* Count of entries before doorbell is rung */
-#define LPFC_QUEUE_MIN_REPOST 8
+#define LPFC_EQ_REPOST 8
+#define LPFC_MQ_REPOST 8
+#define LPFC_CQ_REPOST 64
+#define LPFC_RQ_REPOST 64
+#define LPFC_RELEASE_NOTIFICATION_INTERVAL 32 /* For WQs */
uint32_t queue_id; /* Queue ID assigned by the hardware */
uint32_t assoc_qid; /* Queue ID associated with, for CQ/WQ/MQ */
uint32_t page_count; /* Number of pages allocated for this queue */
/* defines for RQ stats */
#define RQ_no_posted_buf q_cnt_1
#define RQ_no_buf_found q_cnt_2
-#define RQ_buf_trunc q_cnt_3
+#define RQ_buf_posted q_cnt_3
#define RQ_rcv_buf q_cnt_4
uint64_t isr_timestamp;
uint16_t scsi_xri_start;
uint16_t els_xri_cnt;
uint16_t nvmet_xri_cnt;
+ uint16_t nvmet_ctx_cnt;
+ uint16_t nvmet_io_wait_cnt;
+ uint16_t nvmet_io_wait_total;
struct list_head lpfc_els_sgl_list;
struct list_head lpfc_abts_els_sgl_list;
struct list_head lpfc_nvmet_sgl_list;
struct list_head lpfc_abts_nvmet_ctx_list;
struct list_head lpfc_abts_scsi_buf_list;
struct list_head lpfc_abts_nvme_buf_list;
+ struct list_head lpfc_nvmet_ctx_list;
+ struct list_head lpfc_nvmet_io_wait_list;
struct lpfc_sglq **lpfc_sglq_active_list;
struct list_head lpfc_rpi_hdr_list;
unsigned long *rpi_bmask;
spinlock_t abts_scsi_buf_list_lock; /* list of aborted SCSI IOs */
spinlock_t sgl_list_lock; /* list of aborted els IOs */
spinlock_t nvmet_io_lock;
+ spinlock_t nvmet_io_wait_lock; /* IOs waiting for ctx resources */
uint32_t physical_port;
/* CPU to vector mapping information */
uint16_t num_online_cpu;
uint16_t num_present_cpu;
uint16_t curr_disp_cpu;
-
- uint16_t nvmet_mrq_post_idx;
};
enum lpfc_sge_type {
struct lpfc_dmabuf *dmabuf;
uint32_t page_count;
uint32_t start_rpi;
+ uint16_t next_rpi;
};
struct lpfc_rsrc_blks {
int lpfc_mrq_create(struct lpfc_hba *phba, struct lpfc_queue **hrqp,
struct lpfc_queue **drqp, struct lpfc_queue **cqp,
uint32_t subtype);
-void lpfc_rq_adjust_repost(struct lpfc_hba *, struct lpfc_queue *, int);
int lpfc_eq_destroy(struct lpfc_hba *, struct lpfc_queue *);
int lpfc_cq_destroy(struct lpfc_hba *, struct lpfc_queue *);
int lpfc_mq_destroy(struct lpfc_hba *, struct lpfc_queue *);
* included with this package. *
*******************************************************************/
-#define LPFC_DRIVER_VERSION "11.2.0.12"
+#define LPFC_DRIVER_VERSION "11.2.0.14"
#define LPFC_DRIVER_NAME "lpfc"
/* Used for SLI 2/3 */
pmcraid_err("couldn't build passthrough ioadls\n");
goto out_free_cmd;
}
- } else if (request_size < 0) {
- rc = -EINVAL;
- goto out_free_cmd;
}
/* If data is being written into the device, copy the data from user
uint16_t task_id;
uint32_t port_id; /* Remote port fabric ID */
int lun;
- char op; /* SCSI CDB */
+ unsigned char op; /* SCSI CDB */
uint8_t lba[4];
unsigned int bufflen; /* SCSI buffer length */
unsigned int sg_count; /* Number of SG elements */
did = fcport->rdata->ids.port_id;
sid = fcport->sid;
- __fc_fill_fc_hdr(fc_hdr, FC_RCTL_ELS_REQ, sid, did,
+ __fc_fill_fc_hdr(fc_hdr, FC_RCTL_ELS_REQ, did, sid,
FC_TYPE_ELS, FC_FC_FIRST_SEQ | FC_FC_END_SEQ |
FC_FC_SEQ_INIT, 0);
slowpath_params.drv_minor = QEDF_DRIVER_MINOR_VER;
slowpath_params.drv_rev = QEDF_DRIVER_REV_VER;
slowpath_params.drv_eng = QEDF_DRIVER_ENG_VER;
- memcpy(slowpath_params.name, "qedf", QED_DRV_VER_STR_SIZE);
+ strncpy(slowpath_params.name, "qedf", QED_DRV_VER_STR_SIZE);
rc = qed_ops->common->slowpath_start(qedf->cdev, &slowpath_params);
if (rc) {
QEDF_ERR(&(qedf->dbg_ctx), "Cannot start slowpath.\n");
struct scsi_device *sdev;
list_for_each_entry(sdev, &shost->__devices, siblings) {
+ if (sdev->sdev_state == SDEV_DEL)
+ continue;
if (sdev->channel == channel && sdev->id == id &&
sdev->lun ==lun)
return sdev;
#include <scsi/scsi_driver.h>
#include <scsi/scsi_eh.h>
#include <scsi/scsi_host.h>
+#include <scsi/scsi_transport.h> /* __scsi_init_queue() */
#include <scsi/scsi_dh.h>
#include <trace/events/scsi.h>
/* zero out the cmd, except for the embedded scsi_request */
memset((char *)cmd + sizeof(cmd->req), 0,
- sizeof(*cmd) - sizeof(cmd->req));
+ sizeof(*cmd) - sizeof(cmd->req) + shost->hostt->cmd_size);
req->special = cmd;
struct scsi_disk *sdkp = scsi_disk(rq->rq_disk);
u64 sector = blk_rq_pos(rq) >> (ilog2(sdp->sector_size) - 9);
u32 nr_sectors = blk_rq_sectors(rq) >> (ilog2(sdp->sector_size) - 9);
+ int ret;
if (!(rq->cmd_flags & REQ_NOUNMAP)) {
switch (sdkp->zeroing_mode) {
case SD_ZERO_WS16_UNMAP:
- return sd_setup_write_same16_cmnd(cmd, true);
+ ret = sd_setup_write_same16_cmnd(cmd, true);
+ goto out;
case SD_ZERO_WS10_UNMAP:
- return sd_setup_write_same10_cmnd(cmd, true);
+ ret = sd_setup_write_same10_cmnd(cmd, true);
+ goto out;
}
}
if (sdp->no_write_same)
return BLKPREP_INVALID;
+
if (sdkp->ws16 || sector > 0xffffffff || nr_sectors > 0xffff)
- return sd_setup_write_same16_cmnd(cmd, false);
- return sd_setup_write_same10_cmnd(cmd, false);
+ ret = sd_setup_write_same16_cmnd(cmd, false);
+ else
+ ret = sd_setup_write_same10_cmnd(cmd, false);
+
+out:
+ if (sd_is_zoned(sdkp) && ret == BLKPREP_OK)
+ return sd_zbc_write_lock_zone(cmd);
+
+ return ret;
}
static void sd_config_write_same(struct scsi_disk *sdkp)
rq->__data_len = sdp->sector_size;
ret = scsi_init_io(cmd);
rq->__data_len = nr_bytes;
+
+ if (sd_is_zoned(sdkp) && ret != BLKPREP_OK)
+ sd_zbc_write_unlock_zone(cmd);
+
return ret;
}
return retval;
}
-static int sd_sync_cache(struct scsi_disk *sdkp)
+static int sd_sync_cache(struct scsi_disk *sdkp, struct scsi_sense_hdr *sshdr)
{
int retries, res;
struct scsi_device *sdp = sdkp->device;
const int timeout = sdp->request_queue->rq_timeout
* SD_FLUSH_TIMEOUT_MULTIPLIER;
- struct scsi_sense_hdr sshdr;
+ struct scsi_sense_hdr my_sshdr;
if (!scsi_device_online(sdp))
return -ENODEV;
+ /* caller might not be interested in sense, but we need it */
+ if (!sshdr)
+ sshdr = &my_sshdr;
+
for (retries = 3; retries > 0; --retries) {
unsigned char cmd[10] = { 0 };
* Leave the rest of the command zero to indicate
* flush everything.
*/
- res = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, &sshdr,
+ res = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, sshdr,
timeout, SD_MAX_RETRIES, 0, RQF_PM, NULL);
if (res == 0)
break;
sd_print_result(sdkp, "Synchronize Cache(10) failed", res);
if (driver_byte(res) & DRIVER_SENSE)
- sd_print_sense_hdr(sdkp, &sshdr);
+ sd_print_sense_hdr(sdkp, sshdr);
+
/* we need to evaluate the error return */
- if (scsi_sense_valid(&sshdr) &&
- (sshdr.asc == 0x3a || /* medium not present */
- sshdr.asc == 0x20)) /* invalid command */
+ if (scsi_sense_valid(sshdr) &&
+ (sshdr->asc == 0x3a || /* medium not present */
+ sshdr->asc == 0x20)) /* invalid command */
/* this is no error here */
return 0;
if (sdkp->WCE && sdkp->media_present) {
sd_printk(KERN_NOTICE, sdkp, "Synchronizing SCSI cache\n");
- sd_sync_cache(sdkp);
+ sd_sync_cache(sdkp, NULL);
}
if (system_state != SYSTEM_RESTART && sdkp->device->manage_start_stop) {
static int sd_suspend_common(struct device *dev, bool ignore_stop_errors)
{
struct scsi_disk *sdkp = dev_get_drvdata(dev);
+ struct scsi_sense_hdr sshdr;
int ret = 0;
if (!sdkp) /* E.g.: runtime suspend following sd_remove() */
if (sdkp->WCE && sdkp->media_present) {
sd_printk(KERN_NOTICE, sdkp, "Synchronizing SCSI cache\n");
- ret = sd_sync_cache(sdkp);
+ ret = sd_sync_cache(sdkp, &sshdr);
+
if (ret) {
/* ignore OFFLINE device */
if (ret == -ENODEV)
- ret = 0;
- goto done;
+ return 0;
+
+ if (!scsi_sense_valid(&sshdr) ||
+ sshdr.sense_key != ILLEGAL_REQUEST)
+ return ret;
+
+ /*
+ * sshdr.sense_key == ILLEGAL_REQUEST means this drive
+ * doesn't support sync. There's not much to do and
+ * suspend shouldn't fail.
+ */
+ ret = 0;
}
}
ret = 0;
}
-done:
return ret;
}
if ((1 == resp->done) && (!resp->sg_io_owned) &&
((-1 == pack_id) || (resp->header.pack_id == pack_id))) {
resp->done = 2; /* guard against other readers */
- break;
+ write_unlock_irqrestore(&sfp->rq_list_lock, iflags);
+ return resp;
}
}
write_unlock_irqrestore(&sfp->rq_list_lock, iflags);
- return resp;
+ return NULL;
}
/* always adds to end of list */
ufshcd_add_spm_lvl_sysfs_nodes(hba);
}
+static inline void ufshcd_remove_sysfs_nodes(struct ufs_hba *hba)
+{
+ device_remove_file(hba->dev, &hba->rpm_lvl_attr);
+ device_remove_file(hba->dev, &hba->spm_lvl_attr);
+}
+
/**
* ufshcd_shutdown - shutdown routine
* @hba: per adapter instance
*/
void ufshcd_remove(struct ufs_hba *hba)
{
+ ufshcd_remove_sysfs_nodes(hba);
scsi_remove_host(hba->host);
/* disable interrupts */
ufshcd_disable_intr(hba, hba->intr_mask);
{ .compatible = "brcm,bcm7420-sun-top-ctrl", },
{ .compatible = "brcm,bcm7425-sun-top-ctrl", },
{ .compatible = "brcm,bcm7429-sun-top-ctrl", },
- { .compatible = "brcm,bcm7425-sun-top-ctrl", },
+ { .compatible = "brcm,bcm7435-sun-top-ctrl", },
{ .compatible = "brcm,brcmstb-sun-top-ctrl", },
{ }
};
config IMX7_PM_DOMAINS
bool "i.MX7 PM domains"
- select PM_GENERIC_DOMAINS
depends on SOC_IMX7D || (COMPILE_TEST && OF)
+ depends on PM
+ select PM_GENERIC_DOMAINS
default y if SOC_IMX7D
endmenu
* @name: slave channel name
* @config: dma configuration parameters
*
- * Returns pointer to appropriate DMA channel on success or NULL.
+ * Returns pointer to appropriate DMA channel on success or error.
*/
void *knav_dma_open_channel(struct device *dev, const char *name,
struct knav_dma_cfg *config)
+++ /dev/null
-Ion Memory Manager
-
-Ion is a memory manager that allows for sharing of buffers via dma-buf.
-Ion allows for different types of allocation via an abstraction called
-a 'heap'. A heap represents a specific type of memory. Each heap has
-a different type. There can be multiple instances of the same heap
-type.
-
-Specific heap instances are tied to heap IDs. Heap IDs are not to be specified
-in the devicetree.
-
-Required properties for Ion
-
-- compatible: "linux,ion" PLUS a compatible property for the device
-
-All child nodes of a linux,ion node are interpreted as heaps
-
-required properties for heaps
-
-- compatible: compatible string for a heap type PLUS a compatible property
-for the specific instance of the heap. Current heap types
--- linux,ion-heap-system
--- linux,ion-heap-system-contig
--- linux,ion-heap-carveout
--- linux,ion-heap-chunk
--- linux,ion-heap-dma
--- linux,ion-heap-custom
-
-Optional properties
-- memory-region: A phandle to a memory region. Required for DMA heap type
-(see reserved-memory.txt for details on the reservation)
-
-Example:
-
- ion {
- compatbile = "hisilicon,ion", "linux,ion";
-
- ion-system-heap {
- compatbile = "hisilicon,system-heap", "linux,ion-heap-system"
- };
-
- ion-camera-region {
- compatible = "hisilicon,camera-heap", "linux,ion-heap-dma"
- memory-region = <&camera_region>;
- };
-
- ion-fb-region {
- compatbile = "hisilicon,fb-heap", "linux,ion-heap-dma"
- memory-region = <&fb_region>;
- };
- }
rc = ssi_power_mgr_runtime_get(&drvdata->plat_dev->dev);
if (rc != 0) {
SSI_LOG_ERR("ssi_power_mgr_runtime_get returned %x\n",rc);
- spin_unlock_bh(&req_mgr_h->hw_lock);
return rc;
}
#endif
config FSL_DPAA2_ETH
tristate "Freescale DPAA2 Ethernet"
depends on FSL_DPAA2 && FSL_MC_DPIO
+ depends on NETDEVICES && ETHERNET
---help---
Ethernet driver for Freescale DPAA2 SoCs, using the
Freescale MC bus driver
switch (variable) {
case HW_VAR_BSSID:
- rtl92e_writel(dev, BSSIDR, ((u32 *)(val))[0]);
- rtl92e_writew(dev, BSSIDR+2, ((u16 *)(val+2))[0]);
+ /* BSSIDR 2 byte alignment */
+ rtl92e_writew(dev, BSSIDR, *(u16 *)val);
+ rtl92e_writel(dev, BSSIDR + 2, *(u32 *)(val + 2));
break;
case HW_VAR_MEDIA_STATUS:
struct r8192_priv *priv = rtllib_priv(dev);
RT_TRACE(COMP_INIT, "===========>%s()\n", __func__);
- curCR = rtl92e_readl(dev, EPROM_CMD);
+ curCR = rtl92e_readw(dev, EPROM_CMD);
RT_TRACE(COMP_INIT, "read from Reg Cmd9346CR(%x):%x\n", EPROM_CMD,
curCR);
priv->epromtype = (curCR & EPROM_CMD_9356SEL) ? EEPROM_93C56 :
rtl92e_config_rate(dev, &rate_config);
priv->dot11CurrentPreambleMode = PREAMBLE_AUTO;
priv->basic_rate = rate_config &= 0x15f;
- rtl92e_writel(dev, BSSIDR, ((u32 *)net->bssid)[0]);
- rtl92e_writew(dev, BSSIDR+4, ((u16 *)net->bssid)[2]);
+ rtl92e_writew(dev, BSSIDR, *(u16 *)net->bssid);
+ rtl92e_writel(dev, BSSIDR + 2, *(u32 *)(net->bssid + 2));
if (priv->rtllib->iw_mode == IW_MODE_ADHOC) {
rtl92e_writew(dev, ATIMWND, 2);
struct cb_desc *cb_desc, struct sk_buff *skb)
{
struct r8192_priv *priv = rtllib_priv(dev);
- dma_addr_t mapping = pci_map_single(priv->pdev, skb->data, skb->len,
- PCI_DMA_TODEVICE);
+ dma_addr_t mapping;
struct tx_fwinfo_8190pci *pTxFwInfo;
pTxFwInfo = (struct tx_fwinfo_8190pci *)skb->data;
pTxFwInfo->Short = _rtl92e_query_is_short(pTxFwInfo->TxHT,
pTxFwInfo->TxRate, cb_desc);
- if (pci_dma_mapping_error(priv->pdev, mapping))
- netdev_err(dev, "%s(): DMA Mapping error\n", __func__);
if (cb_desc->bAMPDUEnable) {
pTxFwInfo->AllowAggregation = 1;
pTxFwInfo->RxMF = cb_desc->ampdu_factor;
}
memset((u8 *)pdesc, 0, 12);
+
+ mapping = pci_map_single(priv->pdev, skb->data, skb->len,
+ PCI_DMA_TODEVICE);
+ if (pci_dma_mapping_error(priv->pdev, mapping)) {
+ netdev_err(dev, "%s(): DMA Mapping error\n", __func__);
+ return;
+ }
+
pdesc->LINIP = 0;
pdesc->CmdInit = 1;
pdesc->Offset = sizeof(struct tx_fwinfo_8190pci) + 8;
pTsCommonInfo->TClasNum = TCLAS_Num;
}
-static bool IsACValid(unsigned int tid)
-{
- return tid < 7;
-}
-
bool GetTs(struct rtllib_device *ieee, struct ts_common_info **ppTS,
u8 *Addr, u8 TID, enum tr_select TxRxSelect, bool bAddNewTs)
{
if (ieee->current_network.qos_data.supported == 0) {
UP = 0;
} else {
- if (!IsACValid(TID)) {
- netdev_warn(ieee->dev, "%s(): TID(%d) is not valid\n",
- __func__, TID);
- return false;
- }
-
switch (TID) {
case 0:
case 3:
case 7:
UP = 7;
break;
+ default:
+ netdev_warn(ieee->dev, "%s(): TID(%d) is not valid\n",
+ __func__, TID);
+ return false;
}
}
pwdev_priv->power_mgmt = true;
else
pwdev_priv->power_mgmt = false;
- kfree((u8 *)wdev);
return ret;
#define FUSB302_RESUME_RETRY 10
#define FUSB302_RESUME_RETRY_SLEEP 50
-static int fusb302_i2c_write(struct fusb302_chip *chip,
- u8 address, u8 data)
+
+static bool fusb302_is_suspended(struct fusb302_chip *chip)
{
int retry_cnt;
- int ret = 0;
- atomic_set(&chip->i2c_busy, 1);
for (retry_cnt = 0; retry_cnt < FUSB302_RESUME_RETRY; retry_cnt++) {
if (atomic_read(&chip->pm_suspend)) {
- pr_err("fusb302_i2c: pm suspend, retry %d/%d\n",
- retry_cnt + 1, FUSB302_RESUME_RETRY);
+ dev_err(chip->dev, "i2c: pm suspend, retry %d/%d\n",
+ retry_cnt + 1, FUSB302_RESUME_RETRY);
msleep(FUSB302_RESUME_RETRY_SLEEP);
} else {
- break;
+ return false;
}
}
+
+ return true;
+}
+
+static int fusb302_i2c_write(struct fusb302_chip *chip,
+ u8 address, u8 data)
+{
+ int ret = 0;
+
+ atomic_set(&chip->i2c_busy, 1);
+
+ if (fusb302_is_suspended(chip)) {
+ atomic_set(&chip->i2c_busy, 0);
+ return -ETIMEDOUT;
+ }
+
ret = i2c_smbus_write_byte_data(chip->i2c_client, address, data);
if (ret < 0)
fusb302_log(chip, "cannot write 0x%02x to 0x%02x, ret=%d",
static int fusb302_i2c_block_write(struct fusb302_chip *chip, u8 address,
u8 length, const u8 *data)
{
- int retry_cnt;
int ret = 0;
if (length <= 0)
return ret;
atomic_set(&chip->i2c_busy, 1);
- for (retry_cnt = 0; retry_cnt < FUSB302_RESUME_RETRY; retry_cnt++) {
- if (atomic_read(&chip->pm_suspend)) {
- pr_err("fusb302_i2c: pm suspend, retry %d/%d\n",
- retry_cnt + 1, FUSB302_RESUME_RETRY);
- msleep(FUSB302_RESUME_RETRY_SLEEP);
- } else {
- break;
- }
+
+ if (fusb302_is_suspended(chip)) {
+ atomic_set(&chip->i2c_busy, 0);
+ return -ETIMEDOUT;
}
+
ret = i2c_smbus_write_i2c_block_data(chip->i2c_client, address,
length, data);
if (ret < 0)
static int fusb302_i2c_read(struct fusb302_chip *chip,
u8 address, u8 *data)
{
- int retry_cnt;
int ret = 0;
atomic_set(&chip->i2c_busy, 1);
- for (retry_cnt = 0; retry_cnt < FUSB302_RESUME_RETRY; retry_cnt++) {
- if (atomic_read(&chip->pm_suspend)) {
- pr_err("fusb302_i2c: pm suspend, retry %d/%d\n",
- retry_cnt + 1, FUSB302_RESUME_RETRY);
- msleep(FUSB302_RESUME_RETRY_SLEEP);
- } else {
- break;
- }
+
+ if (fusb302_is_suspended(chip)) {
+ atomic_set(&chip->i2c_busy, 0);
+ return -ETIMEDOUT;
}
+
ret = i2c_smbus_read_byte_data(chip->i2c_client, address);
*data = (u8)ret;
if (ret < 0)
static int fusb302_i2c_block_read(struct fusb302_chip *chip, u8 address,
u8 length, u8 *data)
{
- int retry_cnt;
int ret = 0;
if (length <= 0)
return ret;
atomic_set(&chip->i2c_busy, 1);
- for (retry_cnt = 0; retry_cnt < FUSB302_RESUME_RETRY; retry_cnt++) {
- if (atomic_read(&chip->pm_suspend)) {
- pr_err("fusb302_i2c: pm suspend, retry %d/%d\n",
- retry_cnt + 1, FUSB302_RESUME_RETRY);
- msleep(FUSB302_RESUME_RETRY_SLEEP);
- } else {
- break;
- }
+
+ if (fusb302_is_suspended(chip)) {
+ atomic_set(&chip->i2c_busy, 0);
+ return -ETIMEDOUT;
}
+
ret = i2c_smbus_read_i2c_block_data(chip->i2c_client, address,
length, data);
if (ret < 0) {
fusb302_log(chip, "cannot block read 0x%02x, len=%d, ret=%d",
address, length, ret);
- return ret;
+ goto done;
}
if (ret != length) {
fusb302_log(chip, "only read %d/%d bytes from 0x%02x",
ret, length, address);
- return -EIO;
+ ret = -EIO;
}
+
+done:
atomic_set(&chip->i2c_busy, 0);
return ret;
ret = fusb302_i2c_read(chip, FUSB_REG_STATUS0, &data);
if (ret < 0)
return ret;
- chip->vbus_present = !!(FUSB_REG_STATUS0 & FUSB_REG_STATUS0_VBUSOK);
+ chip->vbus_present = !!(data & FUSB_REG_STATUS0_VBUSOK);
ret = fusb302_i2c_read(chip, FUSB_REG_DEVICE_ID, &data);
if (ret < 0)
return ret;
buf[pos++] = FUSB302_TKN_SYNC1;
buf[pos++] = FUSB302_TKN_SYNC2;
- len = pd_header_cnt(msg->header) * 4;
+ len = pd_header_cnt_le(msg->header) * 4;
/* plug 2 for header */
len += 2;
if (len > 0x1F) {
(u8 *)&msg->header);
if (ret < 0)
return ret;
- len = pd_header_cnt(msg->header) * 4;
+ len = pd_header_cnt_le(msg->header) * 4;
/* add 4 to length to include the CRC */
if (len > PD_MAX_PAYLOAD * 4) {
fusb302_log(chip, "PD message too long %d", len);
if (ret < 0) {
fusb302_log(chip,
"cannot set GPIO Int_N to input, ret=%d", ret);
- gpio_free(chip->gpio_int_n);
return ret;
}
ret = gpio_to_irq(chip->gpio_int_n);
if (ret < 0) {
fusb302_log(chip,
"cannot request IRQ for GPIO Int_N, ret=%d", ret);
- gpio_free(chip->gpio_int_n);
return ret;
}
chip->gpio_int_n_irq = ret;
{.compatible = "fcs,fusb302"},
{},
};
+MODULE_DEVICE_TABLE(of, fusb302_dt_match);
static const struct i2c_device_id fusb302_i2c_device_id[] = {
{"typec_fusb302", 0},
{},
};
+MODULE_DEVICE_TABLE(i2c, fusb302_i2c_device_id);
static const struct dev_pm_ops fusb302_pm_ops = {
.suspend = fusb302_pm_suspend,
return pd_header_type(le16_to_cpu(header));
}
+static inline unsigned int pd_header_msgid(u16 header)
+{
+ return (header >> PD_HEADER_ID_SHIFT) & PD_HEADER_ID_MASK;
+}
+
+static inline unsigned int pd_header_msgid_le(__le16 header)
+{
+ return pd_header_msgid(le16_to_cpu(header));
+}
+
#define PD_MAX_PAYLOAD 7
struct pd_message {
* VDM object is minimum of VDM header + 6 additional data objects.
*/
+#define VDO_MAX_OBJECTS 6
+#define VDO_MAX_SIZE (VDO_MAX_OBJECTS + 1)
+
/*
* VDM header
* ----------
* <5> :: reserved (SVDM), command type (UVDM)
* <4:0> :: command
*/
-#define VDO_MAX_SIZE 7
#define VDO(vid, type, custom) \
(((vid) << 16) | \
((type) << 15) | \
.max_register = 0x7F, /* 0x80 .. 0xFF are vendor defined */
};
-const struct tcpc_config tcpci_tcpc_config = {
+static const struct tcpc_config tcpci_tcpc_config = {
.type = TYPEC_PORT_DFP,
.default_role = TYPEC_SINK,
};
unsigned int hard_reset_count;
bool pd_capable;
bool explicit_contract;
+ unsigned int rx_msgid;
/* Partner capabilities/requests */
u32 sink_request;
unsigned int nr_src_pdo;
u32 snk_pdo[PDO_MAX_OBJECTS];
unsigned int nr_snk_pdo;
+ u32 snk_vdo[VDO_MAX_OBJECTS];
+ unsigned int nr_snk_vdo;
unsigned int max_snk_mv;
unsigned int max_snk_ma;
struct pd_mode_data *modep;
int rlen = 0;
u16 svid;
+ int i;
tcpm_log(port, "Rx VDM cmd 0x%x type %d cmd %d len %d",
p0, cmd_type, cmd, cnt);
case CMDT_INIT:
switch (cmd) {
case CMD_DISCOVER_IDENT:
+ /* 6.4.4.3.1: Only respond as UFP (device) */
+ if (port->data_role == TYPEC_DEVICE &&
+ port->nr_snk_vdo) {
+ for (i = 0; i < port->nr_snk_vdo; i++)
+ response[i + 1]
+ = cpu_to_le32(port->snk_vdo[i]);
+ rlen = port->nr_snk_vdo + 1;
+ }
break;
case CMD_DISCOVER_SVID:
break;
break;
case SOFT_RESET_SEND:
port->message_id = 0;
+ port->rx_msgid = -1;
if (port->pwr_role == TYPEC_SOURCE)
next_state = SRC_SEND_CAPABILITIES;
else
port->attached);
if (port->attached) {
+ enum pd_ctrl_msg_type type = pd_header_type_le(msg->header);
+ unsigned int msgid = pd_header_msgid_le(msg->header);
+
+ /*
+ * USB PD standard, 6.6.1.2:
+ * "... if MessageID value in a received Message is the
+ * same as the stored value, the receiver shall return a
+ * GoodCRC Message with that MessageID value and drop
+ * the Message (this is a retry of an already received
+ * Message). Note: this shall not apply to the Soft_Reset
+ * Message which always has a MessageID value of zero."
+ */
+ if (msgid == port->rx_msgid && type != PD_CTRL_SOFT_RESET)
+ goto done;
+ port->rx_msgid = msgid;
+
/*
* If both ends believe to be DFP/host, we have a data role
* mismatch.
}
}
+done:
mutex_unlock(&port->lock);
kfree(event);
}
}
ma = min(ma, port->max_snk_ma);
- /* XXX: Any other flags need to be set? */
- flags = 0;
+ flags = RDO_USB_COMM | RDO_NO_SUSPEND;
/* Set mismatch bit if offered power is less than operating power */
mw = ma * mv / 1000;
port->attached = false;
port->pd_capable = false;
+ /*
+ * First Rx ID should be 0; set this to a sentinel of -1 so that
+ * we can check tcpm_pd_rx_handler() if we had seen it before.
+ */
+ port->rx_msgid = -1;
+
port->tcpc->set_pd_rx(port->tcpc, false);
tcpm_init_vbus(port); /* also disables charging */
tcpm_init_vconn(port);
port->pwr_opmode = TYPEC_PWR_MODE_USB;
port->caps_count = 0;
port->message_id = 0;
+ port->rx_msgid = -1;
port->explicit_contract = false;
tcpm_set_state(port, SRC_SEND_CAPABILITIES, 0);
break;
typec_set_pwr_opmode(port->typec_port, TYPEC_PWR_MODE_USB);
port->pwr_opmode = TYPEC_PWR_MODE_USB;
port->message_id = 0;
+ port->rx_msgid = -1;
port->explicit_contract = false;
tcpm_set_state(port, SNK_DISCOVERY, 0);
break;
/* Soft_Reset states */
case SOFT_RESET:
port->message_id = 0;
+ port->rx_msgid = -1;
tcpm_pd_send_control(port, PD_CTRL_ACCEPT);
if (port->pwr_role == TYPEC_SOURCE)
tcpm_set_state(port, SRC_SEND_CAPABILITIES, 0);
break;
case SOFT_RESET_SEND:
port->message_id = 0;
+ port->rx_msgid = -1;
if (tcpm_pd_send_control(port, PD_CTRL_SOFT_RESET))
tcpm_set_state_cond(port, hard_reset_state(port), 0);
else
break;
case PR_SWAP_SRC_SNK_SOURCE_OFF:
tcpm_set_cc(port, TYPEC_CC_RD);
+ /*
+ * USB-PD standard, 6.2.1.4, Port Power Role:
+ * "During the Power Role Swap Sequence, for the initial Source
+ * Port, the Port Power Role field shall be set to Sink in the
+ * PS_RDY Message indicating that the initial Source’s power
+ * supply is turned off"
+ */
+ tcpm_set_pwr_role(port, TYPEC_SINK);
if (tcpm_pd_send_control(port, PD_CTRL_PS_RDY)) {
tcpm_set_state(port, ERROR_RECOVERY, 0);
break;
tcpm_set_state_cond(port, SNK_UNATTACHED, PD_T_PS_SOURCE_ON);
break;
case PR_SWAP_SRC_SNK_SINK_ON:
- tcpm_set_pwr_role(port, TYPEC_SINK);
tcpm_swap_complete(port, 0);
tcpm_set_state(port, SNK_STARTUP, 0);
break;
case PR_SWAP_SNK_SRC_SOURCE_ON:
tcpm_set_cc(port, tcpm_rp_cc(port));
tcpm_set_vbus(port, true);
- tcpm_pd_send_control(port, PD_CTRL_PS_RDY);
+ /*
+ * USB PD standard, 6.2.1.4:
+ * "Subsequent Messages initiated by the Policy Engine,
+ * such as the PS_RDY Message sent to indicate that Vbus
+ * is ready, will have the Port Power Role field set to
+ * Source."
+ */
tcpm_set_pwr_role(port, TYPEC_SOURCE);
+ tcpm_pd_send_control(port, PD_CTRL_PS_RDY);
tcpm_swap_complete(port, 0);
tcpm_set_state(port, SRC_STARTUP, 0);
break;
return nr_pdo;
}
+static int tcpm_copy_vdos(u32 *dest_vdo, const u32 *src_vdo,
+ unsigned int nr_vdo)
+{
+ unsigned int i;
+
+ if (nr_vdo > VDO_MAX_OBJECTS)
+ nr_vdo = VDO_MAX_OBJECTS;
+
+ for (i = 0; i < nr_vdo; i++)
+ dest_vdo[i] = src_vdo[i];
+
+ return nr_vdo;
+}
+
void tcpm_update_source_capabilities(struct tcpm_port *port, const u32 *pdo,
unsigned int nr_pdo)
{
tcpc->config->nr_src_pdo);
port->nr_snk_pdo = tcpm_copy_pdos(port->snk_pdo, tcpc->config->snk_pdo,
tcpc->config->nr_snk_pdo);
+ port->nr_snk_vdo = tcpm_copy_vdos(port->snk_vdo, tcpc->config->snk_vdo,
+ tcpc->config->nr_snk_vdo);
port->max_snk_mv = tcpc->config->max_snk_mv;
port->max_snk_ma = tcpc->config->max_snk_ma;
const u32 *snk_pdo;
unsigned int nr_snk_pdo;
+ const u32 *snk_vdo;
+ unsigned int nr_snk_vdo;
+
unsigned int max_snk_mv;
unsigned int max_snk_ma;
unsigned int max_snk_mw;
*/
sg_init_table(scatterlist, num_pages);
/* Now set the pages for each scatterlist */
- for (i = 0; i < num_pages; i++)
- sg_set_page(scatterlist + i, pages[i], PAGE_SIZE, 0);
+ for (i = 0; i < num_pages; i++) {
+ unsigned int len = PAGE_SIZE - offset;
+
+ if (len > count)
+ len = count;
+ sg_set_page(scatterlist + i, pages[i], len, offset);
+ offset = 0;
+ count -= len;
+ }
dma_buffers = dma_map_sg(g_dev,
scatterlist,
u32 addr = sg_dma_address(sg);
/* Note: addrs is the address + page_count - 1
- * The firmware expects the block to be page
+ * The firmware expects blocks after the first to be page-
* aligned and a multiple of the page size
*/
WARN_ON(len == 0);
- WARN_ON(len & ~PAGE_MASK);
- WARN_ON(addr & ~PAGE_MASK);
+ WARN_ON(i && (i != (dma_buffers - 1)) && (len & ~PAGE_MASK));
+ WARN_ON(i && (addr & ~PAGE_MASK));
if (k > 0 &&
- ((addrs[k - 1] & PAGE_MASK) |
- ((addrs[k - 1] & ~PAGE_MASK) + 1) << PAGE_SHIFT)
- == addr) {
- addrs[k - 1] += (len >> PAGE_SHIFT);
- } else {
- addrs[k++] = addr | ((len >> PAGE_SHIFT) - 1);
- }
+ ((addrs[k - 1] & PAGE_MASK) +
+ (((addrs[k - 1] & ~PAGE_MASK) + 1) << PAGE_SHIFT))
+ == (addr & PAGE_MASK))
+ addrs[k - 1] += ((len + PAGE_SIZE - 1) >> PAGE_SHIFT);
+ else
+ addrs[k++] = (addr & PAGE_MASK) |
+ (((len + PAGE_SIZE - 1) >> PAGE_SHIFT) - 1);
}
/* Partial cache lines (fragments) require special measures */
# Generic Trusted Execution Environment Configuration
config TEE
tristate "Trusted Execution Environment support"
+ depends on HAVE_ARM_SMCCC || COMPILE_TEST
select DMA_SHARED_BUFFER
select GENERIC_ALLOCATOR
help
config BCM_NS_THERMAL
tristate "Northstar thermal driver"
depends on ARCH_BCM_IPROC || COMPILE_TEST
+ default y if ARCH_BCM_IPROC
help
- Northstar is a family of SoCs that includes e.g. BCM4708, BCM47081,
- BCM4709 and BCM47094. It contains DMU (Device Management Unit) block
- with a thermal sensor that allows checking CPU temperature. This
- driver provides support for it.
+ Support for the Northstar and Northstar Plus family of SoCs (e.g.
+ BCM4708, BCM4709, BCM5301x, BCM95852X, etc). It contains DMU (Device
+ Management Unit) block with a thermal sensor that allows checking CPU
+ temperature.
static int qoriq_tmu_probe(struct platform_device *pdev)
{
int ret;
- const struct thermal_trip *trip;
struct qoriq_tmu_data *data;
struct device_node *np = pdev->dev.of_node;
u32 site = 0;
goto err_tmu;
}
- trip = of_thermal_get_trip_points(data->tz);
-
/* Enable monitoring */
site |= 0x1 << (15 - data->sensor_id);
tmu_write(data, site | TMR_ME | TMR_ALPF, &data->regs->tmr);
* This may be called from any critical situation to trigger a system shutdown
* after a known period of time. By default this is not scheduled.
*/
-void thermal_emergency_poweroff(void)
+static void thermal_emergency_poweroff(void)
{
int poweroff_delay_ms = CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS;
/*
}
/**
- * ti_bandgap_set_continous_mode() - One time enabling of continuous mode
+ * ti_bandgap_set_continuous_mode() - One time enabling of continuous mode
* @bgp: pointer to struct ti_bandgap
*
* Call this function only if HAS(MODE_CONFIG) is set. As this driver may
}
bgp = devm_kzalloc(&pdev->dev, sizeof(*bgp), GFP_KERNEL);
- if (!bgp) {
- dev_err(&pdev->dev, "Unable to allocate mem for driver ref\n");
+ if (!bgp)
return ERR_PTR(-ENOMEM);
- }
of_id = of_match_device(of_ti_bandgap_match, &pdev->dev);
if (of_id)
bgp->conf = of_id->data;
/* register shadow for context save and restore */
- bgp->regval = devm_kzalloc(&pdev->dev, sizeof(*bgp->regval) *
- bgp->conf->sensor_count, GFP_KERNEL);
- if (!bgp->regval) {
- dev_err(&pdev->dev, "Unable to allocate mem for driver ref\n");
+ bgp->regval = devm_kcalloc(&pdev->dev, bgp->conf->sensor_count,
+ sizeof(*bgp->regval), GFP_KERNEL);
+ if (!bgp->regval)
return ERR_PTR(-ENOMEM);
- }
i = 0;
do {
ehv_bc_driver = alloc_tty_driver(count);
if (!ehv_bc_driver) {
ret = -ENOMEM;
- goto error;
+ goto err_free_bcs;
}
ehv_bc_driver->driver_name = "ehv-bc";
ret = tty_register_driver(ehv_bc_driver);
if (ret) {
pr_err("ehv-bc: could not register tty driver (ret=%i)\n", ret);
- goto error;
+ goto err_put_tty_driver;
}
ret = platform_driver_register(&ehv_bc_tty_driver);
if (ret) {
pr_err("ehv-bc: could not register platform driver (ret=%i)\n",
ret);
- goto error;
+ goto err_deregister_tty_driver;
}
return 0;
-error:
- if (ehv_bc_driver) {
- tty_unregister_driver(ehv_bc_driver);
- put_tty_driver(ehv_bc_driver);
- }
-
+err_deregister_tty_driver:
+ tty_unregister_driver(ehv_bc_driver);
+err_put_tty_driver:
+ put_tty_driver(ehv_bc_driver);
+err_free_bcs:
kfree(bcs);
return ret;
}
EXPORT_SYMBOL_GPL(serdev_device_write_wakeup);
+int serdev_device_write_buf(struct serdev_device *serdev,
+ const unsigned char *buf, size_t count)
+{
+ struct serdev_controller *ctrl = serdev->ctrl;
+
+ if (!ctrl || !ctrl->ops->write_buf)
+ return -EINVAL;
+
+ return ctrl->ops->write_buf(ctrl, buf, count);
+}
+EXPORT_SYMBOL_GPL(serdev_device_write_buf);
+
int serdev_device_write(struct serdev_device *serdev,
const unsigned char *buf, size_t count,
unsigned long timeout)
return PTR_ERR(tty);
serport->tty = tty;
- serport->port->client_ops = &client_ops;
- serport->port->client_data = ctrl;
-
if (tty->ops->open)
tty->ops->open(serport->tty, NULL);
else
struct device *parent,
struct tty_driver *drv, int idx)
{
+ const struct tty_port_client_operations *old_ops;
struct serdev_controller *ctrl;
struct serport *serport;
int ret;
ctrl->ops = &ctrl_ops;
+ old_ops = port->client_ops;
+ port->client_ops = &client_ops;
+ port->client_data = ctrl;
+
ret = serdev_controller_add(ctrl);
if (ret)
- goto err_controller_put;
+ goto err_reset_data;
dev_info(&ctrl->dev, "tty port %s%d registered\n", drv->name, idx);
return &ctrl->dev;
-err_controller_put:
+err_reset_data:
+ port->client_data = NULL;
+ port->client_ops = old_ops;
serdev_controller_put(ctrl);
+
return ERR_PTR(ret);
}
-void serdev_tty_port_unregister(struct tty_port *port)
+int serdev_tty_port_unregister(struct tty_port *port)
{
struct serdev_controller *ctrl = port->client_data;
struct serport *serport = serdev_controller_get_drvdata(ctrl);
if (!serport)
- return;
+ return -ENODEV;
serdev_controller_remove(ctrl);
port->client_ops = NULL;
port->client_data = NULL;
serdev_controller_put(ctrl);
+
+ return 0;
}
/*
* These are definitions for the Exar XR17V35X and XR17(C|D)15X
*/
+#define UART_EXAR_INT0 0x80
#define UART_EXAR_SLEEP 0x8b /* Sleep mode */
#define UART_EXAR_DVID 0x8d /* Device identification */
/*
* Check if the device is a Fintek F81216A
*/
- if (port->type == PORT_16550A)
+ if (port->type == PORT_16550A && port->iotype == UPIO_PORT)
fintek_8250_probe(up);
if (up->capabilities != old_capabilities) {
static int exar_handle_irq(struct uart_port *port)
{
unsigned int iir = serial_port_in(port, UART_IIR);
- int ret;
+ int ret = 0;
- ret = serial8250_handle_irq(port, iir);
+ if (((port->type == PORT_XR17V35X) || (port->type == PORT_XR17D15X)) &&
+ serial_port_in(port, UART_EXAR_INT0) != 0)
+ ret = 1;
- if ((port->type == PORT_XR17V35X) ||
- (port->type == PORT_XR17D15X)) {
- serial_port_in(port, 0x80);
- serial_port_in(port, 0x81);
- serial_port_in(port, 0x82);
- serial_port_in(port, 0x83);
- }
+ ret |= serial8250_handle_irq(port, iir);
return ret;
}
serial_port_in(port, UART_RX);
serial_port_in(port, UART_IIR);
serial_port_in(port, UART_MSR);
+ if ((port->type == PORT_XR17V35X) || (port->type == PORT_XR17D15X))
+ serial_port_in(port, UART_EXAR_INT0);
/*
* At this point, there's no way the LSR could still be 0xff;
serial_port_in(port, UART_RX);
serial_port_in(port, UART_IIR);
serial_port_in(port, UART_MSR);
+ if ((port->type == PORT_XR17V35X) || (port->type == PORT_XR17D15X))
+ serial_port_in(port, UART_EXAR_INT0);
up->lsr_saved_flags = 0;
up->msr_saved_flags = 0;
port = &altera_jtaguart_ports[i].port;
uart_remove_one_port(&altera_jtaguart_driver, port);
+ iounmap(port->membase);
return 0;
}
if (port) {
uart_remove_one_port(&altera_uart_driver, port);
port->mapbase = 0;
+ iounmap(port->membase);
}
return 0;
#define UARTn_FRAME 0x04
#define UARTn_FRAME_DATABITS__MASK 0x000f
#define UARTn_FRAME_DATABITS(n) ((n) - 3)
+#define UARTn_FRAME_PARITY__MASK 0x0300
#define UARTn_FRAME_PARITY_NONE 0x0000
#define UARTn_FRAME_PARITY_EVEN 0x0200
#define UARTn_FRAME_PARITY_ODD 0x0300
16 * (4 + (clkdiv >> 6)));
frame = efm32_uart_read32(efm_port, UARTn_FRAME);
- if (frame & UARTn_FRAME_PARITY_ODD)
+ switch (frame & UARTn_FRAME_PARITY__MASK) {
+ case UARTn_FRAME_PARITY_ODD:
*parity = 'o';
- else if (frame & UARTn_FRAME_PARITY_EVEN)
+ break;
+ case UARTn_FRAME_PARITY_EVEN:
*parity = 'e';
- else
+ break;
+ default:
*parity = 'n';
+ }
*bits = (frame & UARTn_FRAME_DATABITS__MASK) -
UARTn_FRAME_DATABITS(4) + 4;
static void __exit ifx_spi_exit(void)
{
/* unregister */
+ spi_unregister_driver(&ifx_spi_driver);
tty_unregister_driver(tty_drv);
put_tty_driver(tty_drv);
- spi_unregister_driver(&ifx_spi_driver);
unregister_reboot_notifier(&ifx_modem_reboot_notifier_block);
}
* and DCD (when they are outputs) or enables the respective
* irqs. So set this bit early, i.e. before requesting irqs.
*/
- writel(UFCR_DCEDTE, sport->port.membase + UFCR);
+ reg = readl(sport->port.membase + UFCR);
+ if (!(reg & UFCR_DCEDTE))
+ writel(reg | UFCR_DCEDTE, sport->port.membase + UFCR);
/*
* Disable UCR3_RI and UCR3_DCD irqs. They are also not
sport->port.membase + UCR3);
} else {
- writel(0, sport->port.membase + UFCR);
+ unsigned long ucr3 = UCR3_DSR;
+
+ reg = readl(sport->port.membase + UFCR);
+ if (reg & UFCR_DCEDTE)
+ writel(reg & ~UFCR_DCEDTE, sport->port.membase + UFCR);
+
+ if (!is_imx1_uart(sport))
+ ucr3 |= IMX21_UCR3_RXDMUXSEL | UCR3_ADNIMP;
+ writel(ucr3, sport->port.membase + UCR3);
}
clk_disable_unprepare(sport->clk_ipg);
mutex_lock(&port->mutex);
tty_dev = device_find_child(uport->dev, &match, serial_match_port);
- if (device_may_wakeup(tty_dev)) {
+ if (tty_dev && device_may_wakeup(tty_dev)) {
if (!enable_irq_wake(uport->irq))
uport->irq_wake = 1;
put_device(tty_dev);
* Register the port whether it's detected or not. This allows
* setserial to be used to alter this port's parameters.
*/
- tty_dev = tty_port_register_device_attr(port, drv->tty_driver,
+ tty_dev = tty_port_register_device_attr_serdev(port, drv->tty_driver,
uport->line, uport->dev, port, uport->tty_groups);
if (likely(!IS_ERR(tty_dev))) {
device_set_wakeup_capable(tty_dev, 1);
/*
* Remove the devices from the tty layer
*/
- tty_unregister_device(drv->tty_driver, uport->line);
+ tty_port_unregister_device(port, drv->tty_driver, uport->line);
tty = tty_port_tty_get(port);
if (tty) {
if (!disc)
return 0;
+ mutex_lock(&tty->atomic_write_lock);
ret = tty_ldisc_receive_buf(disc, p, (char *)f, count);
+ mutex_unlock(&tty->atomic_write_lock);
tty_ldisc_deref(disc);
struct tty_driver *driver, unsigned index,
struct device *device, void *drvdata,
const struct attribute_group **attr_grp)
+{
+ tty_port_link_device(port, driver, index);
+ return tty_register_device_attr(driver, index, device, drvdata,
+ attr_grp);
+}
+EXPORT_SYMBOL_GPL(tty_port_register_device_attr);
+
+/**
+ * tty_port_register_device_attr_serdev - register tty or serdev device
+ * @port: tty_port of the device
+ * @driver: tty_driver for this device
+ * @index: index of the tty
+ * @device: parent if exists, otherwise NULL
+ * @drvdata: driver data for the device
+ * @attr_grp: attribute group for the device
+ *
+ * Register a serdev or tty device depending on if the parent device has any
+ * defined serdev clients or not.
+ */
+struct device *tty_port_register_device_attr_serdev(struct tty_port *port,
+ struct tty_driver *driver, unsigned index,
+ struct device *device, void *drvdata,
+ const struct attribute_group **attr_grp)
{
struct device *dev;
tty_port_link_device(port, driver, index);
dev = serdev_tty_port_register(port, device, driver, index);
- if (PTR_ERR(dev) != -ENODEV)
+ if (PTR_ERR(dev) != -ENODEV) {
/* Skip creating cdev if we registered a serdev device */
return dev;
+ }
return tty_register_device_attr(driver, index, device, drvdata,
attr_grp);
}
-EXPORT_SYMBOL_GPL(tty_port_register_device_attr);
+EXPORT_SYMBOL_GPL(tty_port_register_device_attr_serdev);
+
+/**
+ * tty_port_register_device_serdev - register tty or serdev device
+ * @port: tty_port of the device
+ * @driver: tty_driver for this device
+ * @index: index of the tty
+ * @device: parent if exists, otherwise NULL
+ *
+ * Register a serdev or tty device depending on if the parent device has any
+ * defined serdev clients or not.
+ */
+struct device *tty_port_register_device_serdev(struct tty_port *port,
+ struct tty_driver *driver, unsigned index,
+ struct device *device)
+{
+ return tty_port_register_device_attr_serdev(port, driver, index,
+ device, NULL, NULL);
+}
+EXPORT_SYMBOL_GPL(tty_port_register_device_serdev);
+
+/**
+ * tty_port_unregister_device - deregister a tty or serdev device
+ * @port: tty_port of the device
+ * @driver: tty_driver for this device
+ * @index: index of the tty
+ *
+ * If a tty or serdev device is registered with a call to
+ * tty_port_register_device_serdev() then this function must be called when
+ * the device is gone.
+ */
+void tty_port_unregister_device(struct tty_port *port,
+ struct tty_driver *driver, unsigned index)
+{
+ int ret;
+
+ ret = serdev_tty_port_unregister(port);
+ if (ret == 0)
+ return;
+
+ tty_unregister_device(driver, index);
+}
+EXPORT_SYMBOL_GPL(tty_port_unregister_device);
int tty_port_alloc_xmit_buf(struct tty_port *port)
{
/* check if last port ref was dropped before tty release */
if (WARN_ON(port->itty))
return;
-
- serdev_tty_port_unregister(port);
-
if (port->xmit_buf)
free_page((unsigned long)port->xmit_buf);
tty_port_destroy(port);
map = kzalloc(sizeof(*map), GFP_KERNEL);
if (!map) {
ret = -ENOMEM;
- goto err_map_kobj;
+ goto err_map;
}
kobject_init(&map->kobj, &map_attr_type);
map->mem = mem;
goto err_map_kobj;
ret = kobject_uevent(&map->kobj, KOBJ_ADD);
if (ret)
- goto err_map;
+ goto err_map_kobj;
}
for (pi = 0; pi < MAX_UIO_PORT_REGIONS; pi++) {
portio = kzalloc(sizeof(*portio), GFP_KERNEL);
if (!portio) {
ret = -ENOMEM;
- goto err_portio_kobj;
+ goto err_portio;
}
kobject_init(&portio->kobj, &portio_attr_type);
portio->port = port;
goto err_portio_kobj;
ret = kobject_uevent(&portio->kobj, KOBJ_ADD);
if (ret)
- goto err_portio;
+ goto err_portio_kobj;
}
return 0;
if (userurb) { /* Async */
if (when == SUBMIT)
- dev_info(&udev->dev, "userurb %p, ep%d %s-%s, "
+ dev_info(&udev->dev, "userurb %pK, ep%d %s-%s, "
"length %u\n",
userurb, ep, t, d, length);
else
- dev_info(&udev->dev, "userurb %p, ep%d %s-%s, "
+ dev_info(&udev->dev, "userurb %pK, ep%d %s-%s, "
"actual_length %u status %d\n",
userurb, ep, t, d, length,
timeout_or_status);
if (as) {
int retval;
- snoop(&ps->dev->dev, "reap %p\n", as->userurb);
+ snoop(&ps->dev->dev, "reap %pK\n", as->userurb);
retval = processcompl(as, (void __user * __user *)arg);
free_async(as);
return retval;
as = async_getcompleted(ps);
if (as) {
- snoop(&ps->dev->dev, "reap %p\n", as->userurb);
+ snoop(&ps->dev->dev, "reap %pK\n", as->userurb);
retval = processcompl(as, (void __user * __user *)arg);
free_async(as);
} else {
if (as) {
int retval;
- snoop(&ps->dev->dev, "reap %p\n", as->userurb);
+ snoop(&ps->dev->dev, "reap %pK\n", as->userurb);
retval = processcompl_compat(as, (void __user * __user *)arg);
free_async(as);
return retval;
as = async_getcompleted(ps);
if (as) {
- snoop(&ps->dev->dev, "reap %p\n", as->userurb);
+ snoop(&ps->dev->dev, "reap %pK\n", as->userurb);
retval = processcompl_compat(as, (void __user * __user *)arg);
free_async(as);
} else {
#endif
case USBDEVFS_DISCARDURB:
- snoop(&dev->dev, "%s: DISCARDURB %p\n", __func__, p);
+ snoop(&dev->dev, "%s: DISCARDURB %pK\n", __func__, p);
ret = proc_unlinkurb(ps, p);
break;
if (retval == 0)
retval = -EINPROGRESS;
else if (retval != -EIDRM && retval != -EBUSY)
- dev_dbg(&udev->dev, "hcd_unlink_urb %p fail %d\n",
+ dev_dbg(&udev->dev, "hcd_unlink_urb %pK fail %d\n",
urb, retval);
usb_put_dev(udev);
}
/* kick hcd */
unlink1(hcd, urb, -ESHUTDOWN);
dev_dbg (hcd->self.controller,
- "shutdown urb %p ep%d%s%s\n",
+ "shutdown urb %pK ep%d%s%s\n",
urb, usb_endpoint_num(&ep->desc),
is_in ? "in" : "out",
({ char *s;
hcd->bandwidth_mutex = kmalloc(sizeof(*hcd->bandwidth_mutex),
GFP_KERNEL);
if (!hcd->bandwidth_mutex) {
+ kfree(hcd->address0_mutex);
kfree(hcd);
dev_dbg(dev, "hcd bandwidth mutex alloc failed\n");
return NULL;
}
/* USB 2.0 spec Section 11.24.4.5 */
-static int get_hub_descriptor(struct usb_device *hdev, void *data)
+static int get_hub_descriptor(struct usb_device *hdev,
+ struct usb_hub_descriptor *desc)
{
int i, ret, size;
unsigned dtype;
for (i = 0; i < 3; i++) {
ret = usb_control_msg(hdev, usb_rcvctrlpipe(hdev, 0),
USB_REQ_GET_DESCRIPTOR, USB_DIR_IN | USB_RT_HUB,
- dtype << 8, 0, data, size,
+ dtype << 8, 0, desc, size,
USB_CTRL_GET_TIMEOUT);
- if (ret >= (USB_DT_HUB_NONVAR_SIZE + 2))
+ if (hub_is_superspeed(hdev)) {
+ if (ret == size)
+ return ret;
+ } else if (ret >= USB_DT_HUB_NONVAR_SIZE + 2) {
+ /* Make sure we have the DeviceRemovable field. */
+ size = USB_DT_HUB_NONVAR_SIZE + desc->bNbrPorts / 8 + 1;
+ if (ret < size)
+ return -EMSGSIZE;
return ret;
+ }
}
return -EINVAL;
}
}
mutex_init(&hub->status_mutex);
- hub->descriptor = kmalloc(sizeof(*hub->descriptor), GFP_KERNEL);
+ hub->descriptor = kzalloc(sizeof(*hub->descriptor), GFP_KERNEL);
if (!hub->descriptor) {
ret = -ENOMEM;
goto fail;
/* Request the entire hub descriptor.
* hub->descriptor can handle USB_MAXCHILDREN ports,
- * but the hub can/will return fewer bytes here.
+ * but a (non-SS) hub can/will return fewer bytes here.
*/
ret = get_hub_descriptor(hdev, hub->descriptor);
if (ret < 0) {
message = "can't read hub descriptor";
goto fail;
- } else if (hub->descriptor->bNbrPorts > USB_MAXCHILDREN) {
+ }
+
+ maxchild = USB_MAXCHILDREN;
+ if (hub_is_superspeed(hdev))
+ maxchild = min_t(unsigned, maxchild, USB_SS_MAXPORTS);
+
+ if (hub->descriptor->bNbrPorts > maxchild) {
message = "hub has too many ports!";
ret = -ENODEV;
goto fail;
*
* Find the companion device from platform bus.
*
+ * Takes a reference to the returned struct device which needs to be dropped
+ * after use.
+ *
* Return: On success, a pointer to the companion device, %NULL on failure.
*/
struct device *usb_of_get_companion_dev(struct device *dev)
if (!urb || !urb->complete)
return -EINVAL;
if (urb->hcpriv) {
- WARN_ONCE(1, "URB %p submitted while active\n", urb);
+ WARN_ONCE(1, "URB %pK submitted while active\n", urb);
return -EBUSY;
}
return PTR_ERR(kdwc->usbss);
kdwc->clk = devm_clk_get(kdwc->dev, "usb");
+ if (IS_ERR(kdwc->clk)) {
+ dev_err(kdwc->dev, "unable to get usb clock\n");
+ return PTR_ERR(kdwc->clk);
+ }
error = clk_prepare_enable(kdwc->clk);
if (error < 0) {
#define PCI_DEVICE_ID_INTEL_APL 0x5aaa
#define PCI_DEVICE_ID_INTEL_KBP 0xa2b0
#define PCI_DEVICE_ID_INTEL_GLK 0x31aa
+#define PCI_DEVICE_ID_INTEL_CNPLP 0x9dee
+#define PCI_DEVICE_ID_INTEL_CNPH 0xa36e
#define PCI_INTEL_BXT_DSM_UUID "732b85d5-b7a7-4a1b-9ba0-4bbd00ffd511"
#define PCI_INTEL_BXT_FUNC_PMU_PWR 4
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_APL), },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_KBP), },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_GLK), },
+ { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CNPLP), },
+ { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CNPH), },
{ PCI_DEVICE(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_NL_USB), },
{ } /* Terminating Entry */
};
__dwc3_gadget_start_isoc(dwc, dep, cur_uf);
dep->flags &= ~DWC3_EP_PENDING_REQUEST;
}
+ return 0;
}
- return 0;
+
+ if ((dep->flags & DWC3_EP_BUSY) &&
+ !(dep->flags & DWC3_EP_MISSED_ISOC)) {
+ WARN_ON_ONCE(!dep->resource_index);
+ ret = __dwc3_gadget_kick_transfer(dep,
+ dep->resource_index);
+ }
+
+ goto out;
}
if (!dwc3_calc_trbs_left(dep))
return 0;
ret = __dwc3_gadget_kick_transfer(dep, 0);
+out:
if (ret == -EBUSY)
ret = 0;
return IRQ_HANDLED;
}
+ /*
+ * With PCIe legacy interrupt, test shows that top-half irq handler can
+ * be called again after HW interrupt deassertion. Check if bottom-half
+ * irq event handler completes before caching new event to prevent
+ * losing events.
+ */
+ if (evt->flags & DWC3_EVENT_PENDING)
+ return IRQ_HANDLED;
+
count = dwc3_readl(dwc->regs, DWC3_GEVNTCOUNT(0));
count &= DWC3_GEVNTCOUNT_MASK;
if (!count)
ep->ep->driver_data = ep;
ep->ep->desc = ds;
- comp_desc = (struct usb_ss_ep_comp_descriptor *)(ds +
- USB_DT_ENDPOINT_SIZE);
- ep->ep->maxburst = comp_desc->bMaxBurst + 1;
-
- if (needs_comp_desc)
+ if (needs_comp_desc) {
+ comp_desc = (struct usb_ss_ep_comp_descriptor *)(ds +
+ USB_DT_ENDPOINT_SIZE);
+ ep->ep->maxburst = comp_desc->bMaxBurst + 1;
ep->ep->comp_desc = comp_desc;
+ }
ret = usb_ep_enable(ep->ep);
if (likely(!ret)) {
struct gscons_info *info = &gscons_info;
unregister_console(&gserial_cons);
- if (info->console_thread != NULL)
+ if (!IS_ERR_OR_NULL(info->console_thread))
kthread_stop(info->console_thread);
gs_buf_free(&info->con_buf);
}
HUB_CHAR_COMMON_OCPM);
desc->bNbrPorts = 1;
desc->u.ss.bHubHdrDecLat = 0x04; /* Worst case: 0.4 micro sec*/
- desc->u.ss.DeviceRemovable = 0xffff;
+ desc->u.ss.DeviceRemovable = 0;
}
static inline void hub_descriptor(struct usb_hub_descriptor *desc)
HUB_CHAR_INDV_PORT_LPSM |
HUB_CHAR_COMMON_OCPM);
desc->bNbrPorts = 1;
- desc->u.hs.DeviceRemovable[0] = 0xff;
- desc->u.hs.DeviceRemovable[1] = 0xff;
+ desc->u.hs.DeviceRemovable[0] = 0;
+ desc->u.hs.DeviceRemovable[1] = 0xff; /* PortPwrCtrlMask */
}
static int dummy_hub_control(
}
companion_dev = usb_of_get_companion_dev(hcd->self.controller);
- if (companion_dev)
+ if (companion_dev) {
device_pm_wait_for_dev(hcd->self.controller, companion_dev);
+ put_device(companion_dev);
+ }
ehci_resume(hcd, priv->reset_on_resume);
return 0;
time = 30;
break;
default:
- time = 300;
+ time = 50;
break;
}
pipe = td->pipe;
pipe_stop(r8a66597, pipe);
+ /* Select a different address or endpoint */
new_td = td;
do {
list_move_tail(&new_td->queue,
new_td = td;
break;
}
- } while (td != new_td && td->address == new_td->address);
+ } while (td != new_td && td->address == new_td->address &&
+ td->pipe->info.epnum == new_td->pipe->info.epnum);
start_transfer(r8a66597, new_td);
wait_for_completion(cmd->completion);
if (cmd->status == COMP_COMMAND_ABORTED ||
- cmd->status == COMP_STOPPED) {
+ cmd->status == COMP_COMMAND_RING_STOPPED) {
xhci_warn(xhci, "Timeout while waiting for stop endpoint command\n");
ret = -ETIME;
}
}
if (max_packet) {
- seg->bounce_buf = kzalloc(max_packet, flags | GFP_DMA);
+ seg->bounce_buf = kzalloc(max_packet, flags);
if (!seg->bounce_buf) {
dma_pool_free(xhci->segment_pool, seg->trbs, dma);
kfree(seg);
xhci->dcbaa->dev_context_ptrs[0] = cpu_to_le64(xhci->scratchpad->sp_dma);
for (i = 0; i < num_sp; i++) {
dma_addr_t dma;
- void *buf = dma_alloc_coherent(dev, xhci->page_size, &dma,
+ void *buf = dma_zalloc_coherent(dev, xhci->page_size, &dma,
flags);
if (!buf)
goto fail_sp4;
/* Place limits on the number of roothub ports so that the hub
* descriptors aren't longer than the USB core will allocate.
*/
- if (xhci->num_usb3_ports > 15) {
+ if (xhci->num_usb3_ports > USB_SS_MAXPORTS) {
xhci_dbg_trace(xhci, trace_xhci_dbg_init,
- "Limiting USB 3.0 roothub ports to 15.");
- xhci->num_usb3_ports = 15;
+ "Limiting USB 3.0 roothub ports to %u.",
+ USB_SS_MAXPORTS);
+ xhci->num_usb3_ports = USB_SS_MAXPORTS;
}
if (xhci->num_usb2_ports > USB_MAXCHILDREN) {
xhci_dbg_trace(xhci, trace_xhci_dbg_init,
#define PCI_DEVICE_ID_INTEL_BROXTON_M_XHCI 0x0aa8
#define PCI_DEVICE_ID_INTEL_BROXTON_B_XHCI 0x1aa8
#define PCI_DEVICE_ID_INTEL_APL_XHCI 0x5aa8
+#define PCI_DEVICE_ID_INTEL_DNV_XHCI 0x19d0
static const char hcd_name[] = "xhci_hcd";
pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_BROXTON_M_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_BROXTON_B_XHCI ||
- pdev->device == PCI_DEVICE_ID_INTEL_APL_XHCI)) {
+ pdev->device == PCI_DEVICE_ID_INTEL_APL_XHCI ||
+ pdev->device == PCI_DEVICE_ID_INTEL_DNV_XHCI)) {
xhci->quirks |= XHCI_PME_STUCK_QUIRK;
}
if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
}
if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
(pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI ||
- pdev->device == PCI_DEVICE_ID_INTEL_APL_XHCI))
+ pdev->device == PCI_DEVICE_ID_INTEL_APL_XHCI ||
+ pdev->device == PCI_DEVICE_ID_INTEL_DNV_XHCI))
xhci->quirks |= XHCI_MISSING_CAS;
if (pdev->vendor == PCI_VENDOR_ID_ETRON &&
irq = platform_get_irq(pdev, 0);
if (irq < 0)
- return -ENODEV;
+ return irq;
/*
* sysdev must point to a device that is known to the system firmware
if (i_cmd->status != COMP_COMMAND_ABORTED)
continue;
- i_cmd->status = COMP_STOPPED;
+ i_cmd->status = COMP_COMMAND_RING_STOPPED;
xhci_dbg(xhci, "Turn aborted command %p to no-op\n",
i_cmd->command_trb);
xhci_urb_free_priv(urb_priv);
usb_hcd_unlink_urb_from_ep(hcd, urb);
spin_unlock(&xhci->lock);
- usb_hcd_giveback_urb(hcd, urb, status);
trace_xhci_urb_giveback(urb);
+ usb_hcd_giveback_urb(hcd, urb, status);
spin_lock(&xhci->lock);
}
cmd_comp_code = GET_COMP_CODE(le32_to_cpu(event->status));
/* If CMD ring stopped we own the trbs between enqueue and dequeue */
- if (cmd_comp_code == COMP_STOPPED) {
+ if (cmd_comp_code == COMP_COMMAND_RING_STOPPED) {
complete_all(&xhci->cmd_ring_stop_completion);
return;
}
break;
case TRB_CMD_NOOP:
/* Is this an aborted command turned to NO-OP? */
- if (cmd->status == COMP_STOPPED)
- cmd_comp_code = COMP_STOPPED;
+ if (cmd->status == COMP_COMMAND_RING_STOPPED)
+ cmd_comp_code = COMP_COMMAND_RING_STOPPED;
break;
case TRB_RESET_EP:
WARN_ON(slot_id != TRB_TO_SLOT_ID(
struct xhci_hcd *xhci = hcd_to_xhci(hcd);
union xhci_trb *event_ring_deq;
irqreturn_t ret = IRQ_NONE;
+ unsigned long flags;
dma_addr_t deq;
u64 temp_64;
u32 status;
- spin_lock(&xhci->lock);
+ spin_lock_irqsave(&xhci->lock, flags);
/* Check if the xHC generated the interrupt, or the irq is shared */
status = readl(&xhci->op_regs->status);
if (status == ~(u32)0) {
*/
status |= STS_EINT;
writel(status, &xhci->op_regs->status);
- /* FIXME when MSI-X is supported and there are multiple vectors */
- /* Clear the MSI-X event interrupt status */
- if (hcd->irq) {
+ if (!hcd->msi_enabled) {
u32 irq_pending;
- /* Acknowledge the PCI interrupt */
irq_pending = readl(&xhci->ir_set->irq_pending);
irq_pending |= IMAN_IP;
writel(irq_pending, &xhci->ir_set->irq_pending);
ret = IRQ_HANDLED;
out:
- spin_unlock(&xhci->lock);
+ spin_unlock_irqrestore(&xhci->lock, flags);
return ret;
}
/* fall back to msi*/
ret = xhci_setup_msi(xhci);
- if (!ret)
- /* hcd->irq is 0, we have MSI */
+ if (!ret) {
+ hcd->msi_enabled = 1;
return 0;
+ }
if (!pdev->irq) {
xhci_err(xhci, "No msi-x/msi found and no IRQ in BIOS\n");
switch (*cmd_status) {
case COMP_COMMAND_ABORTED:
- case COMP_STOPPED:
+ case COMP_COMMAND_RING_STOPPED:
xhci_warn(xhci, "Timeout while waiting for configure endpoint command\n");
ret = -ETIME;
break;
switch (*cmd_status) {
case COMP_COMMAND_ABORTED:
- case COMP_STOPPED:
+ case COMP_COMMAND_RING_STOPPED:
xhci_warn(xhci, "Timeout while waiting for evaluate context command\n");
ret = -ETIME;
break;
ret = reset_device_cmd->status;
switch (ret) {
case COMP_COMMAND_ABORTED:
- case COMP_STOPPED:
+ case COMP_COMMAND_RING_STOPPED:
xhci_warn(xhci, "Timeout waiting for reset device command\n");
ret = -ETIME;
goto command_cleanup;
*/
switch (command->status) {
case COMP_COMMAND_ABORTED:
- case COMP_STOPPED:
+ case COMP_COMMAND_RING_STOPPED:
xhci_warn(xhci, "Timeout while waiting for setup device command\n");
ret = -ETIME;
break;
dev->in_ep = in_ep;
- if (udev->descriptor.idVendor != ALEA_VENDOR_ID)
+ if (le16_to_cpu(udev->descriptor.idVendor) != ALEA_VENDOR_ID)
dev->reads_started = 1;
dev->size = size;
info.revision = le16_to_cpu(dev->udev->descriptor.bcdDevice);
/* 0==UNKNOWN, 1==LOW(usb1.1) ,2=FULL(usb1.1), 3=HIGH(usb2.0) */
- info.speed = le16_to_cpu(dev->udev->speed);
+ info.speed = dev->udev->speed;
info.if_num = dev->interface->cur_altsetting->desc.bInterfaceNumber;
info.report_size = dev->report_size;
USB_MAJOR, dev->minor);
exit:
+ kfree(get_version_reply);
return retval;
error:
mutex_unlock(&sisusb->lock);
- return 1;
+ return true;
}
/* Interface routine */
int ret;
struct usb_hcd *hcd = musb->hcd;
- MUSB_HST_MODE(musb);
- musb->xceiv->otg->default_a = 1;
- musb->xceiv->otg->state = OTG_STATE_A_IDLE;
-
+ if (musb->port_mode == MUSB_PORT_MODE_HOST) {
+ MUSB_HST_MODE(musb);
+ musb->xceiv->otg->default_a = 1;
+ musb->xceiv->otg->state = OTG_STATE_A_IDLE;
+ }
otg_set_host(musb->xceiv->otg, &hcd->self);
hcd->self.otg_port = 1;
musb->xceiv->otg->host = &hcd->self;
u32 dma_remaining;
int src_burst, dst_burst;
u16 csr;
+ u32 psize;
int ch;
s8 dmareq;
s8 sync_dev;
if (chdat->tx) {
/* Send transfer_packet_sz packets at a time */
- musb_writel(ep_conf, TUSB_EP_MAX_PACKET_SIZE_OFFSET,
- chdat->transfer_packet_sz);
+ psize = musb_readl(ep_conf, TUSB_EP_MAX_PACKET_SIZE_OFFSET);
+ psize &= ~0x7ff;
+ psize |= chdat->transfer_packet_sz;
+ musb_writel(ep_conf, TUSB_EP_MAX_PACKET_SIZE_OFFSET, psize);
musb_writel(ep_conf, TUSB_EP_TX_OFFSET,
TUSB_EP_CONFIG_XFR_SIZE(chdat->transfer_len));
} else {
/* Receive transfer_packet_sz packets at a time */
- musb_writel(ep_conf, TUSB_EP_MAX_PACKET_SIZE_OFFSET,
- chdat->transfer_packet_sz << 16);
+ psize = musb_readl(ep_conf, TUSB_EP_MAX_PACKET_SIZE_OFFSET);
+ psize &= ~(0x7ff << 16);
+ psize |= (chdat->transfer_packet_sz << 16);
+ musb_writel(ep_conf, TUSB_EP_MAX_PACKET_SIZE_OFFSET, psize);
musb_writel(ep_conf, TUSB_EP_RX_OFFSET,
TUSB_EP_CONFIG_XFR_SIZE(chdat->transfer_len));
{ USB_DEVICE(FTDI_VID, FTDI_PROPOX_ISPCABLEIII_PID) },
{ USB_DEVICE(FTDI_VID, CYBER_CORTEX_AV_PID),
.driver_info = (kernel_ulong_t)&ftdi_jtag_quirk },
- { USB_DEVICE(OLIMEX_VID, OLIMEX_ARM_USB_OCD_PID),
- .driver_info = (kernel_ulong_t)&ftdi_jtag_quirk },
- { USB_DEVICE(OLIMEX_VID, OLIMEX_ARM_USB_OCD_H_PID),
- .driver_info = (kernel_ulong_t)&ftdi_jtag_quirk },
+ { USB_DEVICE_INTERFACE_NUMBER(OLIMEX_VID, OLIMEX_ARM_USB_OCD_PID, 1) },
+ { USB_DEVICE_INTERFACE_NUMBER(OLIMEX_VID, OLIMEX_ARM_USB_OCD_H_PID, 1) },
+ { USB_DEVICE_INTERFACE_NUMBER(OLIMEX_VID, OLIMEX_ARM_USB_TINY_PID, 1) },
+ { USB_DEVICE_INTERFACE_NUMBER(OLIMEX_VID, OLIMEX_ARM_USB_TINY_H_PID, 1) },
{ USB_DEVICE(FIC_VID, FIC_NEO1973_DEBUG_PID),
.driver_info = (kernel_ulong_t)&ftdi_jtag_quirk },
{ USB_DEVICE(FTDI_VID, FTDI_OOCDLINK_PID),
(new_serial.flags & ASYNC_FLAGS));
priv->custom_divisor = new_serial.custom_divisor;
+check_and_exit:
write_latency_timer(port);
-check_and_exit:
if ((old_priv.flags & ASYNC_SPD_MASK) !=
(priv->flags & ASYNC_SPD_MASK)) {
if ((priv->flags & ASYNC_SPD_MASK) == ASYNC_SPD_HI)
/* Olimex */
#define OLIMEX_VID 0x15BA
#define OLIMEX_ARM_USB_OCD_PID 0x0003
+#define OLIMEX_ARM_USB_TINY_PID 0x0004
+#define OLIMEX_ARM_USB_TINY_H_PID 0x002a
#define OLIMEX_ARM_USB_OCD_H_PID 0x002b
/*
if (!baud) {
/* pick a default, any default... */
baud = 9600;
- } else
+ } else {
+ /* Avoid a zero divisor. */
+ baud = min(baud, 461550);
tty_encode_baud_rate(tty, baud, baud);
+ }
edge_port->baud_rate = baud;
config->wBaudRate = (__u16)((461550L + baud/2) / baud);
static int ir_startup(struct usb_serial *serial)
{
struct usb_irda_cs_descriptor *irda_desc;
+ int rates;
irda_desc = irda_usb_find_class_desc(serial, 0);
if (!irda_desc) {
return -ENODEV;
}
+ rates = le16_to_cpu(irda_desc->wBaudRate);
+
dev_dbg(&serial->dev->dev,
"%s - Baud rates supported:%s%s%s%s%s%s%s%s%s\n",
__func__,
- (irda_desc->wBaudRate & USB_IRDA_BR_2400) ? " 2400" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_9600) ? " 9600" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_19200) ? " 19200" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_38400) ? " 38400" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_57600) ? " 57600" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_115200) ? " 115200" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_576000) ? " 576000" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_1152000) ? " 1152000" : "",
- (irda_desc->wBaudRate & USB_IRDA_BR_4000000) ? " 4000000" : "");
+ (rates & USB_IRDA_BR_2400) ? " 2400" : "",
+ (rates & USB_IRDA_BR_9600) ? " 9600" : "",
+ (rates & USB_IRDA_BR_19200) ? " 19200" : "",
+ (rates & USB_IRDA_BR_38400) ? " 38400" : "",
+ (rates & USB_IRDA_BR_57600) ? " 57600" : "",
+ (rates & USB_IRDA_BR_115200) ? " 115200" : "",
+ (rates & USB_IRDA_BR_576000) ? " 576000" : "",
+ (rates & USB_IRDA_BR_1152000) ? " 1152000" : "",
+ (rates & USB_IRDA_BR_4000000) ? " 4000000" : "");
switch (irda_desc->bmAdditionalBOFs) {
case USB_IRDA_AB_48:
return -ENOMEM;
divisor = mct_u232_calculate_baud_rate(serial, value, &speed);
- put_unaligned_le32(cpu_to_le32(divisor), buf);
+ put_unaligned_le32(divisor, buf);
rc = usb_control_msg(serial->dev, usb_sndctrlpipe(serial->dev, 0),
MCT_U232_SET_BAUD_RATE_REQUEST,
MCT_U232_SET_REQUEST_TYPE,
#define TELIT_PRODUCT_LE922_USBCFG0 0x1042
#define TELIT_PRODUCT_LE922_USBCFG3 0x1043
#define TELIT_PRODUCT_LE922_USBCFG5 0x1045
+#define TELIT_PRODUCT_ME910 0x1100
#define TELIT_PRODUCT_LE920 0x1200
#define TELIT_PRODUCT_LE910 0x1201
#define TELIT_PRODUCT_LE910_USBCFG4 0x1206
.reserved = BIT(5) | BIT(6),
};
+static const struct option_blacklist_info telit_me910_blacklist = {
+ .sendsetup = BIT(0),
+ .reserved = BIT(1) | BIT(3),
+};
+
static const struct option_blacklist_info telit_le910_blacklist = {
.sendsetup = BIT(0),
.reserved = BIT(1) | BIT(2),
.driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg3 },
{ USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, TELIT_PRODUCT_LE922_USBCFG5, 0xff),
.driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg0 },
+ { USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_ME910),
+ .driver_info = (kernel_ulong_t)&telit_me910_blacklist },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE910),
.driver_info = (kernel_ulong_t)&telit_le910_blacklist },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE910_USBCFG4),
{DEVICE_SWI(0x1199, 0x9071)}, /* Sierra Wireless MC74xx */
{DEVICE_SWI(0x1199, 0x9078)}, /* Sierra Wireless EM74xx */
{DEVICE_SWI(0x1199, 0x9079)}, /* Sierra Wireless EM74xx */
+ {DEVICE_SWI(0x1199, 0x907a)}, /* Sierra Wireless EM74xx QDL */
+ {DEVICE_SWI(0x1199, 0x907b)}, /* Sierra Wireless EM74xx */
{DEVICE_SWI(0x413c, 0x81a2)}, /* Dell Wireless 5806 Gobi(TM) 4G LTE Mobile Broadband Card */
{DEVICE_SWI(0x413c, 0x81a3)}, /* Dell Wireless 5570 HSPA+ (42Mbps) Mobile Broadband Card */
{DEVICE_SWI(0x413c, 0x81a4)}, /* Dell Wireless 5570e HSPA+ (42Mbps) Mobile Broadband Card */
#define SD_BLOCK_LEN 9
struct ene_ub6250_info {
+
+ /* I/O bounce buffer */
+ u8 *bbuf;
+
/* for 6250 code */
struct SD_STATUS SD_Status;
struct MS_STATUS MS_Status;
static void ene_ub6250_info_destructor(void *extra)
{
+ struct ene_ub6250_info *info = (struct ene_ub6250_info *) extra;
+
if (!extra)
return;
+ kfree(info->bbuf);
}
static int ene_send_scsi_cmd(struct us_data *us, u8 fDir, void *buf, int use_sg)
u8 PageNum, u32 *PageBuf, struct ms_lib_type_extdat *ExtraDat)
{
struct bulk_cb_wrap *bcb = (struct bulk_cb_wrap *) us->iobuf;
+ struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+ u8 *bbuf = info->bbuf;
int result;
- u8 ExtBuf[4];
u32 bn = PhyBlockAddr * 0x20 + PageNum;
result = ene_load_bincode(us, MS_RW_PATTERN);
bcb->CDB[2] = (unsigned char)(PhyBlockAddr>>16);
bcb->CDB[6] = 0x01;
- result = ene_send_scsi_cmd(us, FDIR_READ, &ExtBuf, 0);
+ result = ene_send_scsi_cmd(us, FDIR_READ, bbuf, 0);
if (result != USB_STOR_XFER_GOOD)
return USB_STOR_TRANSPORT_ERROR;
ExtraDat->status0 = 0x10; /* Not yet,fireware support */
ExtraDat->status1 = 0x00; /* Not yet,fireware support */
- ExtraDat->ovrflg = ExtBuf[0];
- ExtraDat->mngflg = ExtBuf[1];
- ExtraDat->logadr = memstick_logaddr(ExtBuf[2], ExtBuf[3]);
+ ExtraDat->ovrflg = bbuf[0];
+ ExtraDat->mngflg = bbuf[1];
+ ExtraDat->logadr = memstick_logaddr(bbuf[2], bbuf[3]);
return USB_STOR_TRANSPORT_GOOD;
}
u8 PageNum, struct ms_lib_type_extdat *ExtraDat)
{
struct bulk_cb_wrap *bcb = (struct bulk_cb_wrap *) us->iobuf;
+ struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+ u8 *bbuf = info->bbuf;
int result;
- u8 ExtBuf[4];
memset(bcb, 0, sizeof(struct bulk_cb_wrap));
bcb->Signature = cpu_to_le32(US_BULK_CB_SIGN);
bcb->CDB[2] = (unsigned char)(PhyBlock>>16);
bcb->CDB[6] = 0x01;
- result = ene_send_scsi_cmd(us, FDIR_READ, &ExtBuf, 0);
+ result = ene_send_scsi_cmd(us, FDIR_READ, bbuf, 0);
if (result != USB_STOR_XFER_GOOD)
return USB_STOR_TRANSPORT_ERROR;
ExtraDat->intr = 0x80; /* Not yet, waiting for fireware support */
ExtraDat->status0 = 0x10; /* Not yet, waiting for fireware support */
ExtraDat->status1 = 0x00; /* Not yet, waiting for fireware support */
- ExtraDat->ovrflg = ExtBuf[0];
- ExtraDat->mngflg = ExtBuf[1];
- ExtraDat->logadr = memstick_logaddr(ExtBuf[2], ExtBuf[3]);
+ ExtraDat->ovrflg = bbuf[0];
+ ExtraDat->mngflg = bbuf[1];
+ ExtraDat->logadr = memstick_logaddr(bbuf[2], bbuf[3]);
return USB_STOR_TRANSPORT_GOOD;
}
u16 PhyBlock, newblk, i;
u16 LogStart, LogEnde;
struct ms_lib_type_extdat extdat;
- u8 buf[0x200];
u32 count = 0, index = 0;
struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+ u8 *bbuf = info->bbuf;
for (PhyBlock = 0; PhyBlock < info->MS_Lib.NumberOfPhyBlock;) {
ms_lib_phy_to_log_range(PhyBlock, &LogStart, &LogEnde);
}
if (count == PhyBlock) {
- ms_lib_read_extrablock(us, PhyBlock, 0, 0x80, &buf);
+ ms_lib_read_extrablock(us, PhyBlock, 0, 0x80,
+ bbuf);
count += 0x80;
}
index = (PhyBlock % 0x80) * 4;
- extdat.ovrflg = buf[index];
- extdat.mngflg = buf[index+1];
- extdat.logadr = memstick_logaddr(buf[index+2], buf[index+3]);
+ extdat.ovrflg = bbuf[index];
+ extdat.mngflg = bbuf[index+1];
+ extdat.logadr = memstick_logaddr(bbuf[index+2],
+ bbuf[index+3]);
if ((extdat.ovrflg & MS_REG_OVR_BKST) != MS_REG_OVR_BKST_OK) {
ms_lib_setacquired_errorblock(us, PhyBlock);
{
struct bulk_cb_wrap *bcb = (struct bulk_cb_wrap *) us->iobuf;
int result;
- u8 buf[0x200];
u16 MSP_BlockSize, MSP_UserAreaBlocks;
struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+ u8 *bbuf = info->bbuf;
printk(KERN_INFO "transport --- ENE_MSInit\n");
bcb->CDB[0] = 0xF1;
bcb->CDB[1] = 0x01;
- result = ene_send_scsi_cmd(us, FDIR_READ, &buf, 0);
+ result = ene_send_scsi_cmd(us, FDIR_READ, bbuf, 0);
if (result != USB_STOR_XFER_GOOD) {
printk(KERN_ERR "Execution MS Init Code Fail !!\n");
return USB_STOR_TRANSPORT_ERROR;
}
/* the same part to test ENE */
- info->MS_Status = *(struct MS_STATUS *)&buf[0];
+ info->MS_Status = *(struct MS_STATUS *) bbuf;
if (info->MS_Status.Insert && info->MS_Status.Ready) {
printk(KERN_INFO "Insert = %x\n", info->MS_Status.Insert);
printk(KERN_INFO "IsMSPHG = %x\n", info->MS_Status.IsMSPHG);
printk(KERN_INFO "WtP= %x\n", info->MS_Status.WtP);
if (info->MS_Status.IsMSPro) {
- MSP_BlockSize = (buf[6] << 8) | buf[7];
- MSP_UserAreaBlocks = (buf[10] << 8) | buf[11];
+ MSP_BlockSize = (bbuf[6] << 8) | bbuf[7];
+ MSP_UserAreaBlocks = (bbuf[10] << 8) | bbuf[11];
info->MSP_TotalBlock = MSP_BlockSize * MSP_UserAreaBlocks;
} else {
ms_card_init(us); /* Card is MS (to ms.c)*/
}
usb_stor_dbg(us, "MS Init Code OK !!\n");
} else {
- usb_stor_dbg(us, "MS Card Not Ready --- %x\n", buf[0]);
+ usb_stor_dbg(us, "MS Card Not Ready --- %x\n", bbuf[0]);
return USB_STOR_TRANSPORT_ERROR;
}
static int ene_sd_init(struct us_data *us)
{
int result;
- u8 buf[0x200];
struct bulk_cb_wrap *bcb = (struct bulk_cb_wrap *) us->iobuf;
struct ene_ub6250_info *info = (struct ene_ub6250_info *) us->extra;
+ u8 *bbuf = info->bbuf;
usb_stor_dbg(us, "transport --- ENE_SDInit\n");
/* SD Init Part-1 */
bcb->Flags = US_BULK_FLAG_IN;
bcb->CDB[0] = 0xF1;
- result = ene_send_scsi_cmd(us, FDIR_READ, &buf, 0);
+ result = ene_send_scsi_cmd(us, FDIR_READ, bbuf, 0);
if (result != USB_STOR_XFER_GOOD) {
usb_stor_dbg(us, "Execution SD Init Code Fail !!\n");
return USB_STOR_TRANSPORT_ERROR;
}
- info->SD_Status = *(struct SD_STATUS *)&buf[0];
+ info->SD_Status = *(struct SD_STATUS *) bbuf;
if (info->SD_Status.Insert && info->SD_Status.Ready) {
struct SD_STATUS *s = &info->SD_Status;
- ene_get_card_status(us, (unsigned char *)&buf);
+ ene_get_card_status(us, bbuf);
usb_stor_dbg(us, "Insert = %x\n", s->Insert);
usb_stor_dbg(us, "Ready = %x\n", s->Ready);
usb_stor_dbg(us, "IsMMC = %x\n", s->IsMMC);
usb_stor_dbg(us, "HiSpeed = %x\n", s->HiSpeed);
usb_stor_dbg(us, "WtP = %x\n", s->WtP);
} else {
- usb_stor_dbg(us, "SD Card Not Ready --- %x\n", buf[0]);
+ usb_stor_dbg(us, "SD Card Not Ready --- %x\n", bbuf[0]);
return USB_STOR_TRANSPORT_ERROR;
}
return USB_STOR_TRANSPORT_GOOD;
static int ene_init(struct us_data *us)
{
int result;
- u8 misc_reg03 = 0;
+ u8 misc_reg03;
struct ene_ub6250_info *info = (struct ene_ub6250_info *)(us->extra);
+ u8 *bbuf = info->bbuf;
- result = ene_get_card_type(us, REG_CARD_STATUS, &misc_reg03);
+ result = ene_get_card_type(us, REG_CARD_STATUS, bbuf);
if (result != USB_STOR_XFER_GOOD)
return USB_STOR_TRANSPORT_ERROR;
+ misc_reg03 = bbuf[0];
if (misc_reg03 & 0x01) {
if (!info->SD_Status.Ready) {
result = ene_sd_init(us);
const struct usb_device_id *id)
{
int result;
- u8 misc_reg03 = 0;
+ u8 misc_reg03;
struct us_data *us;
+ struct ene_ub6250_info *info;
result = usb_stor_probe1(&us, intf, id,
(id - ene_ub6250_usb_ids) + ene_ub6250_unusual_dev_list,
return result;
/* FIXME: where should the code alloc extra buf ? */
- if (!us->extra) {
- us->extra = kzalloc(sizeof(struct ene_ub6250_info), GFP_KERNEL);
- if (!us->extra)
- return -ENOMEM;
- us->extra_destructor = ene_ub6250_info_destructor;
+ us->extra = kzalloc(sizeof(struct ene_ub6250_info), GFP_KERNEL);
+ if (!us->extra)
+ return -ENOMEM;
+ us->extra_destructor = ene_ub6250_info_destructor;
+
+ info = (struct ene_ub6250_info *)(us->extra);
+ info->bbuf = kmalloc(512, GFP_KERNEL);
+ if (!info->bbuf) {
+ kfree(us->extra);
+ return -ENOMEM;
}
us->transport_name = "ene_ub6250";
return result;
/* probe card type */
- result = ene_get_card_type(us, REG_CARD_STATUS, &misc_reg03);
+ result = ene_get_card_type(us, REG_CARD_STATUS, info->bbuf);
if (result != USB_STOR_XFER_GOOD) {
usb_stor_disconnect(intf);
return USB_STOR_TRANSPORT_ERROR;
}
+ misc_reg03 = info->bbuf[0];
if (!(misc_reg03 & 0x01)) {
pr_info("ums_eneub6250: This driver only supports SD/MS cards. "
"It does not support SM cards.\n");
static inline void hub_descriptor(struct usb_hub_descriptor *desc)
{
+ int width;
+
memset(desc, 0, sizeof(*desc));
desc->bDescriptorType = USB_DT_HUB;
- desc->bDescLength = 9;
desc->wHubCharacteristics = cpu_to_le16(
HUB_CHAR_INDV_PORT_LPSM | HUB_CHAR_COMMON_OCPM);
+
desc->bNbrPorts = VHCI_HC_PORTS;
- desc->u.hs.DeviceRemovable[0] = 0xff;
- desc->u.hs.DeviceRemovable[1] = 0xff;
+ BUILD_BUG_ON(VHCI_HC_PORTS > USB_MAXCHILDREN);
+ width = desc->bNbrPorts / 8 + 1;
+ desc->bDescLength = USB_DT_HUB_NONVAR_SIZE + 2 * width;
+ memset(&desc->u.hs.DeviceRemovable[0], 0, width);
+ memset(&desc->u.hs.DeviceRemovable[width], 0xff, width);
}
static int vhci_hub_control(struct usb_hcd *hcd, u16 typeReq, u16 wValue,
static
int i1480_usb_probe(struct usb_interface *iface, const struct usb_device_id *id)
{
+ struct usb_device *udev = interface_to_usbdev(iface);
struct i1480_usb *i1480_usb;
struct i1480 *i1480;
struct device *dev = &iface->dev;
iface->cur_altsetting->desc.bInterfaceNumber);
goto error;
}
- if (iface->num_altsetting > 1
- && interface_to_usbdev(iface)->descriptor.idProduct == 0xbabe) {
+ if (iface->num_altsetting > 1 &&
+ le16_to_cpu(udev->descriptor.idProduct) == 0xbabe) {
/* Need altsetting #1 [HW QUIRK] or EP1 won't work */
result = usb_set_interface(interface_to_usbdev(iface), 0, 1);
if (result < 0)
if (def_disp_name)
core.default_display_name = def_disp_name;
- else if (pdata->default_display_name)
- core.default_display_name = pdata->default_display_name;
register_pm_notifier(&omap_dss_pm_notif_block);
config ORION_WATCHDOG
tristate "Orion watchdog"
- depends on ARCH_ORION5X || ARCH_DOVE || MACH_DOVE || ARCH_MVEBU || COMPILE_TEST
+ depends on ARCH_ORION5X || ARCH_DOVE || MACH_DOVE || ARCH_MVEBU || (COMPILE_TEST && !ARCH_EBSA110)
depends on ARM
select WATCHDOG_CORE
help
if (!wdt)
return -ENOMEM;
+ spin_lock_init(&wdt->lock);
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
wdt->base = devm_ioremap_resource(dev, res);
if (IS_ERR(wdt->base))
return ret;
}
- spin_lock_init(&wdt->lock);
platform_set_drvdata(pdev, wdt);
watchdog_set_drvdata(&bcm_kona_wdt_wdd, wdt);
bcm_kona_wdt_wdd.parent = &pdev->dev;
/* Counter maximum value */
#define CDNS_WDT_COUNTER_MAX 0xFFF
-static int wdt_timeout = CDNS_WDT_DEFAULT_TIMEOUT;
+static int wdt_timeout;
static int nowayout = WATCHDOG_NOWAYOUT;
module_param(wdt_timeout, int, 0);
iTCO_vendor_pre_keepalive(p->smi_res, wd_dev->timeout);
+ /* Reset the timeout status bit so that the timer
+ * needs to count down twice again before rebooting */
+ outw(0x0008, TCO1_STS(p)); /* write 1 to clear bit */
+
/* Reload the timer by writing to the TCO Timer Counter register */
- if (p->iTCO_version >= 2) {
+ if (p->iTCO_version >= 2)
outw(0x01, TCO_RLD(p));
- } else if (p->iTCO_version == 1) {
- /* Reset the timeout status bit so that the timer
- * needs to count down twice again before rebooting */
- outw(0x0008, TCO1_STS(p)); /* write 1 to clear bit */
-
+ else if (p->iTCO_version == 1)
outb(0x01, TCO_RLD(p));
- }
spin_unlock(&p->io_lock);
return 0;
unsigned char val8;
unsigned int tmrval;
- tmrval = seconds_to_ticks(p, t);
-
- /* For TCO v1 the timer counts down twice before rebooting */
- if (p->iTCO_version == 1)
- tmrval /= 2;
+ /* The timer counts down twice before rebooting */
+ tmrval = seconds_to_ticks(p, t) / 2;
/* from the specs: */
/* "Values of 0h-3h are ignored and should not be attempted" */
spin_lock(&p->io_lock);
val16 = inw(TCO_RLD(p));
val16 &= 0x3ff;
+ if (!(inw(TCO1_STS(p)) & 0x0008))
+ val16 += (inw(TCOv2_TMR(p)) & 0x3ff);
spin_unlock(&p->io_lock);
time_left = ticks_to_seconds(p, val16);
return -ENODEV;
}
+ if (iface_desc->desc.bNumEndpoints < 1)
+ return -ENODEV;
+
/* check out the endpoint: it has to be Interrupt & IN */
endpoint = &iface_desc->endpoint[0].desc;
* Licensed under GPLv2.
*/
+#include <linux/delay.h>
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/kernel.h>
struct watchdog_device wdd;
void __iomem *reg_base;
u32 mr;
+ unsigned long last_ping;
};
static int wdt_timeout = WDT_DEFAULT_TIMEOUT;
"Watchdog cannot be stopped once started (default="
__MODULE_STRING(WATCHDOG_NOWAYOUT) ")");
+#define wdt_enabled (!(wdt->mr & AT91_WDT_WDDIS))
+
#define wdt_read(wdt, field) \
readl_relaxed((wdt)->reg_base + (field))
-#define wdt_write(wtd, field, val) \
- writel_relaxed((val), (wdt)->reg_base + (field))
+/* 4 slow clock periods is 4/32768 = 122.07µs*/
+#define WDT_DELAY usecs_to_jiffies(123)
+
+static void wdt_write(struct sama5d4_wdt *wdt, u32 field, u32 val)
+{
+ /*
+ * WDT_CR and WDT_MR must not be modified within three slow clock
+ * periods following a restart of the watchdog performed by a write
+ * access in WDT_CR.
+ */
+ while (time_before(jiffies, wdt->last_ping + WDT_DELAY))
+ usleep_range(30, 125);
+ writel_relaxed(val, wdt->reg_base + field);
+ wdt->last_ping = jiffies;
+}
+
+static void wdt_write_nosleep(struct sama5d4_wdt *wdt, u32 field, u32 val)
+{
+ if (time_before(jiffies, wdt->last_ping + WDT_DELAY))
+ udelay(123);
+ writel_relaxed(val, wdt->reg_base + field);
+ wdt->last_ping = jiffies;
+}
static int sama5d4_wdt_start(struct watchdog_device *wdd)
{
wdt->mr &= ~AT91_WDT_WDD;
wdt->mr |= AT91_WDT_SET_WDV(value);
wdt->mr |= AT91_WDT_SET_WDD(value);
- wdt_write(wdt, AT91_WDT_MR, wdt->mr);
+
+ /*
+ * WDDIS has to be 0 when updating WDD/WDV. The datasheet states: When
+ * setting the WDDIS bit, and while it is set, the fields WDV and WDD
+ * must not be modified.
+ * If the watchdog is enabled, then the timeout can be updated. Else,
+ * wait that the user enables it.
+ */
+ if (wdt_enabled)
+ wdt_write(wdt, AT91_WDT_MR, wdt->mr & ~AT91_WDT_WDDIS);
wdd->timeout = timeout;
static int sama5d4_wdt_init(struct sama5d4_wdt *wdt)
{
- struct watchdog_device *wdd = &wdt->wdd;
- u32 value = WDT_SEC2TICKS(wdd->timeout);
u32 reg;
-
/*
- * Because the fields WDV and WDD must not be modified when the WDDIS
- * bit is set, so clear the WDDIS bit before writing the WDT_MR.
+ * When booting and resuming, the bootloader may have changed the
+ * watchdog configuration.
+ * If the watchdog is already running, we can safely update it.
+ * Else, we have to disable it properly.
*/
- reg = wdt_read(wdt, AT91_WDT_MR);
- reg &= ~AT91_WDT_WDDIS;
- wdt_write(wdt, AT91_WDT_MR, reg);
-
- wdt->mr |= AT91_WDT_SET_WDD(value);
- wdt->mr |= AT91_WDT_SET_WDV(value);
-
- wdt_write(wdt, AT91_WDT_MR, wdt->mr);
-
+ if (wdt_enabled) {
+ wdt_write_nosleep(wdt, AT91_WDT_MR, wdt->mr);
+ } else {
+ reg = wdt_read(wdt, AT91_WDT_MR);
+ if (!(reg & AT91_WDT_WDDIS))
+ wdt_write_nosleep(wdt, AT91_WDT_MR,
+ reg | AT91_WDT_WDDIS);
+ }
return 0;
}
struct resource *res;
void __iomem *regs;
u32 irq = 0;
+ u32 timeout;
int ret;
wdt = devm_kzalloc(&pdev->dev, sizeof(*wdt), GFP_KERNEL);
wdd->ops = &sama5d4_wdt_ops;
wdd->min_timeout = MIN_WDT_TIMEOUT;
wdd->max_timeout = MAX_WDT_TIMEOUT;
+ wdt->last_ping = jiffies;
watchdog_set_drvdata(wdd, wdt);
return ret;
}
+ timeout = WDT_SEC2TICKS(wdd->timeout);
+
+ wdt->mr |= AT91_WDT_SET_WDD(timeout);
+ wdt->mr |= AT91_WDT_SET_WDV(timeout);
+
ret = sama5d4_wdt_init(wdt);
if (ret)
return ret;
{
struct sama5d4_wdt *wdt = dev_get_drvdata(dev);
- wdt_write(wdt, AT91_WDT_MR, wdt->mr & ~AT91_WDT_WDDIS);
- if (wdt->mr & AT91_WDT_WDDIS)
- wdt_write(wdt, AT91_WDT_MR, wdt->mr);
+ sama5d4_wdt_init(wdt);
return 0;
}
pr_crit("Would Reboot\n");
#else
pr_crit("Initiating system reboot\n");
- emergency_restart(NULL);
+ emergency_restart();
#endif
#else
pr_crit("Reset in 5ms\n");
base = platform_get_resource(pdev, IORESOURCE_MEM, 0);
wdt->reg_base = devm_ioremap_resource(dev, base);
- if (IS_ERR(wdt->reg_base)) {
- dev_err(dev, "ioremap failed\n");
+ if (IS_ERR(wdt->reg_base))
return PTR_ERR(wdt->reg_base);
- }
zx2967_wdt_reset_sysctrl(dev);
}
size = i_size_read(inode);
- if (!(mode & FALLOC_FL_KEEP_SIZE))
+ if (!(mode & FALLOC_FL_KEEP_SIZE)) {
endoff = offset + length;
+ ret = inode_newsize_ok(inode, endoff);
+ if (ret)
+ goto unlock;
+ }
if (fi->fmode & CEPH_FILE_MODE_LAZY)
want = CEPH_CAP_FILE_BUFFER | CEPH_CAP_FILE_LAZYIO;
u32 acllen = 0;
int rc = 0;
struct tcon_link *tlink = cifs_sb_tlink(cifs_sb);
- struct cifs_tcon *tcon;
+ struct smb_version_operations *ops;
cifs_dbg(NOISY, "converting ACL to mode for %s\n", path);
if (IS_ERR(tlink))
return PTR_ERR(tlink);
- tcon = tlink_tcon(tlink);
- if (pfid && (tcon->ses->server->ops->get_acl_by_fid))
- pntsd = tcon->ses->server->ops->get_acl_by_fid(cifs_sb, pfid,
- &acllen);
- else if (tcon->ses->server->ops->get_acl)
- pntsd = tcon->ses->server->ops->get_acl(cifs_sb, inode, path,
- &acllen);
+ ops = tlink_tcon(tlink)->ses->server->ops;
+
+ if (pfid && (ops->get_acl_by_fid))
+ pntsd = ops->get_acl_by_fid(cifs_sb, pfid, &acllen);
+ else if (ops->get_acl)
+ pntsd = ops->get_acl(cifs_sb, inode, path, &acllen);
else {
cifs_put_tlink(tlink);
return -EOPNOTSUPP;
struct cifs_ntsd *pnntsd = NULL; /* modified acl to be sent to server */
struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
struct tcon_link *tlink = cifs_sb_tlink(cifs_sb);
- struct cifs_tcon *tcon;
+ struct smb_version_operations *ops;
if (IS_ERR(tlink))
return PTR_ERR(tlink);
- tcon = tlink_tcon(tlink);
+
+ ops = tlink_tcon(tlink)->ses->server->ops;
cifs_dbg(NOISY, "set ACL from mode for %s\n", path);
/* Get the security descriptor */
- if (tcon->ses->server->ops->get_acl == NULL) {
+ if (ops->get_acl == NULL) {
cifs_put_tlink(tlink);
return -EOPNOTSUPP;
}
- pntsd = tcon->ses->server->ops->get_acl(cifs_sb, inode, path,
- &secdesclen);
+ pntsd = ops->get_acl(cifs_sb, inode, path, &secdesclen);
if (IS_ERR(pntsd)) {
rc = PTR_ERR(pntsd);
cifs_dbg(VFS, "%s: error %d getting sec desc\n", __func__, rc);
cifs_dbg(NOISY, "build_sec_desc rc: %d\n", rc);
- if (tcon->ses->server->ops->set_acl == NULL)
+ if (ops->set_acl == NULL)
rc = -EOPNOTSUPP;
if (!rc) {
/* Set the security descriptor */
- rc = tcon->ses->server->ops->set_acl(pnntsd, secdesclen, inode,
- path, aclflag);
+ rc = ops->set_acl(pnntsd, secdesclen, inode, path, aclflag);
cifs_dbg(NOISY, "set_cifs_acl rc: %d\n", rc);
}
cifs_put_tlink(tlink);
int (*validate_negotiate)(const unsigned int, struct cifs_tcon *);
ssize_t (*query_all_EAs)(const unsigned int, struct cifs_tcon *,
const unsigned char *, const unsigned char *, char *,
- size_t, const struct nls_table *, int);
+ size_t, struct cifs_sb_info *);
int (*set_EA)(const unsigned int, struct cifs_tcon *, const char *,
const char *, const void *, const __u16,
const struct nls_table *, int);
extern ssize_t CIFSSMBQAllEAs(const unsigned int xid, struct cifs_tcon *tcon,
const unsigned char *searchName,
const unsigned char *ea_name, char *EAData,
- size_t bufsize, const struct nls_table *nls_codepage,
- int remap_special_chars);
+ size_t bufsize, struct cifs_sb_info *cifs_sb);
extern int CIFSSMBSetEA(const unsigned int xid, struct cifs_tcon *tcon,
const char *fileName, const char *ea_name,
const void *ea_value, const __u16 ea_value_len,
{
struct TCP_Server_Info *server = mid->callback_data;
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
add_credits(server, 1, CIFS_ECHO_OP);
}
}
queue_work(cifsiod_wq, &rdata->work);
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
add_credits(server, 1, 0);
}
{
struct cifs_writedata *wdata = mid->callback_data;
struct cifs_tcon *tcon = tlink_tcon(wdata->cfile->tlink);
- struct TCP_Server_Info *server = tcon->ses->server;
unsigned int written;
WRITE_RSP *smb = (WRITE_RSP *)mid->resp_buf;
}
queue_work(cifsiod_wq, &wdata->work);
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
add_credits(tcon->ses->server, 1, 0);
}
CIFSSMBQAllEAs(const unsigned int xid, struct cifs_tcon *tcon,
const unsigned char *searchName, const unsigned char *ea_name,
char *EAData, size_t buf_size,
- const struct nls_table *nls_codepage, int remap)
+ struct cifs_sb_info *cifs_sb)
{
/* BB assumes one setup word */
TRANSACTION2_QPI_REQ *pSMB = NULL;
TRANSACTION2_QPI_RSP *pSMBr = NULL;
+ int remap = cifs_remap(cifs_sb);
+ struct nls_table *nls_codepage = cifs_sb->local_nls;
int rc = 0;
int bytes_returned;
int list_len;
struct cifs_tcon *tcon = tlink_tcon(cfile->tlink);
int rc = 0;
- down_read(&cinode->lock_sem);
+ down_read_nested(&cinode->lock_sem, SINGLE_DEPTH_NESTING);
if (cinode->can_cache_brlcks) {
/* can cache locks - no need to relock */
up_read(&cinode->lock_sem);
rc = tcon->ses->server->ops->query_all_EAs(xid, tcon, path,
"SETFILEBITS", ea_value, 4 /* size of buf */,
- cifs_sb->local_nls,
- cifs_remap(cifs_sb));
+ cifs_sb);
cifs_put_tlink(tlink);
if (rc < 0)
return (int)rc;
goto tcon_exit;
}
- if (rsp->ShareType & SMB2_SHARE_TYPE_DISK)
+ switch (rsp->ShareType) {
+ case SMB2_SHARE_TYPE_DISK:
cifs_dbg(FYI, "connection to disk share\n");
- else if (rsp->ShareType & SMB2_SHARE_TYPE_PIPE) {
+ break;
+ case SMB2_SHARE_TYPE_PIPE:
tcon->ipc = true;
cifs_dbg(FYI, "connection to pipe share\n");
- } else if (rsp->ShareType & SMB2_SHARE_TYPE_PRINT) {
- tcon->print = true;
+ break;
+ case SMB2_SHARE_TYPE_PRINT:
+ tcon->ipc = true;
cifs_dbg(FYI, "connection to printer\n");
- } else {
+ break;
+ default:
cifs_dbg(VFS, "unknown share type %d\n", rsp->ShareType);
rc = -EOPNOTSUPP;
goto tcon_error_exit;
if (mid->mid_state == MID_RESPONSE_RECEIVED)
credits_received = le16_to_cpu(rsp->hdr.sync_hdr.CreditRequest);
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
add_credits(server, credits_received, CIFS_ECHO_OP);
}
cifs_stats_fail_inc(tcon, SMB2_READ_HE);
queue_work(cifsiod_wq, &rdata->work);
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
add_credits(server, credits_received, 0);
}
{
struct cifs_writedata *wdata = mid->callback_data;
struct cifs_tcon *tcon = tlink_tcon(wdata->cfile->tlink);
- struct TCP_Server_Info *server = tcon->ses->server;
unsigned int written;
struct smb2_write_rsp *rsp = (struct smb2_write_rsp *)mid->resp_buf;
unsigned int credits_received = 1;
cifs_stats_fail_inc(tcon, SMB2_WRITE_HE);
queue_work(cifsiod_wq, &wdata->work);
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
add_credits(tcon->ses->server, credits_received, 0);
}
now = jiffies;
/* commands taking longer than one second are indications that
something is wrong, unless it is quite a slow link or server */
- if ((now - midEntry->when_alloc) > HZ) {
+ if (time_after(now, midEntry->when_alloc + HZ)) {
if ((cifsFYI & CIFS_TIMER) && (midEntry->command != command)) {
pr_debug(" CIFS slow rsp: cmd %d mid %llu",
midEntry->command, midEntry->mid);
}
spin_unlock(&GlobalMid_Lock);
- mutex_lock(&server->srv_mutex);
DeleteMidQEntry(mid);
- mutex_unlock(&server->srv_mutex);
return rc;
}
if (pTcon->ses->server->ops->query_all_EAs)
rc = pTcon->ses->server->ops->query_all_EAs(xid, pTcon,
- full_path, name, value, size,
- cifs_sb->local_nls, cifs_remap(cifs_sb));
+ full_path, name, value, size, cifs_sb);
break;
case XATTR_CIFS_ACL: {
if (pTcon->ses->server->ops->query_all_EAs)
rc = pTcon->ses->server->ops->query_all_EAs(xid, pTcon,
- full_path, NULL, data, buf_size,
- cifs_sb->local_nls, cifs_remap(cifs_sb));
+ full_path, NULL, data, buf_size, cifs_sb);
list_ea_exit:
kfree(full_path);
free_xid(xid);
iomap->bdev = bdev;
iomap->offset = (u64)first_block << blkbits;
if (blk_queue_dax(bdev->bd_queue))
- iomap->dax_dev = dax_get_by_host(bdev->bd_disk->disk_name);
+ iomap->dax_dev = fs_dax_get_by_host(bdev->bd_disk->disk_name);
else
iomap->dax_dev = NULL;
ext2_iomap_end(struct inode *inode, loff_t offset, loff_t length,
ssize_t written, unsigned flags, struct iomap *iomap)
{
- put_dax(iomap->dax_dev);
+ fs_put_dax(iomap->dax_dev);
if (iomap->type == IOMAP_MAPPED &&
written < length &&
(flags & IOMAP_WRITE))
bdev = inode->i_sb->s_bdev;
iomap->bdev = bdev;
if (blk_queue_dax(bdev->bd_queue))
- iomap->dax_dev = dax_get_by_host(bdev->bd_disk->disk_name);
+ iomap->dax_dev = fs_dax_get_by_host(bdev->bd_disk->disk_name);
else
iomap->dax_dev = NULL;
iomap->offset = first_block << blkbits;
int blkbits = inode->i_blkbits;
bool truncate = false;
- put_dax(iomap->dax_dev);
+ fs_put_dax(iomap->dax_dev);
if (!(flags & IOMAP_WRITE) || (flags & IOMAP_FAULT))
return 0;
int err;
char *suffix = "";
- if (sb->s_bdev)
+ if (sb->s_bdev) {
suffix = "-fuseblk";
+ /*
+ * sb->s_bdi points to blkdev's bdi however we want to redirect
+ * it to our private bdi...
+ */
+ bdi_put(sb->s_bdi);
+ sb->s_bdi = &noop_backing_dev_info;
+ }
err = super_setup_bdi_name(sb, "%u:%u%s", MAJOR(fc->dev),
MINOR(fc->dev), suffix);
if (err)
xfs_bmbt_rec_t *frp;
xfs_fsblock_t nextbno;
xfs_extnum_t num_recs;
- xfs_extnum_t start;
num_recs = xfs_btree_get_numrecs(block);
if (unlikely(i + num_recs > room)) {
* Copy records into the extent records.
*/
frp = XFS_BMBT_REC_ADDR(mp, block, 1);
- start = i;
for (j = 0; j < num_recs; j++, i++, frp++) {
xfs_bmbt_rec_host_t *trp = xfs_iext_get_ext(ifp, i);
trp->l0 = be64_to_cpu(frp->l0);
}
temp = xfs_bmap_worst_indlen(bma->ip, temp);
temp2 = xfs_bmap_worst_indlen(bma->ip, temp2);
- diff = (int)(temp + temp2 - startblockval(PREV.br_startblock) -
- (bma->cur ? bma->cur->bc_private.b.allocated : 0));
+ diff = (int)(temp + temp2 -
+ (startblockval(PREV.br_startblock) -
+ (bma->cur ?
+ bma->cur->bc_private.b.allocated : 0)));
if (diff > 0) {
error = xfs_mod_fdblocks(bma->ip->i_mount,
-((int64_t)diff), false);
temp = da_new;
if (bma->cur)
temp += bma->cur->bc_private.b.allocated;
- ASSERT(temp <= da_old);
if (temp < da_old)
xfs_mod_fdblocks(bma->ip->i_mount,
(int64_t)(da_old - temp), false);
xfs_btree_readahead_ptr(cur, ptr, 1);
/* save for the next iteration of the loop */
- lptr = *ptr;
+ xfs_btree_copy_ptrs(cur, &lptr, ptr, 1);
}
/* for each buffer in the level */
if (mp->m_sb.sb_agblocks >= XFS_REFC_COW_START)
return -EOPNOTSUPP;
- error = xfs_alloc_read_agf(mp, NULL, agno, 0, &agbp);
+ INIT_LIST_HEAD(&debris);
+
+ /*
+ * In this first part, we use an empty transaction to gather up
+ * all the leftover CoW extents so that we can subsequently
+ * delete them. The empty transaction is used to avoid
+ * a buffer lock deadlock if there happens to be a loop in the
+ * refcountbt because we're allowed to re-grab a buffer that is
+ * already attached to our transaction. When we're done
+ * recording the CoW debris we cancel the (empty) transaction
+ * and everything goes away cleanly.
+ */
+ error = xfs_trans_alloc_empty(mp, &tp);
if (error)
return error;
- cur = xfs_refcountbt_init_cursor(mp, NULL, agbp, agno, NULL);
+
+ error = xfs_alloc_read_agf(mp, tp, agno, 0, &agbp);
+ if (error)
+ goto out_trans;
+ cur = xfs_refcountbt_init_cursor(mp, tp, agbp, agno, NULL);
/* Find all the leftover CoW staging extents. */
- INIT_LIST_HEAD(&debris);
memset(&low, 0, sizeof(low));
memset(&high, 0, sizeof(high));
low.rc.rc_startblock = XFS_REFC_COW_START;
if (error)
goto out_cursor;
xfs_btree_del_cursor(cur, XFS_BTREE_NOERROR);
- xfs_buf_relse(agbp);
+ xfs_trans_brelse(tp, agbp);
+ xfs_trans_cancel(tp);
/* Now iterate the list to free the leftovers */
- list_for_each_entry(rr, &debris, rr_list) {
+ list_for_each_entry_safe(rr, n, &debris, rr_list) {
/* Set up transaction. */
error = xfs_trans_alloc(mp, &M_RES(mp)->tr_write, 0, 0, 0, &tp);
if (error)
error = xfs_trans_commit(tp);
if (error)
goto out_free;
+
+ list_del(&rr->rr_list);
+ kmem_free(rr);
}
+ return error;
+out_defer:
+ xfs_defer_cancel(&dfops);
+out_trans:
+ xfs_trans_cancel(tp);
out_free:
/* Free the leftover list */
list_for_each_entry_safe(rr, n, &debris, rr_list) {
out_cursor:
xfs_btree_del_cursor(cur, XFS_BTREE_ERROR);
- xfs_buf_relse(agbp);
- goto out_free;
-
-out_defer:
- xfs_defer_cancel(&dfops);
- xfs_trans_cancel(tp);
- goto out_free;
+ xfs_trans_brelse(tp, agbp);
+ goto out_trans;
}
}
break;
default:
+ /* Local format data forks report no extents. */
+ if (ip->i_d.di_format == XFS_DINODE_FMT_LOCAL) {
+ bmv->bmv_entries = 0;
+ return 0;
+ }
if (ip->i_d.di_format != XFS_DINODE_FMT_EXTENTS &&
- ip->i_d.di_format != XFS_DINODE_FMT_BTREE &&
- ip->i_d.di_format != XFS_DINODE_FMT_LOCAL)
+ ip->i_d.di_format != XFS_DINODE_FMT_BTREE)
return -EINVAL;
if (xfs_get_extsz_hint(ip) ||
* extents.
*/
if (map[i].br_startblock == DELAYSTARTBLOCK &&
- map[i].br_startoff <= XFS_B_TO_FSB(mp, XFS_ISIZE(ip)))
+ map[i].br_startoff < XFS_B_TO_FSB(mp, XFS_ISIZE(ip)))
ASSERT((iflags & BMV_IF_DELALLOC) != 0);
if (map[i].br_startblock == HOLESTARTBLOCK &&
index = startoff >> PAGE_SHIFT;
endoff = XFS_FSB_TO_B(mp, map->br_startoff + map->br_blockcount);
- end = endoff >> PAGE_SHIFT;
+ end = (endoff - 1) >> PAGE_SHIFT;
do {
int want;
unsigned nr_pages;
unsigned int i;
- want = min_t(pgoff_t, end - index, PAGEVEC_SIZE);
+ want = min_t(pgoff_t, end - index, PAGEVEC_SIZE - 1) + 1;
nr_pages = pagevec_lookup(&pvec, inode->i_mapping, index,
want);
- /*
- * No page mapped into given range. If we are searching holes
- * and if this is the first time we got into the loop, it means
- * that the given offset is landed in a hole, return it.
- *
- * If we have already stepped through some block buffers to find
- * holes but they all contains data. In this case, the last
- * offset is already updated and pointed to the end of the last
- * mapped page, if it does not reach the endpoint to search,
- * that means there should be a hole between them.
- */
- if (nr_pages == 0) {
- /* Data search found nothing */
- if (type == DATA_OFF)
- break;
-
- ASSERT(type == HOLE_OFF);
- if (lastoff == startoff || lastoff < endoff) {
- found = true;
- *offset = lastoff;
- }
- break;
- }
-
- /*
- * At lease we found one page. If this is the first time we
- * step into the loop, and if the first page index offset is
- * greater than the given search offset, a hole was found.
- */
- if (type == HOLE_OFF && lastoff == startoff &&
- lastoff < page_offset(pvec.pages[0])) {
- found = true;
+ if (nr_pages == 0)
break;
- }
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
* file mapping. However, page->index will not change
* because we have a reference on the page.
*
- * Searching done if the page index is out of range.
- * If the current offset is not reaches the end of
- * the specified search range, there should be a hole
- * between them.
+ * If current page offset is beyond where we've ended,
+ * we've found a hole.
*/
- if (page->index > end) {
- if (type == HOLE_OFF && lastoff < endoff) {
- *offset = lastoff;
- found = true;
- }
+ if (type == HOLE_OFF && lastoff < endoff &&
+ lastoff < page_offset(pvec.pages[i])) {
+ found = true;
+ *offset = lastoff;
goto out;
}
+ /* Searching done if the page index is out of range. */
+ if (page->index > end)
+ goto out;
lock_page(page);
/*
/*
* The number of returned pages less than our desired, search
- * done. In this case, nothing was found for searching data,
- * but we found a hole behind the last offset.
+ * done.
*/
- if (nr_pages < want) {
- if (type == HOLE_OFF) {
- *offset = lastoff;
- found = true;
- }
+ if (nr_pages < want)
break;
- }
index = pvec.pages[i - 1]->index + 1;
pagevec_release(&pvec);
} while (index <= end);
+ /* No page at lastoff and we are not done - we found a hole. */
+ if (type == HOLE_OFF && lastoff < endoff) {
+ *offset = lastoff;
+ found = true;
+ }
out:
pagevec_release(&pvec);
return found;
struct xfs_fsmap dkeys[2]; /* per-dev keys */
struct xfs_getfsmap_dev handlers[XFS_GETFSMAP_DEVS];
struct xfs_getfsmap_info info = { NULL };
+ bool use_rmap;
int i;
int error = 0;
!xfs_getfsmap_is_valid_device(mp, &head->fmh_keys[1]))
return -EINVAL;
+ use_rmap = capable(CAP_SYS_ADMIN) &&
+ xfs_sb_version_hasrmapbt(&mp->m_sb);
head->fmh_entries = 0;
/* Set up our device handlers. */
memset(handlers, 0, sizeof(handlers));
handlers[0].dev = new_encode_dev(mp->m_ddev_targp->bt_dev);
- if (xfs_sb_version_hasrmapbt(&mp->m_sb))
+ if (use_rmap)
handlers[0].fn = xfs_getfsmap_datadev_rmapbt;
else
handlers[0].fn = xfs_getfsmap_datadev_bnobt;
/* optionally associate a dax device with the iomap bdev */
bdev = iomap->bdev;
if (blk_queue_dax(bdev->bd_queue))
- iomap->dax_dev = dax_get_by_host(bdev->bd_disk->disk_name);
+ iomap->dax_dev = fs_dax_get_by_host(bdev->bd_disk->disk_name);
else
iomap->dax_dev = NULL;
unsigned flags,
struct iomap *iomap)
{
- put_dax(iomap->dax_dev);
+ fs_put_dax(iomap->dax_dev);
if ((flags & IOMAP_WRITE) && iomap->type == IOMAP_DELALLOC)
return xfs_file_iomap_end_delalloc(XFS_I(inode), offset,
length, written, iomap);
#define DP_EDP_PWMGEN_BIT_COUNT 0x724
#define DP_EDP_PWMGEN_BIT_COUNT_CAP_MIN 0x725
#define DP_EDP_PWMGEN_BIT_COUNT_CAP_MAX 0x726
+# define DP_EDP_PWMGEN_BIT_COUNT_MASK (0x1f << 0)
#define DP_EDP_BACKLIGHT_CONTROL_STATUS 0x727
#define DP_EDP_BACKLIGHT_FREQ_SET 0x728
+# define DP_EDP_BACKLIGHT_FREQ_BASE_KHZ 27000
#define DP_EDP_BACKLIGHT_FREQ_CAP_MIN_MSB 0x72a
#define DP_EDP_BACKLIGHT_FREQ_CAP_MIN_MID 0x72b
#define DRIVER_RENDER 0x8000
#define DRIVER_ATOMIC 0x10000
#define DRIVER_KMS_LEGACY_CONTEXT 0x20000
+#define DRIVER_SYNCOBJ 0x40000
/**
* struct drm_driver - DRM driver structure
/** @table_lock: Protects @object_idr. */
spinlock_t table_lock;
+ /** @syncobj_idr: Mapping of sync object handles to object pointers. */
+ struct idr syncobj_idr;
+ /** @syncobj_table_lock: Protects @syncobj_idr. */
+ spinlock_t syncobj_table_lock;
+
/** @filp: Pointer to the core file structure. */
struct file *filp;
--- /dev/null
+/*
+ * Copyright © 2017 Red Hat
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ * Authors:
+ *
+ */
+#ifndef __DRM_SYNCOBJ_H__
+#define __DRM_SYNCOBJ_H__
+
+#include "linux/dma-fence.h"
+
+/**
+ * struct drm_syncobj - sync object.
+ *
+ * This structure defines a generic sync object which wraps a dma fence.
+ */
+struct drm_syncobj {
+ /**
+ * @refcount:
+ *
+ * Reference count of this object.
+ */
+ struct kref refcount;
+ /**
+ * @fence:
+ * NULL or a pointer to the fence bound to this object.
+ */
+ struct dma_fence *fence;
+ /**
+ * @file:
+ * a file backing for this syncobj.
+ */
+ struct file *file;
+};
+
+void drm_syncobj_free(struct kref *kref);
+
+/**
+ * drm_syncobj_get - acquire a syncobj reference
+ * @obj: sync object
+ *
+ * This acquires additional reference to @obj. It is illegal to call this
+ * without already holding a reference. No locks required.
+ */
+static inline void
+drm_syncobj_get(struct drm_syncobj *obj)
+{
+ kref_get(&obj->refcount);
+}
+
+/**
+ * drm_syncobj_put - release a reference to a sync object.
+ * @obj: sync object.
+ */
+static inline void
+drm_syncobj_put(struct drm_syncobj *obj)
+{
+ kref_put(&obj->refcount, drm_syncobj_free);
+}
+
+struct drm_syncobj *drm_syncobj_find(struct drm_file *file_private,
+ u32 handle);
+void drm_syncobj_replace_fence(struct drm_file *file_private,
+ struct drm_syncobj *syncobj,
+ struct dma_fence *fence);
+int drm_syncobj_fence_get(struct drm_file *file_private,
+ u32 handle,
+ struct dma_fence **fence);
+void drm_syncobj_free(struct kref *kref);
+
+#endif
#define HDMI_MAX_ELD_BYTES 128
-struct intel_hdmi_lpe_audio_eld {
- int port_id;
- int pipe_id;
- unsigned char eld_data[HDMI_MAX_ELD_BYTES];
+struct intel_hdmi_lpe_audio_port_pdata {
+ u8 eld[HDMI_MAX_ELD_BYTES];
+ int port;
+ int pipe;
+ int ls_clock;
+ bool dp_output;
};
struct intel_hdmi_lpe_audio_pdata {
- bool notify_pending;
- int tmds_clock_speed;
- bool hdmi_connected;
- bool dp_output;
- int link_rate;
- struct intel_hdmi_lpe_audio_eld eld;
- void (*notify_audio_lpe)(struct platform_device *pdev);
+ struct intel_hdmi_lpe_audio_port_pdata port[3]; /* for ports B,C,D */
+ int num_ports;
+ int num_pipes;
+
+ void (*notify_audio_lpe)(struct platform_device *pdev, int port); /* port: 0==B,1==C,2==D */
spinlock_t lpe_audio_slock;
};
/* either a GICv2 CPU interface */
gpa_t vgic_cpu_base;
/* or a number of GICv3 redistributor regions */
- gpa_t vgic_redist_base;
+ struct {
+ gpa_t vgic_redist_base;
+ gpa_t vgic_redist_free_offset;
+ };
};
/* distributor enabled */
bool kick_requeue_list);
void blk_mq_kick_requeue_list(struct request_queue *q);
void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs);
-void blk_mq_abort_requeue_list(struct request_queue *q);
void blk_mq_complete_request(struct request *rq);
bool blk_mq_queue_stopped(struct request_queue *q);
*/
s64 min_value;
u64 max_value;
+ u32 min_align;
+ u32 aux_off;
+ u32 aux_off_align;
};
enum bpf_stack_slot_type {
struct bpf_prog *prog; /* eBPF program being verified */
struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
int stack_size; /* number of states to be processed */
+ bool strict_alignment; /* perform strict pointer alignment checks */
struct bpf_verifier_state cur_state; /* current verifier state */
struct bpf_verifier_state_list **explored_states; /* search pruning optimization */
const struct bpf_ext_analyzer_ops *analyzer_ops; /* external analyzer ops */
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/string.h>
+
#ifdef CONFIG_CEPH_LIB_PRETTYDEBUG
/*
*/
# if defined(DEBUG) || defined(CONFIG_DYNAMIC_DEBUG)
-extern const char *ceph_file_part(const char *s, int len);
# define dout(fmt, ...) \
pr_debug("%.*s %12.12s:%-4d : " fmt, \
8 - (int)sizeof(KBUILD_MODNAME), " ", \
- ceph_file_part(__FILE__, sizeof(__FILE__)), \
- __LINE__, ##__VA_ARGS__)
+ kbasename(__FILE__), __LINE__, ##__VA_ARGS__)
# else
/* faux printk call just to see any compiler warnings. */
# define dout(fmt, ...) do { \
void **, pfn_t *);
};
+#if IS_ENABLED(CONFIG_DAX)
+struct dax_device *dax_get_by_host(const char *host);
+void put_dax(struct dax_device *dax_dev);
+#else
+static inline struct dax_device *dax_get_by_host(const char *host)
+{
+ return NULL;
+}
+
+static inline void put_dax(struct dax_device *dax_dev)
+{
+}
+#endif
+
int bdev_dax_pgoff(struct block_device *, sector_t, size_t, pgoff_t *pgoff);
#if IS_ENABLED(CONFIG_FS_DAX)
int __bdev_dax_supported(struct super_block *sb, int blocksize);
{
return __bdev_dax_supported(sb, blocksize);
}
+
+static inline struct dax_device *fs_dax_get_by_host(const char *host)
+{
+ return dax_get_by_host(host);
+}
+
+static inline void fs_put_dax(struct dax_device *dax_dev)
+{
+ put_dax(dax_dev);
+}
+
#else
static inline int bdev_dax_supported(struct super_block *sb, int blocksize)
{
return -EOPNOTSUPP;
}
-#endif
-#if IS_ENABLED(CONFIG_DAX)
-struct dax_device *dax_get_by_host(const char *host);
-void put_dax(struct dax_device *dax_dev);
-#else
-static inline struct dax_device *dax_get_by_host(const char *host)
+static inline struct dax_device *fs_dax_get_by_host(const char *host)
{
return NULL;
}
-static inline void put_dax(struct dax_device *dax_dev)
+static inline void fs_put_dax(struct dax_device *dax_dev)
{
}
#endif
.off = OFF, \
.imm = IMM })
+/* Unconditional jumps, goto pc + off16 */
+
+#define BPF_JMP_A(OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_JA, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = 0 })
+
/* Function call */
#define BPF_EMIT_CALL(FUNC) \
static inline netdev_features_t vlan_features_check(const struct sk_buff *skb,
netdev_features_t features)
{
- if (skb_vlan_tagged_multi(skb))
- features = netdev_intersect_features(features,
- NETIF_F_SG |
- NETIF_F_HIGHDMA |
- NETIF_F_FRAGLIST |
- NETIF_F_HW_CSUM |
- NETIF_F_HW_VLAN_CTAG_TX |
- NETIF_F_HW_VLAN_STAG_TX);
+ if (skb_vlan_tagged_multi(skb)) {
+ /* In the case of multi-tagged packets, use a direct mask
+ * instead of using netdev_interesect_features(), to make
+ * sure that only devices supporting NETIF_F_HW_CSUM will
+ * have checksum offloading support.
+ */
+ features &= NETIF_F_SG | NETIF_F_HIGHDMA | NETIF_F_HW_CSUM |
+ NETIF_F_FRAGLIST | NETIF_F_HW_VLAN_CTAG_TX |
+ NETIF_F_HW_VLAN_STAG_TX;
+ }
return features;
}
int write, void __user *buffer,
size_t *length, loff_t *ppos);
#endif
+extern void wait_for_kprobe_optimizer(void);
+#else
+static inline void wait_for_kprobe_optimizer(void) { }
#endif /* CONFIG_OPTPROBES */
#ifdef CONFIG_KPROBES_ON_FTRACE
extern void kprobe_ftrace_handler(unsigned long ip, unsigned long parent_ip,
};
enum {
- CQE_RSS_HTYPE_IP = 0x3 << 6,
- CQE_RSS_HTYPE_L4 = 0x3 << 2,
+ CQE_RSS_HTYPE_IP = 0x3 << 2,
+ /* cqe->rss_hash_type[3:2] - IP destination selected for hash
+ * (00 = none, 01 = IPv4, 10 = IPv6, 11 = Reserved)
+ */
+ CQE_RSS_HTYPE_L4 = 0x3 << 6,
+ /* cqe->rss_hash_type[7:6] - L4 destination selected for hash
+ * (00 = none, 01 = TCP. 10 = UDP, 11 = IPSEC.SPI
+ */
};
enum {
typedef void (*mlx5_cmd_cbk_t)(int status, void *context);
+enum {
+ MLX5_CMD_ENT_STATE_PENDING_COMP,
+};
+
struct mlx5_cmd_work_ent {
+ unsigned long state;
struct mlx5_cmd_msg *in;
struct mlx5_cmd_msg *out;
void *uout;
void mlx5_rsc_event(struct mlx5_core_dev *dev, u32 rsn, int event_type);
void mlx5_srq_event(struct mlx5_core_dev *dev, u32 srqn, int event_type);
struct mlx5_core_srq *mlx5_core_get_srq(struct mlx5_core_dev *dev, u32 srqn);
-void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec);
+void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool forced);
void mlx5_cq_event(struct mlx5_core_dev *dev, u32 cqn, int event_type);
int mlx5_create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq, u8 vecidx,
int nent, u64 mask, const char *name,
int max_fte;
u32 level;
u32 flags;
- u32 underlay_qpn;
};
struct mlx5_flow_table *
void mlx5_fc_destroy(struct mlx5_core_dev *dev, struct mlx5_fc *counter);
void mlx5_fc_query_cached(struct mlx5_fc *counter,
u64 *bytes, u64 *packets, u64 *lastuse);
+int mlx5_fs_add_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);
+int mlx5_fs_remove_rx_underlay_qpn(struct mlx5_core_dev *dev, u32 underlay_qpn);
+
#endif
int dev_get_phys_port_name(struct net_device *dev,
char *name, size_t len);
int dev_change_proto_down(struct net_device *dev, bool proto_down);
-int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
- int fd, u32 flags);
struct sk_buff *validate_xmit_skb_list(struct sk_buff *skb, struct net_device *dev);
struct sk_buff *dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
struct netdev_queue *txq, int *ret);
+
+typedef int (*xdp_op_t)(struct net_device *dev, struct netdev_xdp *xdp);
+int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
+ int fd, u32 flags);
+bool __dev_xdp_attached(struct net_device *dev, xdp_op_t xdp_op);
+
int __dev_forward_skb(struct net_device *dev, struct sk_buff *skb);
int dev_forward_skb(struct net_device *dev, struct sk_buff *skb);
bool is_skb_forwardable(const struct net_device *dev,
int xt_target_to_user(const struct xt_entry_target *t,
struct xt_entry_target __user *u);
int xt_data_to_user(void __user *dst, const void *src,
- int usersize, int size);
+ int usersize, int size, int aligned_size);
void *xt_copy_counters_from_user(const void __user *user, unsigned int len,
struct xt_counters_info *info, bool compat);
/* True if the target is not a standard target */
#define INVALID_TARGET (info->target < -NUM_STANDARD_TARGETS || info->target >= 0)
+static inline bool ebt_invalid_target(int target)
+{
+ return (target < -NUM_STANDARD_TARGETS || target >= 0);
+}
+
#endif
/* FC Port role bitmask - can merge with FC Port Roles in fc transport */
#define FC_PORT_ROLE_NVME_INITIATOR 0x10
-#define FC_PORT_ROLE_NVME_TARGET 0x11
-#define FC_PORT_ROLE_NVME_DISCOVERY 0x12
+#define FC_PORT_ROLE_NVME_TARGET 0x20
+#define FC_PORT_ROLE_NVME_DISCOVERY 0x40
/**
* sequence in one LLDD operation. Errors during Data
* sequence transmit must not allow RSP sequence to be sent.
*/
- NVMET_FCTGTFEAT_NEEDS_CMD_CPUSCHED = (1 << 1),
- /* Bit 1: When 0, the LLDD will deliver FCP CMD
- * on the CPU it should be affinitized to. Thus work will
- * be scheduled on the cpu received on. When 1, the LLDD
- * may not deliver the CMD on the CPU it should be worked
- * on. The transport should pick a cpu to schedule the work
- * on.
- */
- NVMET_FCTGTFEAT_CMD_IN_ISR = (1 << 2),
+ NVMET_FCTGTFEAT_CMD_IN_ISR = (1 << 1),
/* Bit 2: When 0, the LLDD is calling the cmd rcv handler
* in a non-isr context, allowing the transport to finish
* op completion in the calling context. When 1, the LLDD
* requiring the transport to transition to a workqueue
* for op completion.
*/
- NVMET_FCTGTFEAT_OPDONE_IN_ISR = (1 << 3),
+ NVMET_FCTGTFEAT_OPDONE_IN_ISR = (1 << 2),
/* Bit 3: When 0, the LLDD is calling the op done handler
* in a non-isr context, allowing the transport to finish
* op completion in the calling context. When 1, the LLDD
#include <linux/ioport.h>
#include <linux/of.h>
-typedef int const (*of_irq_init_cb_t)(struct device_node *, struct device_node *);
+typedef int (*of_irq_init_cb_t)(struct device_node *, struct device_node *);
/*
* Workarounds only applied to 32bit powermac machines
const char *bus_id,
struct device *parent);
+extern int of_platform_device_destroy(struct device *dev, void *data);
extern int of_platform_bus_probe(struct device_node *root,
const struct of_device_id *matches,
struct device *parent);
PCI_DEV_FLAGS_BRIDGE_XLATE_ROOT = (__force pci_dev_flags_t) (1 << 9),
/* Do not use FLR even if device advertises PCI_AF_CAP */
PCI_DEV_FLAGS_NO_FLR_RESET = (__force pci_dev_flags_t) (1 << 10),
+ /*
+ * Resume before calling the driver's system suspend hooks, disabling
+ * the direct_complete optimization.
+ */
+ PCI_DEV_FLAGS_NEEDS_RESUME = (__force pci_dev_flags_t) (1 << 11),
};
enum pci_irq_reroute_variant {
unsigned int max_vecs, unsigned int flags,
const struct irq_affinity *aff_desc)
{
- if (min_vecs > 1)
- return -EINVAL;
- return 1;
+ if ((flags & PCI_IRQ_LEGACY) && min_vecs == 1 && dev->irq)
+ return 1;
+ return -ENOSPC;
}
static inline void pci_free_irq_vectors(struct pci_dev *dev)
/* Board specific data */
struct omap_dss_board_info {
- const char *default_display_name;
int (*dsi_enable_pads)(int dsi_id, unsigned int lane_mask);
void (*dsi_disable_pads)(int dsi_id, unsigned int lane_mask);
int (*set_min_bus_tput)(struct device *dev, unsigned long r);
unsigned long addr, unsigned long data);
extern void ptrace_notify(int exit_code);
extern void __ptrace_link(struct task_struct *child,
- struct task_struct *new_parent);
+ struct task_struct *new_parent,
+ const struct cred *ptracer_cred);
extern void __ptrace_unlink(struct task_struct *child);
extern void exit_ptrace(struct task_struct *tracer, struct list_head *dead);
#define PTRACE_MODE_READ 0x01
if (unlikely(ptrace) && current->ptrace) {
child->ptrace = current->ptrace;
- __ptrace_link(child, current->parent);
+ __ptrace_link(child, current->parent, current->ptracer_cred);
if (child->ptrace & PT_SEIZED)
task_set_jobctl_pending(child, JOBCTL_TRAP_STOP);
set_tsk_thread_flag(child, TIF_SIGPENDING);
}
+ else
+ child->ptracer_cred = NULL;
}
/**
void serdev_device_close(struct serdev_device *);
unsigned int serdev_device_set_baudrate(struct serdev_device *, unsigned int);
void serdev_device_set_flow_control(struct serdev_device *, bool);
+int serdev_device_write_buf(struct serdev_device *, const unsigned char *, size_t);
void serdev_device_wait_until_sent(struct serdev_device *, long);
int serdev_device_get_tiocm(struct serdev_device *);
int serdev_device_set_tiocm(struct serdev_device *, int, int);
return 0;
}
static inline void serdev_device_set_flow_control(struct serdev_device *sdev, bool enable) {}
+static inline int serdev_device_write_buf(struct serdev_device *serdev,
+ const unsigned char *buf,
+ size_t count)
+{
+ return -ENODEV;
+}
static inline void serdev_device_wait_until_sent(struct serdev_device *sdev, long timeout) {}
static inline int serdev_device_get_tiocm(struct serdev_device *serdev)
{
struct device *serdev_tty_port_register(struct tty_port *port,
struct device *parent,
struct tty_driver *drv, int idx);
-void serdev_tty_port_unregister(struct tty_port *port);
+int serdev_tty_port_unregister(struct tty_port *port);
#else
static inline struct device *serdev_tty_port_register(struct tty_port *port,
struct device *parent,
{
return ERR_PTR(-ENODEV);
}
-static inline void serdev_tty_port_unregister(struct tty_port *port) {}
-#endif /* CONFIG_SERIAL_DEV_CTRL_TTYPORT */
-
-static inline int serdev_device_write_buf(struct serdev_device *serdev,
- const unsigned char *data,
- size_t count)
+static inline int serdev_tty_port_unregister(struct tty_port *port)
{
- return serdev_device_write(serdev, data, count, 0);
+ return -ENODEV;
}
+#endif /* CONFIG_SERIAL_DEV_CTRL_TTYPORT */
#endif /*_LINUX_SERDEV_H */
#ifndef __LINUX_SOC_RENESAS_RCAR_RST_H__
#define __LINUX_SOC_RENESAS_RCAR_RST_H__
+#if defined(CONFIG_ARCH_RCAR_GEN1) || defined(CONFIG_ARCH_RCAR_GEN2) || \
+ defined(CONFIG_ARCH_R8A7795) || defined(CONFIG_ARCH_R8A7796)
int rcar_rst_read_mode_pins(u32 *mode);
+#else
+static inline int rcar_rst_read_mode_pins(u32 *mode) { return -ENODEV; }
+#endif
#endif /* __LINUX_SOC_RENESAS_RCAR_RST_H__ */
struct tty_driver *driver, unsigned index,
struct device *device, void *drvdata,
const struct attribute_group **attr_grp);
+extern struct device *tty_port_register_device_serdev(struct tty_port *port,
+ struct tty_driver *driver, unsigned index,
+ struct device *device);
+extern struct device *tty_port_register_device_attr_serdev(struct tty_port *port,
+ struct tty_driver *driver, unsigned index,
+ struct device *device, void *drvdata,
+ const struct attribute_group **attr_grp);
+extern void tty_port_unregister_device(struct tty_port *port,
+ struct tty_driver *driver, unsigned index);
extern int tty_port_alloc_xmit_buf(struct tty_port *port);
extern void tty_port_free_xmit_buf(struct tty_port *port);
extern void tty_port_destroy(struct tty_port *port);
unsigned rh_registered:1;/* is root hub registered? */
unsigned rh_pollable:1; /* may we poll the root hub? */
unsigned msix_enabled:1; /* driver has MSI-X enabled? */
+ unsigned msi_enabled:1; /* driver has MSI enabled? */
unsigned remove_phy:1; /* auto-remove USB phy */
/* The next flag is a stopgap, to be removed when all the HCDs
};
extern int usbnet_generic_cdc_bind(struct usbnet *, struct usb_interface *);
+extern int usbnet_ether_cdc_bind(struct usbnet *dev, struct usb_interface *intf);
extern int usbnet_cdc_bind(struct usbnet *, struct usb_interface *);
extern void usbnet_cdc_unbind(struct usbnet *, struct usb_interface *);
extern void usbnet_cdc_status(struct usbnet *, struct urb *);
};
};
+struct dst_metrics {
+ u32 metrics[RTAX_MAX];
+ atomic_t refcnt;
+};
+extern const struct dst_metrics dst_default_metrics;
+
u32 *dst_cow_metrics_generic(struct dst_entry *dst, unsigned long old);
-extern const u32 dst_default_metrics[];
#define DST_METRICS_READ_ONLY 0x1UL
+#define DST_METRICS_REFCOUNTED 0x2UL
#define DST_METRICS_FLAGS 0x3UL
#define __DST_METRICS_PTR(Y) \
((u32 *)((Y) & ~DST_METRICS_FLAGS))
__be32 fib_prefsrc;
u32 fib_tb_id;
u32 fib_priority;
- u32 *fib_metrics;
-#define fib_mtu fib_metrics[RTAX_MTU-1]
-#define fib_window fib_metrics[RTAX_WINDOW-1]
-#define fib_rtt fib_metrics[RTAX_RTT-1]
-#define fib_advmss fib_metrics[RTAX_ADVMSS-1]
+ struct dst_metrics *fib_metrics;
+#define fib_mtu fib_metrics->metrics[RTAX_MTU-1]
+#define fib_window fib_metrics->metrics[RTAX_WINDOW-1]
+#define fib_rtt fib_metrics->metrics[RTAX_RTT-1]
+#define fib_advmss fib_metrics->metrics[RTAX_ADVMSS-1]
int fib_nhs;
#ifdef CONFIG_IP_ROUTE_MULTIPATH
int fib_weight;
#ifndef _NF_CONNTRACK_HELPER_H
#define _NF_CONNTRACK_HELPER_H
+#include <linux/refcount.h>
#include <net/netfilter/nf_conntrack.h>
#include <net/netfilter/nf_conntrack_extend.h>
#include <net/netfilter/nf_conntrack_expect.h>
struct hlist_node hnode; /* Internal use. */
char name[NF_CT_HELPER_NAME_LEN]; /* name of the module */
+ refcount_t refcnt;
struct module *me; /* pointer to self */
const struct nf_conntrack_expect_policy *expect_policy;
struct nf_conntrack_helper *nf_conntrack_helper_try_module_get(const char *name,
u16 l3num,
u8 protonum);
+void nf_conntrack_helper_put(struct nf_conntrack_helper *helper);
+
void nf_ct_helper_init(struct nf_conntrack_helper *helper,
u16 l3num, u16 protonum, const char *name,
u16 default_port, u16 spec_port, u32 id,
int nft_data_init(const struct nft_ctx *ctx,
struct nft_data *data, unsigned int size,
struct nft_data_desc *desc, const struct nlattr *nla);
-void nft_data_uninit(const struct nft_data *data, enum nft_data_types type);
+void nft_data_release(const struct nft_data *data, enum nft_data_types type);
int nft_data_dump(struct sk_buff *skb, int attr, const struct nft_data *data,
enum nft_data_types type, unsigned int len);
#include <linux/types.h>
#include <net/act_api.h>
+#include <linux/tc_act/tc_csum.h>
struct tcf_csum {
struct tc_action common;
};
#define to_tcf_csum(a) ((struct tcf_csum *)a)
+static inline bool is_tcf_csum(const struct tc_action *a)
+{
+#ifdef CONFIG_NET_CLS_ACT
+ if (a->ops && a->ops->type == TCA_ACT_CSUM)
+ return true;
+#endif
+ return false;
+}
+
+static inline u32 tcf_csum_update_flags(const struct tc_action *a)
+{
+ return to_tcf_csum(a)->update_flags;
+}
+
#endif /* __NET_TC_CSUM_H */
/* sysctl_net_x25.c */
#ifdef CONFIG_SYSCTL
-void x25_register_sysctl(void);
+int x25_register_sysctl(void);
void x25_unregister_sysctl(void);
#else
-static inline void x25_register_sysctl(void) {};
+static inline int x25_register_sysctl(void) { return 0; };
static inline void x25_unregister_sysctl(void) {};
#endif /* CONFIG_SYSCTL */
struct flow_cache_object flo;
struct xfrm_policy *pols[XFRM_POLICY_TYPE_MAX];
int num_pols, num_xfrms;
-#ifdef CONFIG_XFRM_SUB_POLICY
- struct flowi *origin;
- struct xfrm_selector *partner;
-#endif
u32 xfrm_genid;
u32 policy_genid;
u32 route_mtu_cached;
dst_release(xdst->route);
if (likely(xdst->u.dst.xfrm))
xfrm_state_put(xdst->u.dst.xfrm);
-#ifdef CONFIG_XFRM_SUB_POLICY
- kfree(xdst->origin);
- xdst->origin = NULL;
- kfree(xdst->partner);
- xdst->partner = NULL;
-#endif
}
#endif
#define DRM_CAP_ADDFB2_MODIFIERS 0x10
#define DRM_CAP_PAGE_FLIP_TARGET 0x11
#define DRM_CAP_CRTC_IN_VBLANK_EVENT 0x12
+#define DRM_CAP_SYNCOBJ 0x13
/** DRM_IOCTL_GET_CAP ioctl argument type */
struct drm_get_cap {
__s32 fd;
};
+struct drm_syncobj_create {
+ __u32 handle;
+ __u32 flags;
+};
+
+struct drm_syncobj_destroy {
+ __u32 handle;
+ __u32 pad;
+};
+
+#define DRM_SYNCOBJ_FD_TO_HANDLE_FLAGS_IMPORT_SYNC_FILE (1 << 0)
+#define DRM_SYNCOBJ_HANDLE_TO_FD_FLAGS_EXPORT_SYNC_FILE (1 << 0)
+struct drm_syncobj_handle {
+ __u32 handle;
+ __u32 flags;
+
+ __s32 fd;
+ __u32 pad;
+};
+
#if defined(__cplusplus)
}
#endif
#define DRM_IOCTL_MODE_CREATEPROPBLOB DRM_IOWR(0xBD, struct drm_mode_create_blob)
#define DRM_IOCTL_MODE_DESTROYPROPBLOB DRM_IOWR(0xBE, struct drm_mode_destroy_blob)
+#define DRM_IOCTL_SYNCOBJ_CREATE DRM_IOWR(0xBF, struct drm_syncobj_create)
+#define DRM_IOCTL_SYNCOBJ_DESTROY DRM_IOWR(0xC0, struct drm_syncobj_destroy)
+#define DRM_IOCTL_SYNCOBJ_HANDLE_TO_FD DRM_IOWR(0xC1, struct drm_syncobj_handle)
+#define DRM_IOCTL_SYNCOBJ_FD_TO_HANDLE DRM_IOWR(0xC2, struct drm_syncobj_handle)
+
/**
* Device specific ioctls should only be in their respective headers
* The device specific ioctl range is from 0x40 to 0x9f.
*/
#define I915_PARAM_HAS_EXEC_FENCE 44
+/* Query whether DRM_I915_GEM_EXECBUFFER2 supports the ability to capture
+ * user specified bufffers for post-mortem debugging of GPU hangs. See
+ * EXEC_OBJECT_CAPTURE.
+ */
+#define I915_PARAM_HAS_EXEC_CAPTURE 45
+
typedef struct drm_i915_getparam {
__s32 param;
/*
#define I915_GEM_DOMAIN_VERTEX 0x00000020
/** GTT domain - aperture and scanout */
#define I915_GEM_DOMAIN_GTT 0x00000040
+/** WC domain - uncached access */
+#define I915_GEM_DOMAIN_WC 0x00000080
/** @} */
struct drm_i915_gem_exec_object {
* I915_PARAM_HAS_EXEC_FENCE to order execbufs and execute them asynchronously.
*/
#define EXEC_OBJECT_ASYNC (1<<6)
+/* Request that the contents of this execobject be copied into the error
+ * state upon a GPU hang involving this batch for post-mortem debugging.
+ * These buffers are recorded in no particular order as "user" in
+ * /sys/class/drm/cardN/error. Query I915_PARAM_HAS_EXEC_CAPTURE to see
+ * if the kernel supports this flag.
+ */
+#define EXEC_OBJECT_CAPTURE (1<<7)
/* All remaining bits are MBZ and RESERVED FOR FUTURE USE */
-#define __EXEC_OBJECT_UNKNOWN_FLAGS -(EXEC_OBJECT_ASYNC<<1)
+#define __EXEC_OBJECT_UNKNOWN_FLAGS -(EXEC_OBJECT_CAPTURE<<1)
__u64 flags;
union {
#define DRM_OMAP_GET_PARAM 0x00
#define DRM_OMAP_SET_PARAM 0x01
#define DRM_OMAP_GEM_NEW 0x03
-#define DRM_OMAP_GEM_CPU_PREP 0x04
-#define DRM_OMAP_GEM_CPU_FINI 0x05
+#define DRM_OMAP_GEM_CPU_PREP 0x04 /* Deprecated, to be removed */
+#define DRM_OMAP_GEM_CPU_FINI 0x05 /* Deprecated, to be removed */
#define DRM_OMAP_GEM_INFO 0x06
#define DRM_OMAP_NUM_IOCTLS 0x07
*/
#define BPF_F_ALLOW_OVERRIDE (1U << 0)
+/* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
+ * verifier will perform strict alignment checking as if the kernel
+ * has been built with CONFIG_EFFICIENT_UNALIGNED_ACCESS not set,
+ * and NET_IP_ALIGN defined to 2.
+ */
+#define BPF_F_STRICT_ALIGNMENT (1U << 0)
+
#define BPF_PSEUDO_MAP_FD 1
/* flags for BPF_MAP_UPDATE_ELEM command */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied buffer */
__u32 kern_version; /* checked when prog_type=kprobe */
+ __u32 prog_flags;
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
/* XDP section */
#define XDP_FLAGS_UPDATE_IF_NOEXIST (1U << 0)
-#define XDP_FLAGS_SKB_MODE (2U << 0)
+#define XDP_FLAGS_SKB_MODE (1U << 1)
+#define XDP_FLAGS_DRV_MODE (1U << 2)
#define XDP_FLAGS_MASK (XDP_FLAGS_UPDATE_IF_NOEXIST | \
- XDP_FLAGS_SKB_MODE)
+ XDP_FLAGS_SKB_MODE | \
+ XDP_FLAGS_DRV_MODE)
+
+/* These are stored into IFLA_XDP_ATTACHED on dump. */
+enum {
+ XDP_ATTACHED_NONE = 0,
+ XDP_ATTACHED_DRV,
+ XDP_ATTACHED_SKB,
+};
enum {
IFLA_XDP_UNSPEC,
*/
#define USB_MAXCHILDREN 31
+/* See USB 3.1 spec Table 10-5 */
+#define USB_SS_MAXPORTS 15
+
/*
* Hub request types
*/
#define WINCONx_ENWIN_F (1 << 0)
/* SHADOWCON */
+#define SHADOWCON_PROTECT_MASK GENMASK(14, 10)
#define SHADOWCON_Wx_PROTECT(n) (1 << (10 + (n)))
/* VIDOSDxD */
array->map.key_size = attr->key_size;
array->map.value_size = attr->value_size;
array->map.max_entries = attr->max_entries;
+ array->map.map_flags = attr->map_flags;
array->elem_size = elem_size;
if (!percpu)
trie->map.key_size = attr->key_size;
trie->map.value_size = attr->value_size;
trie->map.max_entries = attr->max_entries;
+ trie->map.map_flags = attr->map_flags;
trie->data_size = attr->key_size -
offsetof(struct bpf_lpm_trie_key, data);
trie->max_prefixlen = trie->data_size * 8;
smap->map.key_size = attr->key_size;
smap->map.value_size = value_size;
smap->map.max_entries = attr->max_entries;
+ smap->map.map_flags = attr->map_flags;
smap->n_buckets = n_buckets;
smap->map.pages = round_up(cost, PAGE_SIZE) >> PAGE_SHIFT;
EXPORT_SYMBOL_GPL(bpf_prog_get_type);
/* last field in 'union bpf_attr' used by this command */
-#define BPF_PROG_LOAD_LAST_FIELD kern_version
+#define BPF_PROG_LOAD_LAST_FIELD prog_flags
static int bpf_prog_load(union bpf_attr *attr)
{
if (CHECK_ATTR(BPF_PROG_LOAD))
return -EINVAL;
+ if (attr->prog_flags & ~BPF_F_STRICT_ALIGNMENT)
+ return -EINVAL;
+
/* copy eBPF program license from user space */
if (strncpy_from_user(license, u64_to_user_ptr(attr->license),
sizeof(license) - 1) < 0)
struct bpf_verifier_stack_elem *next;
};
-#define BPF_COMPLEXITY_LIMIT_INSNS 65536
+#define BPF_COMPLEXITY_LIMIT_INSNS 98304
#define BPF_COMPLEXITY_LIMIT_STACK 1024
#define BPF_MAP_PTR_POISON ((void *)0xeB9F + POISON_POINTER_DELTA)
if (reg->max_value != BPF_REGISTER_MAX_RANGE)
verbose(",max_value=%llu",
(unsigned long long)reg->max_value);
+ if (reg->min_align)
+ verbose(",min_align=%u", reg->min_align);
+ if (reg->aux_off)
+ verbose(",aux_off=%u", reg->aux_off);
+ if (reg->aux_off_align)
+ verbose(",aux_off_align=%u", reg->aux_off_align);
}
for (i = 0; i < MAX_BPF_STACK; i += BPF_REG_SIZE) {
if (state->stack_slot_type[i] == STACK_SPILL)
BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5
};
+static void mark_reg_not_init(struct bpf_reg_state *regs, u32 regno)
+{
+ BUG_ON(regno >= MAX_BPF_REG);
+
+ memset(®s[regno], 0, sizeof(regs[regno]));
+ regs[regno].type = NOT_INIT;
+ regs[regno].min_value = BPF_REGISTER_MIN_RANGE;
+ regs[regno].max_value = BPF_REGISTER_MAX_RANGE;
+}
+
static void init_reg_state(struct bpf_reg_state *regs)
{
int i;
- for (i = 0; i < MAX_BPF_REG; i++) {
- regs[i].type = NOT_INIT;
- regs[i].imm = 0;
- regs[i].min_value = BPF_REGISTER_MIN_RANGE;
- regs[i].max_value = BPF_REGISTER_MAX_RANGE;
- }
+ for (i = 0; i < MAX_BPF_REG; i++)
+ mark_reg_not_init(regs, i);
/* frame pointer */
regs[BPF_REG_FP].type = FRAME_PTR;
{
regs[regno].min_value = BPF_REGISTER_MIN_RANGE;
regs[regno].max_value = BPF_REGISTER_MAX_RANGE;
+ regs[regno].min_align = 0;
}
static void mark_reg_unknown_value_and_range(struct bpf_reg_state *regs,
}
static int check_pkt_ptr_alignment(const struct bpf_reg_state *reg,
- int off, int size)
+ int off, int size, bool strict)
{
- if (reg->id && size != 1) {
- verbose("Unknown alignment. Only byte-sized access allowed in packet access.\n");
- return -EACCES;
+ int ip_align;
+ int reg_off;
+
+ /* Byte size accesses are always allowed. */
+ if (!strict || size == 1)
+ return 0;
+
+ reg_off = reg->off;
+ if (reg->id) {
+ if (reg->aux_off_align % size) {
+ verbose("Packet access is only %u byte aligned, %d byte access not allowed\n",
+ reg->aux_off_align, size);
+ return -EACCES;
+ }
+ reg_off += reg->aux_off;
}
- /* skb->data is NET_IP_ALIGN-ed */
- if ((NET_IP_ALIGN + reg->off + off) % size != 0) {
+ /* For platforms that do not have a Kconfig enabling
+ * CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS the value of
+ * NET_IP_ALIGN is universally set to '2'. And on platforms
+ * that do set CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, we get
+ * to this code only in strict mode where we want to emulate
+ * the NET_IP_ALIGN==2 checking. Therefore use an
+ * unconditional IP align value of '2'.
+ */
+ ip_align = 2;
+ if ((ip_align + reg_off + off) % size != 0) {
verbose("misaligned packet access off %d+%d+%d size %d\n",
- NET_IP_ALIGN, reg->off, off, size);
+ ip_align, reg_off, off, size);
return -EACCES;
}
}
static int check_val_ptr_alignment(const struct bpf_reg_state *reg,
- int size)
+ int size, bool strict)
{
- if (size != 1) {
+ if (strict && size != 1) {
verbose("Unknown alignment. Only byte-sized access allowed in value access.\n");
return -EACCES;
}
return 0;
}
-static int check_ptr_alignment(const struct bpf_reg_state *reg,
+static int check_ptr_alignment(struct bpf_verifier_env *env,
+ const struct bpf_reg_state *reg,
int off, int size)
{
+ bool strict = env->strict_alignment;
+
switch (reg->type) {
case PTR_TO_PACKET:
- return IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ? 0 :
- check_pkt_ptr_alignment(reg, off, size);
+ return check_pkt_ptr_alignment(reg, off, size, strict);
case PTR_TO_MAP_VALUE_ADJ:
- return IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ? 0 :
- check_val_ptr_alignment(reg, size);
+ return check_val_ptr_alignment(reg, size, strict);
default:
if (off % size != 0) {
verbose("misaligned access off %d size %d\n",
if (size < 0)
return size;
- err = check_ptr_alignment(reg, off, size);
+ err = check_ptr_alignment(env, reg, off, size);
if (err)
return err;
value_regno);
/* note that reg.[id|off|range] == 0 */
state->regs[value_regno].type = reg_type;
+ state->regs[value_regno].aux_off = 0;
+ state->regs[value_regno].aux_off_align = 0;
}
} else if (reg->type == FRAME_PTR || reg->type == PTR_TO_STACK) {
struct bpf_verifier_state *state = &env->cur_state;
const struct bpf_func_proto *fn = NULL;
struct bpf_reg_state *regs = state->regs;
- struct bpf_reg_state *reg;
struct bpf_call_arg_meta meta;
bool changes_data;
int i, err;
}
/* reset caller saved regs */
- for (i = 0; i < CALLER_SAVED_REGS; i++) {
- reg = regs + caller_saved[i];
- reg->type = NOT_INIT;
- reg->imm = 0;
- }
+ for (i = 0; i < CALLER_SAVED_REGS; i++)
+ mark_reg_not_init(regs, caller_saved[i]);
/* update return register */
if (fn->ret_type == RET_INTEGER) {
*/
dst_reg->off += imm;
} else {
+ bool had_id;
+
if (src_reg->type == PTR_TO_PACKET) {
/* R6=pkt(id=0,off=0,r=62) R7=imm22; r7 += r6 */
tmp_reg = *dst_reg; /* save r7 state */
src_reg->imm);
return -EACCES;
}
+
+ had_id = (dst_reg->id != 0);
+
/* dst_reg stays as pkt_ptr type and since some positive
* integer value was added to the pointer, increment its 'id'
*/
dst_reg->id = ++env->id_gen;
- /* something was added to pkt_ptr, set range and off to zero */
+ /* something was added to pkt_ptr, set range to zero */
+ dst_reg->aux_off += dst_reg->off;
dst_reg->off = 0;
dst_reg->range = 0;
+ if (had_id)
+ dst_reg->aux_off_align = min(dst_reg->aux_off_align,
+ src_reg->min_align);
+ else
+ dst_reg->aux_off_align = src_reg->min_align;
}
return 0;
}
reg->min_value = BPF_REGISTER_MIN_RANGE;
}
+static u32 calc_align(u32 imm)
+{
+ if (!imm)
+ return 1U << 31;
+ return imm - ((imm - 1) & imm);
+}
+
static void adjust_reg_min_max_vals(struct bpf_verifier_env *env,
struct bpf_insn *insn)
{
s64 min_val = BPF_REGISTER_MIN_RANGE;
u64 max_val = BPF_REGISTER_MAX_RANGE;
u8 opcode = BPF_OP(insn->code);
+ u32 dst_align, src_align;
dst_reg = ®s[insn->dst_reg];
+ src_align = 0;
if (BPF_SRC(insn->code) == BPF_X) {
check_reg_overflow(®s[insn->src_reg]);
min_val = regs[insn->src_reg].min_value;
regs[insn->src_reg].type != UNKNOWN_VALUE) {
min_val = BPF_REGISTER_MIN_RANGE;
max_val = BPF_REGISTER_MAX_RANGE;
+ src_align = 0;
+ } else {
+ src_align = regs[insn->src_reg].min_align;
}
} else if (insn->imm < BPF_REGISTER_MAX_RANGE &&
(s64)insn->imm > BPF_REGISTER_MIN_RANGE) {
min_val = max_val = insn->imm;
+ src_align = calc_align(insn->imm);
}
+ dst_align = dst_reg->min_align;
+
/* We don't know anything about what was done to this register, mark it
* as unknown.
*/
dst_reg->min_value += min_val;
if (dst_reg->max_value != BPF_REGISTER_MAX_RANGE)
dst_reg->max_value += max_val;
+ dst_reg->min_align = min(src_align, dst_align);
break;
case BPF_SUB:
if (dst_reg->min_value != BPF_REGISTER_MIN_RANGE)
dst_reg->min_value -= min_val;
if (dst_reg->max_value != BPF_REGISTER_MAX_RANGE)
dst_reg->max_value -= max_val;
+ dst_reg->min_align = min(src_align, dst_align);
break;
case BPF_MUL:
if (dst_reg->min_value != BPF_REGISTER_MIN_RANGE)
dst_reg->min_value *= min_val;
if (dst_reg->max_value != BPF_REGISTER_MAX_RANGE)
dst_reg->max_value *= max_val;
+ dst_reg->min_align = max(src_align, dst_align);
break;
case BPF_AND:
/* Disallow AND'ing of negative numbers, ain't nobody got time
else
dst_reg->min_value = 0;
dst_reg->max_value = max_val;
+ dst_reg->min_align = max(src_align, dst_align);
break;
case BPF_LSH:
/* Gotta have special overflow logic here, if we're shifting
* more than MAX_RANGE then just assume we have an invalid
* range.
*/
- if (min_val > ilog2(BPF_REGISTER_MAX_RANGE))
+ if (min_val > ilog2(BPF_REGISTER_MAX_RANGE)) {
dst_reg->min_value = BPF_REGISTER_MIN_RANGE;
- else if (dst_reg->min_value != BPF_REGISTER_MIN_RANGE)
- dst_reg->min_value <<= min_val;
-
+ dst_reg->min_align = 1;
+ } else {
+ if (dst_reg->min_value != BPF_REGISTER_MIN_RANGE)
+ dst_reg->min_value <<= min_val;
+ if (!dst_reg->min_align)
+ dst_reg->min_align = 1;
+ dst_reg->min_align <<= min_val;
+ }
if (max_val > ilog2(BPF_REGISTER_MAX_RANGE))
dst_reg->max_value = BPF_REGISTER_MAX_RANGE;
else if (dst_reg->max_value != BPF_REGISTER_MAX_RANGE)
/* RSH by a negative number is undefined, and the BPF_RSH is an
* unsigned shift, so make the appropriate casts.
*/
- if (min_val < 0 || dst_reg->min_value < 0)
+ if (min_val < 0 || dst_reg->min_value < 0) {
dst_reg->min_value = BPF_REGISTER_MIN_RANGE;
- else
+ } else {
dst_reg->min_value =
(u64)(dst_reg->min_value) >> min_val;
+ }
+ if (min_val < 0) {
+ dst_reg->min_align = 1;
+ } else {
+ dst_reg->min_align >>= (u64) min_val;
+ if (!dst_reg->min_align)
+ dst_reg->min_align = 1;
+ }
if (dst_reg->max_value != BPF_REGISTER_MAX_RANGE)
dst_reg->max_value >>= max_val;
break;
regs[insn->dst_reg].imm = insn->imm;
regs[insn->dst_reg].max_value = insn->imm;
regs[insn->dst_reg].min_value = insn->imm;
+ regs[insn->dst_reg].min_align = calc_align(insn->imm);
}
} else if (opcode > BPF_END) {
{
struct bpf_reg_state *regs = env->cur_state.regs;
u8 mode = BPF_MODE(insn->code);
- struct bpf_reg_state *reg;
int i, err;
if (!may_access_skb(env->prog->type)) {
}
/* reset caller saved regs to unreadable */
- for (i = 0; i < CALLER_SAVED_REGS; i++) {
- reg = regs + caller_saved[i];
- reg->type = NOT_INIT;
- reg->imm = 0;
- }
+ for (i = 0; i < CALLER_SAVED_REGS; i++)
+ mark_reg_not_init(regs, caller_saved[i]);
/* mark destination R0 register as readable, since it contains
* the value fetched from the packet
env->explored_states[t + 1] = STATE_LIST_MARK;
} else {
/* conditional jump with two edges */
+ env->explored_states[t] = STATE_LIST_MARK;
ret = push_insn(t, t + 1, FALLTHROUGH, env);
if (ret == 1)
goto peek_stack;
/* the following conditions reduce the number of explored insns
* from ~140k to ~80k for ultra large programs that use a lot of ptr_to_packet
*/
-static bool compare_ptrs_to_packet(struct bpf_reg_state *old,
+static bool compare_ptrs_to_packet(struct bpf_verifier_env *env,
+ struct bpf_reg_state *old,
struct bpf_reg_state *cur)
{
if (old->id != cur->id)
* 'if (R4 > data_end)' and all further insn were already good with r=20,
* so they will be good with r=30 and we can prune the search.
*/
- if (old->off <= cur->off &&
+ if (!env->strict_alignment && old->off <= cur->off &&
old->off >= old->range && cur->off >= cur->range)
return true;
rcur->type != NOT_INIT))
continue;
+ /* Don't care about the reg->id in this case. */
+ if (rold->type == PTR_TO_MAP_VALUE_OR_NULL &&
+ rcur->type == PTR_TO_MAP_VALUE_OR_NULL &&
+ rold->map_ptr == rcur->map_ptr)
+ continue;
+
if (rold->type == PTR_TO_PACKET && rcur->type == PTR_TO_PACKET &&
- compare_ptrs_to_packet(rold, rcur))
+ compare_ptrs_to_packet(env, rold, rcur))
continue;
return false;
goto process_bpf_exit;
}
- if (log_level && do_print_state) {
- verbose("\nfrom %d to %d:", prev_insn_idx, insn_idx);
+ if (need_resched())
+ cond_resched();
+
+ if (log_level > 1 || (log_level && do_print_state)) {
+ if (log_level > 1)
+ verbose("%d:", insn_idx);
+ else
+ verbose("\nfrom %d to %d:",
+ prev_insn_idx, insn_idx);
print_verifier_state(&env->cur_state);
do_print_state = false;
}
log_level = 0;
}
+ env->strict_alignment = !!(attr->prog_flags & BPF_F_STRICT_ALIGNMENT);
+ if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS))
+ env->strict_alignment = true;
+
ret = replace_map_fd_with_map_ptr(env);
if (ret < 0)
goto skip_full_check;
log_level = 0;
+ env->strict_alignment = false;
+ if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS))
+ env->strict_alignment = true;
+
env->explored_states = kcalloc(env->prog->len,
sizeof(struct bpf_verifier_state_list *),
GFP_KERNEL);
if (!p)
goto fork_out;
+ /*
+ * This _must_ happen before we call free_task(), i.e. before we jump
+ * to any of the bad_fork_* labels. This is to avoid freeing
+ * p->set_child_tid which is (ab)used as a kthread's data pointer for
+ * kernel threads (PF_KTHREAD).
+ */
+ p->set_child_tid = (clone_flags & CLONE_CHILD_SETTID) ? child_tidptr : NULL;
+ /*
+ * Clear TID on mm_release()?
+ */
+ p->clear_child_tid = (clone_flags & CLONE_CHILD_CLEARTID) ? child_tidptr : NULL;
+
ftrace_graph_init_task(p);
rt_mutex_init_task(p);
}
}
- p->set_child_tid = (clone_flags & CLONE_CHILD_SETTID) ? child_tidptr : NULL;
- /*
- * Clear TID on mm_release()?
- */
- p->clear_child_tid = (clone_flags & CLONE_CHILD_CLEARTID) ? child_tidptr : NULL;
#ifdef CONFIG_BLOCK
p->plug = NULL;
#endif
*/
recalc_sigpending();
if (signal_pending(current)) {
- spin_unlock(¤t->sighand->siglock);
- write_unlock_irq(&tasklist_lock);
retval = -ERESTARTNOINTR;
goto bad_fork_cancel_cgroup;
}
+ if (unlikely(!(ns_of_pid(pid)->nr_hashed & PIDNS_HASH_ADDING))) {
+ retval = -ENOMEM;
+ goto bad_fork_cancel_cgroup;
+ }
if (likely(p->pid)) {
ptrace_init_task(p, (clone_flags & CLONE_PTRACE) || trace);
return p;
bad_fork_cancel_cgroup:
+ spin_unlock(¤t->sighand->siglock);
+ write_unlock_irq(&tasklist_lock);
cgroup_cancel_fork(p);
bad_fork_free_pid:
cgroup_threadgroup_change_end(current);
if (!desc)
return;
- __irq_do_set_handler(desc, handle, 1, NULL);
desc->irq_common_data.handler_data = data;
+ __irq_do_set_handler(desc, handle, 1, NULL);
irq_put_desc_busunlock(desc, flags);
}
return module_alloc(PAGE_SIZE);
}
-static void free_insn_page(void *page)
+void __weak free_insn_page(void *page)
{
module_memfree(page);
}
}
/* Wait for completing optimization and unoptimization */
-static void wait_for_kprobe_optimizer(void)
+void wait_for_kprobe_optimizer(void)
{
mutex_lock(&kprobe_mutex);
* The vaddr this probe is installed will soon
* be vfreed buy not synced to disk. Hence,
* disarming the breakpoint isn't needed.
+ *
+ * Note, this will also move any optimized probes
+ * that are pending to be removed from their
+ * corresponding lists to the freeing_list and
+ * will not be touched by the delayed
+ * kprobe_optimizer work handler.
*/
kill_kprobe(p);
}
int ret;
raw_spin_lock_irq(&lock->wait_lock);
-
- set_current_state(TASK_INTERRUPTIBLE);
-
/* sleep on the mutex */
+ set_current_state(TASK_INTERRUPTIBLE);
ret = __rt_mutex_slowlock(lock, TASK_INTERRUPTIBLE, to, waiter);
-
+ /*
+ * try_to_take_rt_mutex() sets the waiter bit unconditionally. We might
+ * have to fix that up.
+ */
+ fixup_rt_mutex_waiters(lock);
raw_spin_unlock_irq(&lock->wait_lock);
return ret;
bool cleanup = false;
raw_spin_lock_irq(&lock->wait_lock);
+ /*
+ * Do an unconditional try-lock, this deals with the lock stealing
+ * state where __rt_mutex_futex_unlock() -> mark_wakeup_next_waiter()
+ * sets a NULL owner.
+ *
+ * We're not interested in the return value, because the subsequent
+ * test on rt_mutex_owner() will infer that. If the trylock succeeded,
+ * we will own the lock and it will have removed the waiter. If we
+ * failed the trylock, we're still not owner and we need to remove
+ * ourselves.
+ */
+ try_to_take_rt_mutex(lock, current, waiter);
/*
* Unless we're the owner; we're still enqueued on the wait_list.
* So check if we became owner, if not, take us off the wait_list.
*/
if (rt_mutex_owner(lock) != current) {
remove_waiter(lock, waiter);
- fixup_rt_mutex_waiters(lock);
cleanup = true;
}
-
/*
* try_to_take_rt_mutex() sets the waiter bit unconditionally. We might
* have to fix that up.
* if reparented.
*/
for (;;) {
- set_current_state(TASK_UNINTERRUPTIBLE);
+ set_current_state(TASK_INTERRUPTIBLE);
if (pid_ns->nr_hashed == init_pids)
break;
schedule();
* Numbers of normal and highmem page frames allocated for hibernation image
* before suspending devices.
*/
-unsigned int alloc_normal, alloc_highmem;
+static unsigned int alloc_normal, alloc_highmem;
/*
* Memory bitmap used for marking saveable pages (during hibernation) or
* hibernation image pages (during restore)
}
+void __ptrace_link(struct task_struct *child, struct task_struct *new_parent,
+ const struct cred *ptracer_cred)
+{
+ BUG_ON(!list_empty(&child->ptrace_entry));
+ list_add(&child->ptrace_entry, &new_parent->ptraced);
+ child->parent = new_parent;
+ child->ptracer_cred = get_cred(ptracer_cred);
+}
+
/*
* ptrace a task: make the debugger its new parent and
* move it to the ptrace list.
*
* Must be called with the tasklist lock write-held.
*/
-void __ptrace_link(struct task_struct *child, struct task_struct *new_parent)
+static void ptrace_link(struct task_struct *child, struct task_struct *new_parent)
{
- BUG_ON(!list_empty(&child->ptrace_entry));
- list_add(&child->ptrace_entry, &new_parent->ptraced);
- child->parent = new_parent;
rcu_read_lock();
- child->ptracer_cred = get_cred(__task_cred(new_parent));
+ __ptrace_link(child, new_parent, __task_cred(new_parent));
rcu_read_unlock();
}
flags |= PT_SEIZED;
task->ptrace = flags;
- __ptrace_link(task, current);
+ ptrace_link(task, current);
/* SEIZE doesn't trap tracee on attach */
if (!seize)
*/
if (!ret && !(current->real_parent->flags & PF_EXITING)) {
current->ptrace = PT_PTRACED;
- __ptrace_link(current, current->real_parent);
+ ptrace_link(current, current->real_parent);
}
}
write_unlock_irq(&tasklist_lock);
}
EXPORT_SYMBOL(schedule);
+/*
+ * synchronize_rcu_tasks() makes sure that no task is stuck in preempted
+ * state (have scheduled out non-voluntarily) by making sure that all
+ * tasks have either left the run queue or have gone into user space.
+ * As idle tasks do not do either, they must not ever be preempted
+ * (schedule out non-voluntarily).
+ *
+ * schedule_idle() is similar to schedule_preempt_disable() except that it
+ * never enables preemption because it does not call sched_submit_work().
+ */
+void __sched schedule_idle(void)
+{
+ /*
+ * As this skips calling sched_submit_work(), which the idle task does
+ * regardless because that function is a nop when the task is in a
+ * TASK_RUNNING state, make sure this isn't used someplace that the
+ * current task can be in any other state. Note, idle is always in the
+ * TASK_RUNNING state.
+ */
+ WARN_ON_ONCE(current->state);
+ do {
+ __schedule(false);
+ } while (need_resched());
+}
+
#ifdef CONFIG_CONTEXT_TRACKING
asmlinkage __visible void __sched schedule_user(void)
{
sugov_update_commit(sg_policy, time, next_f);
}
-static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu)
+static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time)
{
struct sugov_policy *sg_policy = sg_cpu->sg_policy;
struct cpufreq_policy *policy = sg_policy->policy;
- u64 last_freq_update_time = sg_policy->last_freq_update_time;
unsigned long util = 0, max = 1;
unsigned int j;
* enough, don't take the CPU into account as it probably is
* idle now (and clear iowait_boost for it).
*/
- delta_ns = last_freq_update_time - j_sg_cpu->last_update;
+ delta_ns = time - j_sg_cpu->last_update;
if (delta_ns > TICK_NSEC) {
j_sg_cpu->iowait_boost = 0;
continue;
if (flags & SCHED_CPUFREQ_RT_DL)
next_f = sg_policy->policy->cpuinfo.max_freq;
else
- next_f = sugov_next_freq_shared(sg_cpu);
+ next_f = sugov_next_freq_shared(sg_cpu, time);
sugov_update_commit(sg_policy, time, next_f);
}
smp_mb__after_atomic();
sched_ttwu_pending();
- schedule_preempt_disabled();
+ schedule_idle();
if (unlikely(klp_patch_pending(current)))
klp_update_patch_state(current);
}
#endif
+extern void schedule_idle(void);
+
extern void sysrq_sched_debug_show(void);
extern void sched_init_granularity(void);
extern void update_max_interval(void);
* At the hard limit, we just die.
* No need to calculate anything else now.
*/
- pr_info("CPU Watchdog Timeout (hard): %s[%d]\n",
- tsk->comm, task_pid_nr(tsk));
+ if (print_fatal_signals) {
+ pr_info("CPU Watchdog Timeout (hard): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
+ }
__group_send_sig_info(SIGKILL, SEND_SIG_PRIV, tsk);
return;
}
soft += USEC_PER_SEC;
sig->rlim[RLIMIT_RTTIME].rlim_cur = soft;
}
- pr_info("RT Watchdog Timeout (soft): %s[%d]\n",
- tsk->comm, task_pid_nr(tsk));
+ if (print_fatal_signals) {
+ pr_info("RT Watchdog Timeout (soft): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
+ }
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
}
}
* At the hard limit, we just die.
* No need to calculate anything else now.
*/
- pr_info("RT Watchdog Timeout (hard): %s[%d]\n",
- tsk->comm, task_pid_nr(tsk));
+ if (print_fatal_signals) {
+ pr_info("RT Watchdog Timeout (hard): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
+ }
__group_send_sig_info(SIGKILL, SEND_SIG_PRIV, tsk);
return;
}
/*
* At the soft limit, send a SIGXCPU every second.
*/
- pr_info("CPU Watchdog Timeout (soft): %s[%d]\n",
- tsk->comm, task_pid_nr(tsk));
+ if (print_fatal_signals) {
+ pr_info("CPU Watchdog Timeout (soft): %s[%d]\n",
+ tsk->comm, task_pid_nr(tsk));
+ }
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
if (soft < hard) {
soft++;
goto out;
if (attr == &dev_attr_act_mask) {
- if (sscanf(buf, "%llx", &value) != 1) {
+ if (kstrtoull(buf, 0, &value)) {
/* Assume it is a list of trace category names */
ret = blk_trace_str2mask(buf);
if (ret < 0)
goto out;
value = ret;
}
- } else if (sscanf(buf, "%llu", &value) != 1)
+ } else if (kstrtoull(buf, 0, &value))
goto out;
ret = -ENXIO;
int i, ret = -ENODEV;
int size;
- if (glob && (strcmp(glob, "*") == 0 || !strlen(glob)))
+ if (!glob || !strlen(glob) || !strcmp(glob, "*"))
func_g.search = NULL;
- else if (glob) {
+ else {
int not;
func_g.type = filter_parse_regex(glob, strlen(glob),
return ret;
}
+void clear_ftrace_function_probes(struct trace_array *tr)
+{
+ struct ftrace_func_probe *probe, *n;
+
+ list_for_each_entry_safe(probe, n, &tr->func_probes, list)
+ unregister_ftrace_function_probe_func(NULL, tr, probe->probe_ops);
+}
+
static LIST_HEAD(ftrace_commands);
static DEFINE_MUTEX(ftrace_cmd_mutex);
}
out:
- kfree(fgd->new_hash);
+ free_ftrace_hash(fgd->new_hash);
kfree(fgd);
return ret;
return 0;
}
-early_initcall(init_trace_selftests);
+core_initcall(init_trace_selftests);
#else
static inline int run_tracer_selftest(struct tracer *type)
{
void __trace_stack(struct trace_array *tr, unsigned long flags, int skip,
int pc)
{
- __ftrace_trace_stack(tr->trace_buffer.buffer, flags, skip, pc, NULL);
+ struct ring_buffer *buffer = tr->trace_buffer.buffer;
+
+ if (rcu_is_watching()) {
+ __ftrace_trace_stack(buffer, flags, skip, pc, NULL);
+ return;
+ }
+
+ /*
+ * When an NMI triggers, RCU is enabled via rcu_nmi_enter(),
+ * but if the above rcu_is_watching() failed, then the NMI
+ * triggered someplace critical, and rcu_irq_enter() should
+ * not be called from NMI.
+ */
+ if (unlikely(in_nmi()))
+ return;
+
+ /*
+ * It is possible that a function is being traced in a
+ * location that RCU is not watching. A call to
+ * rcu_irq_enter() will make sure that it is, but there's
+ * a few internal rcu functions that could be traced
+ * where that wont work either. In those cases, we just
+ * do nothing.
+ */
+ if (unlikely(rcu_irq_enter_disabled()))
+ return;
+
+ rcu_irq_enter_irqson();
+ __ftrace_trace_stack(buffer, flags, skip, pc, NULL);
+ rcu_irq_exit_irqson();
}
/**
}
tracing_set_nop(tr);
+ clear_ftrace_function_probes(tr);
event_trace_del_tracer(tr);
ftrace_clear_pids(tr);
ftrace_destroy_function_files(tr);
extern int
unregister_ftrace_function_probe_func(char *glob, struct trace_array *tr,
struct ftrace_probe_ops *ops);
+extern void clear_ftrace_function_probes(struct trace_array *tr);
int register_ftrace_command(struct ftrace_func_command *cmd);
int unregister_ftrace_command(struct ftrace_func_command *cmd);
{
return -EINVAL;
}
+static inline void clear_ftrace_function_probes(struct trace_array *tr)
+{
+}
+
/*
* The ops parameter passed in is usually undefined.
* This must be a macro.
end:
release_all_trace_kprobes();
+ /*
+ * Wait for the optimizer work to finish. Otherwise it might fiddle
+ * with probes in already freed __init text.
+ */
+ wait_for_kprobe_optimizer();
if (warn)
pr_cont("NG: Some tests are failed. Please check them.\n");
else
{ },
{ { 0, 1 } },
},
+ {
+ "JMP_JSGE_K: Signed jump: value walk 1",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 0),
+ BPF_LD_IMM64(R1, -3),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 6),
+ BPF_ALU64_IMM(BPF_ADD, R1, 1),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 4),
+ BPF_ALU64_IMM(BPF_ADD, R1, 1),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 2),
+ BPF_ALU64_IMM(BPF_ADD, R1, 1),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 1),
+ BPF_EXIT_INSN(), /* bad exit */
+ BPF_ALU32_IMM(BPF_MOV, R0, 1), /* good exit */
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 1 } },
+ },
+ {
+ "JMP_JSGE_K: Signed jump: value walk 2",
+ .u.insns_int = {
+ BPF_ALU32_IMM(BPF_MOV, R0, 0),
+ BPF_LD_IMM64(R1, -3),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 4),
+ BPF_ALU64_IMM(BPF_ADD, R1, 2),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 2),
+ BPF_ALU64_IMM(BPF_ADD, R1, 2),
+ BPF_JMP_IMM(BPF_JSGE, R1, 0, 1),
+ BPF_EXIT_INSN(), /* bad exit */
+ BPF_ALU32_IMM(BPF_MOV, R0, 1), /* good exit */
+ BPF_EXIT_INSN(),
+ },
+ INTERNAL,
+ { },
+ { { 0, 1 } },
+ },
/* BPF_JMP | BPF_JGT | BPF_K */
{
"JMP_JGT_K: if (3 > 2) return 1",
goto error_xenbus;
}
priv->tag = xenbus_read(xbt, dev->nodename, "tag", NULL);
- if (!priv->tag) {
- ret = -EINVAL;
+ if (IS_ERR(priv->tag)) {
+ ret = PTR_ERR(priv->tag);
goto error_xenbus;
}
ret = xenbus_transaction_end(xbt, 0);
.otherend_changed = xen_9pfs_front_changed,
};
-int p9_trans_xen_init(void)
+static int p9_trans_xen_init(void)
{
if (!xen_domain())
return -ENODEV;
}
module_init(p9_trans_xen_init);
-void p9_trans_xen_exit(void)
+static void p9_trans_xen_exit(void)
{
v9fs_unregister_trans(&p9_xen_trans);
return xenbus_unregister_driver(&xen_9pfs_front_driver);
return -EPROTONOSUPPORT;
}
}
+
+ if (data[IFLA_BR_VLAN_DEFAULT_PVID]) {
+ __u16 defpvid = nla_get_u16(data[IFLA_BR_VLAN_DEFAULT_PVID]);
+
+ if (defpvid >= VLAN_VID_MASK)
+ return -EINVAL;
+ }
#endif
return 0;
br_debug(br, "using kernel STP\n");
/* To start timers on any ports left in blocking */
+ mod_timer(&br->hello_timer, jiffies + br->hello_time);
br_port_state_selection(br);
}
if (br->dev->flags & IFF_UP) {
br_config_bpdu_generation(br);
- if (br->stp_enabled != BR_USER_STP)
+ if (br->stp_enabled == BR_KERNEL_STP)
mod_timer(&br->hello_timer,
round_jiffies(jiffies + br->hello_time));
}
if (e->ethproto != htons(ETH_P_ARP) ||
e->invflags & EBT_IPROTO)
return -EINVAL;
+ if (ebt_invalid_target(info->target))
+ return -EINVAL;
+
return 0;
}
strlcpy(name, _name, sizeof(name));
if (copy_to_user(um, name, EBT_FUNCTION_MAXNAMELEN) ||
put_user(datasize, (int __user *)(um + EBT_FUNCTION_MAXNAMELEN)) ||
- xt_data_to_user(um + entrysize, data, usersize, datasize))
+ xt_data_to_user(um + entrysize, data, usersize, datasize,
+ XT_ALIGN(datasize)))
return -EFAULT;
return 0;
if (match->compat_to_user(cm->data, m->data))
return -EFAULT;
} else {
- if (xt_data_to_user(cm->data, m->data, match->usersize, msize))
+ if (xt_data_to_user(cm->data, m->data, match->usersize, msize,
+ COMPAT_XT_ALIGN(msize)))
return -EFAULT;
}
if (target->compat_to_user(cm->data, t->data))
return -EFAULT;
} else {
- if (xt_data_to_user(cm->data, t->data, target->usersize, tsize))
+ if (xt_data_to_user(cm->data, t->data, target->usersize, tsize,
+ COMPAT_XT_ALIGN(tsize)))
return -EFAULT;
}
struct timespec validity;
void *tp, *tpend;
void **ptp;
- struct ceph_crypto_key new_session_key;
+ struct ceph_crypto_key new_session_key = { 0 };
struct ceph_buffer *new_ticket_blob;
unsigned long new_expires, new_renew_after;
u64 new_secret_id;
dout(" ticket blob is %d bytes\n", dlen);
ceph_decode_need(ptp, tpend, 1 + sizeof(u64), bad);
blob_struct_v = ceph_decode_8(ptp);
+ if (blob_struct_v != 1)
+ goto bad;
+
new_secret_id = ceph_decode_64(ptp);
ret = ceph_decode_buffer(&new_ticket_blob, ptp, tpend);
if (ret)
type, ceph_entity_type_name(type), th->secret_id,
(int)th->ticket_blob->vec.iov_len);
xi->have_keys |= th->service;
-
-out:
- return ret;
+ return 0;
bad:
ret = -EINVAL;
- goto out;
+out:
+ ceph_crypto_key_destroy(&new_session_key);
+ return ret;
}
static int ceph_x_proc_ticket_reply(struct ceph_auth_client *ac,
module_param_cb(supported_features, ¶m_ops_supported_features, NULL,
S_IRUGO);
-/*
- * find filename portion of a path (/foo/bar/baz -> baz)
- */
-const char *ceph_file_part(const char *s, int len)
-{
- const char *e = s + len;
-
- while (e != s && *(e-1) != '/')
- e--;
- return e;
-}
-EXPORT_SYMBOL(ceph_file_part);
-
const char *ceph_msg_type_name(int type)
{
switch (type) {
* Returns true if the result moves the cursor on to the next piece
* of the data item.
*/
-static bool ceph_msg_data_advance(struct ceph_msg_data_cursor *cursor,
- size_t bytes)
+static void ceph_msg_data_advance(struct ceph_msg_data_cursor *cursor,
+ size_t bytes)
{
bool new_piece;
new_piece = true;
}
cursor->need_crc = new_piece;
-
- return new_piece;
}
static size_t sizeof_footer(struct ceph_connection *con)
size_t page_offset;
size_t length;
bool last_piece;
- bool need_crc;
int ret;
page = ceph_msg_data_next(cursor, &page_offset, &length,
}
if (do_datacrc && cursor->need_crc)
crc = ceph_crc32c_page(crc, page, page_offset, length);
- need_crc = ceph_msg_data_advance(cursor, (size_t)ret);
+ ceph_msg_data_advance(cursor, (size_t)ret);
}
dout("%s %p msg %p done\n", __func__, con, msg);
struct ceph_msg *m;
u64 ack = le64_to_cpu(con->in_temp_ack);
u64 seq;
+ bool reconnect = (con->in_tag == CEPH_MSGR_TAG_SEQ);
+ struct list_head *list = reconnect ? &con->out_queue : &con->out_sent;
- while (!list_empty(&con->out_sent)) {
- m = list_first_entry(&con->out_sent, struct ceph_msg,
- list_head);
+ /*
+ * In the reconnect case, con_fault() has requeued messages
+ * in out_sent. We should cleanup old messages according to
+ * the reconnect seq.
+ */
+ while (!list_empty(list)) {
+ m = list_first_entry(list, struct ceph_msg, list_head);
+ if (reconnect && m->needs_out_seq)
+ break;
seq = le64_to_cpu(m->hdr.seq);
if (seq > ack)
break;
m->ack_stamp = jiffies;
ceph_msg_remove(m);
}
+
prepare_read_tag(con);
}
if (do_datacrc)
crc = ceph_crc32c_page(crc, page, page_offset, ret);
- (void) ceph_msg_data_advance(cursor, (size_t)ret);
+ ceph_msg_data_advance(cursor, (size_t)ret);
}
if (do_datacrc)
con->in_data_crc = crc;
int i, err = -EINVAL;
struct ceph_fsid fsid;
u32 epoch, num_mon;
- u16 version;
u32 len;
ceph_decode_32_safe(&p, end, len, bad);
ceph_decode_need(&p, end, len, bad);
dout("monmap_decode %p %p len %d\n", p, end, (int)(end-p));
-
- ceph_decode_16_safe(&p, end, version, bad);
+ p += sizeof(u16); /* skip version */
ceph_decode_need(&p, end, sizeof(fsid) + 2*sizeof(u32), bad);
ceph_decode_copy(&p, &fsid, sizeof(fsid));
u32 yes;
struct crush_rule *r;
+ err = -EINVAL;
ceph_decode_32_safe(p, end, yes, bad);
if (!yes) {
dout("crush_decode NO rule %d off %x %p to %p\n",
}
EXPORT_SYMBOL(dev_change_proto_down);
+bool __dev_xdp_attached(struct net_device *dev, xdp_op_t xdp_op)
+{
+ struct netdev_xdp xdp;
+
+ memset(&xdp, 0, sizeof(xdp));
+ xdp.command = XDP_QUERY_PROG;
+
+ /* Query must always succeed. */
+ WARN_ON(xdp_op(dev, &xdp) < 0);
+ return xdp.prog_attached;
+}
+
+static int dev_xdp_install(struct net_device *dev, xdp_op_t xdp_op,
+ struct netlink_ext_ack *extack,
+ struct bpf_prog *prog)
+{
+ struct netdev_xdp xdp;
+
+ memset(&xdp, 0, sizeof(xdp));
+ xdp.command = XDP_SETUP_PROG;
+ xdp.extack = extack;
+ xdp.prog = prog;
+
+ return xdp_op(dev, &xdp);
+}
+
/**
* dev_change_xdp_fd - set or clear a bpf program for a device rx path
* @dev: device
int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
int fd, u32 flags)
{
- int (*xdp_op)(struct net_device *dev, struct netdev_xdp *xdp);
const struct net_device_ops *ops = dev->netdev_ops;
struct bpf_prog *prog = NULL;
- struct netdev_xdp xdp;
+ xdp_op_t xdp_op, xdp_chk;
int err;
ASSERT_RTNL();
- xdp_op = ops->ndo_xdp;
+ xdp_op = xdp_chk = ops->ndo_xdp;
+ if (!xdp_op && (flags & XDP_FLAGS_DRV_MODE))
+ return -EOPNOTSUPP;
if (!xdp_op || (flags & XDP_FLAGS_SKB_MODE))
xdp_op = generic_xdp_install;
+ if (xdp_op == xdp_chk)
+ xdp_chk = generic_xdp_install;
if (fd >= 0) {
- if (flags & XDP_FLAGS_UPDATE_IF_NOEXIST) {
- memset(&xdp, 0, sizeof(xdp));
- xdp.command = XDP_QUERY_PROG;
-
- err = xdp_op(dev, &xdp);
- if (err < 0)
- return err;
- if (xdp.prog_attached)
- return -EBUSY;
- }
+ if (xdp_chk && __dev_xdp_attached(dev, xdp_chk))
+ return -EEXIST;
+ if ((flags & XDP_FLAGS_UPDATE_IF_NOEXIST) &&
+ __dev_xdp_attached(dev, xdp_op))
+ return -EBUSY;
prog = bpf_prog_get_type(fd, BPF_PROG_TYPE_XDP);
if (IS_ERR(prog))
return PTR_ERR(prog);
}
- memset(&xdp, 0, sizeof(xdp));
- xdp.command = XDP_SETUP_PROG;
- xdp.extack = extack;
- xdp.prog = prog;
-
- err = xdp_op(dev, &xdp);
+ err = dev_xdp_install(dev, xdp_op, extack, prog);
if (err < 0 && prog)
bpf_prog_put(prog);
}
EXPORT_SYMBOL(dst_discard_out);
-const u32 dst_default_metrics[RTAX_MAX + 1] = {
+const struct dst_metrics dst_default_metrics = {
/* This initializer is needed to force linker to place this variable
* into const section. Otherwise it might end into bss section.
* We really want to avoid false sharing on this variable, and catch
* any writes on it.
*/
- [RTAX_MAX] = 0xdeadbeef,
+ .refcnt = ATOMIC_INIT(1),
};
void dst_init(struct dst_entry *dst, struct dst_ops *ops,
if (dev)
dev_hold(dev);
dst->ops = ops;
- dst_init_metrics(dst, dst_default_metrics, true);
+ dst_init_metrics(dst, dst_default_metrics.metrics, true);
dst->expires = 0UL;
dst->path = dst;
dst->from = NULL;
u32 *dst_cow_metrics_generic(struct dst_entry *dst, unsigned long old)
{
- u32 *p = kmalloc(sizeof(u32) * RTAX_MAX, GFP_ATOMIC);
+ struct dst_metrics *p = kmalloc(sizeof(*p), GFP_ATOMIC);
if (p) {
- u32 *old_p = __DST_METRICS_PTR(old);
+ struct dst_metrics *old_p = (struct dst_metrics *)__DST_METRICS_PTR(old);
unsigned long prev, new;
- memcpy(p, old_p, sizeof(u32) * RTAX_MAX);
+ atomic_set(&p->refcnt, 1);
+ memcpy(p->metrics, old_p->metrics, sizeof(p->metrics));
new = (unsigned long) p;
prev = cmpxchg(&dst->_metrics, old, new);
if (prev != old) {
kfree(p);
- p = __DST_METRICS_PTR(prev);
+ p = (struct dst_metrics *)__DST_METRICS_PTR(prev);
if (prev & DST_METRICS_READ_ONLY)
p = NULL;
+ } else if (prev & DST_METRICS_REFCOUNTED) {
+ if (atomic_dec_and_test(&old_p->refcnt))
+ kfree(old_p);
}
}
- return p;
+ BUILD_BUG_ON(offsetof(struct dst_metrics, metrics) != 0);
+ return (u32 *)p;
}
EXPORT_SYMBOL(dst_cow_metrics_generic);
{
unsigned long prev, new;
- new = ((unsigned long) dst_default_metrics) | DST_METRICS_READ_ONLY;
+ new = ((unsigned long) &dst_default_metrics) | DST_METRICS_READ_ONLY;
prev = cmpxchg(&dst->_metrics, old, new);
if (prev == old)
kfree(__DST_METRICS_PTR(old));
func == bpf_skb_change_head ||
func == bpf_skb_change_tail ||
func == bpf_skb_pull_data ||
+ func == bpf_clone_redirect ||
func == bpf_l3_csum_replace ||
func == bpf_l4_csum_replace ||
func == bpf_xdp_adjust_head)
lladdr = neigh->ha;
}
- if (new & NUD_CONNECTED)
- neigh->confirmed = jiffies;
- neigh->updated = jiffies;
-
/* If entry was valid and address is not changed,
do not change entry state, if new one is STALE.
*/
}
}
+ /* Update timestamps only once we know we will make a change to the
+ * neighbour entry. Otherwise we risk to move the locktime window with
+ * noop updates and ignore relevant ARP updates.
+ */
+ if (new != old || lladdr != neigh->ha) {
+ if (new & NUD_CONNECTED)
+ neigh->confirmed = jiffies;
+ neigh->updated = jiffies;
+ }
+
if (new != old) {
neigh_del_timer(neigh);
if (new & NUD_PROBE)
goto out;
}
+static int __net_init net_defaults_init_net(struct net *net)
+{
+ net->core.sysctl_somaxconn = SOMAXCONN;
+ return 0;
+}
+
+static struct pernet_operations net_defaults_ops = {
+ .init = net_defaults_init_net,
+};
+
+static __init int net_defaults_init(void)
+{
+ if (register_pernet_subsys(&net_defaults_ops))
+ panic("Cannot initialize net default settings");
+
+ return 0;
+}
+
+core_initcall(net_defaults_init);
#ifdef CONFIG_NET_NS
static struct ucounts *inc_net_namespaces(struct user_namespace *ns)
static size_t rtnl_xdp_size(void)
{
size_t xdp_size = nla_total_size(0) + /* nest IFLA_XDP */
- nla_total_size(1) + /* XDP_ATTACHED */
- nla_total_size(4); /* XDP_FLAGS */
+ nla_total_size(1); /* XDP_ATTACHED */
return xdp_size;
}
return 0;
}
+static u8 rtnl_xdp_attached_mode(struct net_device *dev)
+{
+ const struct net_device_ops *ops = dev->netdev_ops;
+
+ ASSERT_RTNL();
+
+ if (rcu_access_pointer(dev->xdp_prog))
+ return XDP_ATTACHED_SKB;
+ if (ops->ndo_xdp && __dev_xdp_attached(dev, ops->ndo_xdp))
+ return XDP_ATTACHED_DRV;
+
+ return XDP_ATTACHED_NONE;
+}
+
static int rtnl_xdp_fill(struct sk_buff *skb, struct net_device *dev)
{
struct nlattr *xdp;
- u32 xdp_flags = 0;
- u8 val = 0;
int err;
xdp = nla_nest_start(skb, IFLA_XDP);
if (!xdp)
return -EMSGSIZE;
- if (rcu_access_pointer(dev->xdp_prog)) {
- xdp_flags = XDP_FLAGS_SKB_MODE;
- val = 1;
- } else if (dev->netdev_ops->ndo_xdp) {
- struct netdev_xdp xdp_op = {};
-
- xdp_op.command = XDP_QUERY_PROG;
- err = dev->netdev_ops->ndo_xdp(dev, &xdp_op);
- if (err)
- goto err_cancel;
- val = xdp_op.prog_attached;
- }
- err = nla_put_u8(skb, IFLA_XDP_ATTACHED, val);
+
+ err = nla_put_u8(skb, IFLA_XDP_ATTACHED,
+ rtnl_xdp_attached_mode(dev));
if (err)
goto err_cancel;
- if (xdp_flags) {
- err = nla_put_u32(skb, IFLA_XDP_FLAGS, xdp_flags);
- if (err)
- goto err_cancel;
- }
nla_nest_end(skb, xdp);
return 0;
cb->nlh->nlmsg_seq, 0,
flags,
ext_filter_mask);
- /* If we ran out of room on the first message,
- * we're in trouble
- */
- WARN_ON((err == -EMSGSIZE) && (skb->len == 0));
- if (err < 0)
- goto out;
+ if (err < 0) {
+ if (likely(skb->len))
+ goto out;
+
+ goto out_err;
+ }
nl_dump_check_consistent(cb, nlmsg_hdr(skb));
cont:
}
}
out:
+ err = skb->len;
+out_err:
cb->args[1] = idx;
cb->args[0] = h;
- return skb->len;
+ return err;
}
int rtnl_nla_parse_ifla(struct nlattr **tb, const struct nlattr *head, int len,
err = -EINVAL;
goto errout;
}
+ if ((xdp_flags & XDP_FLAGS_SKB_MODE) &&
+ (xdp_flags & XDP_FLAGS_DRV_MODE)) {
+ err = -EINVAL;
+ goto errout;
+ }
}
if (xdp[IFLA_XDP_FD]) {
int err = 0;
int fidx = 0;
- if (nlmsg_parse(cb->nlh, sizeof(struct ifinfomsg), tb,
- IFLA_MAX, ifla_policy, NULL) == 0) {
+ err = nlmsg_parse(cb->nlh, sizeof(struct ifinfomsg), tb,
+ IFLA_MAX, ifla_policy, NULL);
+ if (err < 0) {
+ return -EINVAL;
+ } else if (err == 0) {
if (tb[IFLA_MASTER])
br_idx = nla_get_u32(tb[IFLA_MASTER]);
}
err = br_dev->netdev_ops->ndo_bridge_getlink(
skb, portid, seq, dev,
filter_mask, NLM_F_MULTI);
- if (err < 0 && err != -EOPNOTSUPP)
- break;
+ if (err < 0 && err != -EOPNOTSUPP) {
+ if (likely(skb->len))
+ break;
+
+ goto out_err;
+ }
}
idx++;
}
seq, dev,
filter_mask,
NLM_F_MULTI);
- if (err < 0 && err != -EOPNOTSUPP)
- break;
+ if (err < 0 && err != -EOPNOTSUPP) {
+ if (likely(skb->len))
+ break;
+
+ goto out_err;
+ }
}
idx++;
}
}
+ err = skb->len;
+out_err:
rcu_read_unlock();
cb->args[0] = idx;
- return skb->len;
+ return err;
}
static inline size_t bridge_nlmsg_size(void)
#include <trace/events/sock.h>
-#ifdef CONFIG_INET
#include <net/tcp.h>
-#endif
-
#include <net/busy_poll.h>
static DEFINE_MUTEX(proto_list_mutex);
* delay queue. We want to allow the owner socket to send more
* packets, as if they were already TX completed by a typical driver.
* But we also want to keep skb->sk set because some packet schedulers
- * rely on it (sch_fq for example). So we set skb->truesize to a small
- * amount (1) and decrease sk_wmem_alloc accordingly.
+ * rely on it (sch_fq for example).
*/
void skb_orphan_partial(struct sk_buff *skb)
{
- /* If this skb is a TCP pure ACK or already went here,
- * we have nothing to do. 2 is already a very small truesize.
- */
- if (skb->truesize <= 2)
+ if (skb_is_tcp_pure_ack(skb))
return;
- /* TCP stack sets skb->ooo_okay based on sk_wmem_alloc,
- * so we do not completely orphan skb, but transfert all
- * accounted bytes but one, to avoid unexpected reorders.
- */
if (skb->destructor == sock_wfree
#ifdef CONFIG_INET
|| skb->destructor == tcp_wfree
#endif
) {
- atomic_sub(skb->truesize - 1, &skb->sk->sk_wmem_alloc);
- skb->truesize = 1;
+ struct sock *sk = skb->sk;
+
+ if (atomic_inc_not_zero(&sk->sk_refcnt)) {
+ atomic_sub(skb->truesize, &sk->sk_wmem_alloc);
+ skb->destructor = sock_efree;
+ }
} else {
skb_orphan(skb);
}
{
struct ctl_table *tbl;
- net->core.sysctl_somaxconn = SOMAXCONN;
-
tbl = netns_core_table;
if (!net_eq(net, &init_net)) {
tbl = kmemdup(tbl, sizeof(netns_core_table), GFP_KERNEL);
newsk->sk_backlog_rcv = dccp_v4_do_rcv;
newnp->pktoptions = NULL;
newnp->opt = NULL;
+ newnp->ipv6_mc_list = NULL;
+ newnp->ipv6_ac_list = NULL;
+ newnp->ipv6_fl_list = NULL;
newnp->mcast_oif = inet6_iif(skb);
newnp->mcast_hops = ipv6_hdr(skb)->hop_limit;
/* Clone RX bits */
newnp->rxopt.all = np->rxopt.all;
+ newnp->ipv6_mc_list = NULL;
+ newnp->ipv6_ac_list = NULL;
+ newnp->ipv6_fl_list = NULL;
newnp->pktoptions = NULL;
newnp->opt = NULL;
newnp->mcast_oif = inet6_iif(skb);
}
EXPORT_SYMBOL(arp_xmit);
+static bool arp_is_garp(struct net *net, struct net_device *dev,
+ int *addr_type, __be16 ar_op,
+ __be32 sip, __be32 tip,
+ unsigned char *sha, unsigned char *tha)
+{
+ bool is_garp = tip == sip;
+
+ /* Gratuitous ARP _replies_ also require target hwaddr to be
+ * the same as source.
+ */
+ if (is_garp && ar_op == htons(ARPOP_REPLY))
+ is_garp =
+ /* IPv4 over IEEE 1394 doesn't provide target
+ * hardware address field in its ARP payload.
+ */
+ tha &&
+ !memcmp(tha, sha, dev->addr_len);
+
+ if (is_garp) {
+ *addr_type = inet_addr_type_dev_table(net, dev, sip);
+ if (*addr_type != RTN_UNICAST)
+ is_garp = false;
+ }
+ return is_garp;
+}
+
/*
* Process an arp request.
*/
unsigned char *arp_ptr;
struct rtable *rt;
unsigned char *sha;
+ unsigned char *tha = NULL;
__be32 sip, tip;
u16 dev_type = dev->type;
int addr_type;
break;
#endif
default:
+ tha = arp_ptr;
arp_ptr += dev->addr_len;
}
memcpy(&tip, arp_ptr, 4);
n = __neigh_lookup(&arp_tbl, &sip, dev, 0);
- if (IN_DEV_ARP_ACCEPT(in_dev)) {
- unsigned int addr_type = inet_addr_type_dev_table(net, dev, sip);
+ addr_type = -1;
+ if (n || IN_DEV_ARP_ACCEPT(in_dev)) {
+ is_garp = arp_is_garp(net, dev, &addr_type, arp->ar_op,
+ sip, tip, sha, tha);
+ }
+ if (IN_DEV_ARP_ACCEPT(in_dev)) {
/* Unsolicited ARP is not accepted by default.
It is possible, that this option should be enabled for some
devices (strip is candidate)
*/
- is_garp = arp->ar_op == htons(ARPOP_REQUEST) && tip == sip &&
- addr_type == RTN_UNICAST;
-
if (!n &&
- ((arp->ar_op == htons(ARPOP_REPLY) &&
- addr_type == RTN_UNICAST) || is_garp))
+ (is_garp ||
+ (arp->ar_op == htons(ARPOP_REPLY) &&
+ (addr_type == RTN_UNICAST ||
+ (addr_type < 0 &&
+ /* postpone calculation to as late as possible */
+ inet_addr_type_dev_table(net, dev, sip) ==
+ RTN_UNICAST)))))
n = __neigh_lookup(&arp_tbl, &sip, dev, 1);
}
u8 *tail;
u8 *vaddr;
int nfrags;
+ int esph_offset;
struct page *page;
struct sk_buff *trailer;
int tailen = esp->tailen;
}
cow:
+ esph_offset = (unsigned char *)esp->esph - skb_transport_header(skb);
+
nfrags = skb_cow_data(skb, tailen, &trailer);
if (nfrags < 0)
goto out;
tail = skb_tail_pointer(trailer);
- esp->esph = ip_esp_hdr(skb);
+ esp->esph = (struct ip_esp_hdr *)(skb_transport_header(skb) + esph_offset);
skip_cow:
esp_output_fill_trailer(tail, esp->tfclen, esp->plen, esp->proto);
unsigned int e = 0, s_e;
struct fib_table *tb;
struct hlist_head *head;
- int dumped = 0;
+ int dumped = 0, err;
if (nlmsg_len(cb->nlh) >= sizeof(struct rtmsg) &&
((struct rtmsg *) nlmsg_data(cb->nlh))->rtm_flags & RTM_F_CLONED)
if (dumped)
memset(&cb->args[2], 0, sizeof(cb->args) -
2 * sizeof(cb->args[0]));
- if (fib_table_dump(tb, skb, cb) < 0)
- goto out;
+ err = fib_table_dump(tb, skb, cb);
+ if (err < 0) {
+ if (likely(skb->len))
+ goto out;
+
+ goto out_err;
+ }
dumped = 1;
next:
e++;
}
}
out:
+ err = skb->len;
+out_err:
rcu_read_unlock();
cb->args[1] = e;
cb->args[0] = h;
- return skb->len;
+ return err;
}
/* Prepare and feed intra-kernel routing request.
static void free_fib_info_rcu(struct rcu_head *head)
{
struct fib_info *fi = container_of(head, struct fib_info, rcu);
+ struct dst_metrics *m;
change_nexthops(fi) {
if (nexthop_nh->nh_dev)
rt_fibinfo_free(&nexthop_nh->nh_rth_input);
} endfor_nexthops(fi);
- if (fi->fib_metrics != (u32 *) dst_default_metrics)
- kfree(fi->fib_metrics);
+ m = fi->fib_metrics;
+ if (m != &dst_default_metrics && atomic_dec_and_test(&m->refcnt))
+ kfree(m);
kfree(fi);
}
val = 255;
if (type == RTAX_FEATURES && (val & ~RTAX_FEATURE_MASK))
return -EINVAL;
- fi->fib_metrics[type - 1] = val;
+ fi->fib_metrics->metrics[type - 1] = val;
}
if (ecn_ca)
- fi->fib_metrics[RTAX_FEATURES - 1] |= DST_FEATURE_ECN_CA;
+ fi->fib_metrics->metrics[RTAX_FEATURES - 1] |= DST_FEATURE_ECN_CA;
return 0;
}
goto failure;
fib_info_cnt++;
if (cfg->fc_mx) {
- fi->fib_metrics = kzalloc(sizeof(u32) * RTAX_MAX, GFP_KERNEL);
+ fi->fib_metrics = kzalloc(sizeof(*fi->fib_metrics), GFP_KERNEL);
if (!fi->fib_metrics)
goto failure;
+ atomic_set(&fi->fib_metrics->refcnt, 1);
} else
- fi->fib_metrics = (u32 *) dst_default_metrics;
+ fi->fib_metrics = (struct dst_metrics *)&dst_default_metrics;
fi->fib_net = net;
fi->fib_protocol = cfg->fc_protocol;
if (fi->fib_priority &&
nla_put_u32(skb, RTA_PRIORITY, fi->fib_priority))
goto nla_put_failure;
- if (rtnetlink_put_metrics(skb, fi->fib_metrics) < 0)
+ if (rtnetlink_put_metrics(skb, fi->fib_metrics->metrics) < 0)
goto nla_put_failure;
if (fi->fib_prefsrc &&
/* rcu_read_lock is hold by caller */
hlist_for_each_entry_rcu(fa, &l->leaf, fa_list) {
+ int err;
+
if (i < s_i) {
i++;
continue;
continue;
}
- if (fib_dump_info(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq,
- RTM_NEWROUTE,
- tb->tb_id,
- fa->fa_type,
- xkey,
- KEYLENGTH - fa->fa_slen,
- fa->fa_tos,
- fa->fa_info, NLM_F_MULTI) < 0) {
+ err = fib_dump_info(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, RTM_NEWROUTE,
+ tb->tb_id, fa->fa_type,
+ xkey, KEYLENGTH - fa->fa_slen,
+ fa->fa_tos, fa->fa_info, NLM_F_MULTI);
+ if (err < 0) {
cb->args[4] = i;
- return -1;
+ return err;
}
i++;
}
t_key key = cb->args[3];
while ((l = leaf_walk_rcu(&tp, key)) != NULL) {
- if (fn_trie_dump_leaf(l, tb, skb, cb) < 0) {
+ int err;
+
+ err = fn_trie_dump_leaf(l, tb, skb, cb);
+ if (err < 0) {
cb->args[3] = key;
cb->args[2] = count;
- return -1;
+ return err;
}
++count;
struct net *net = dev_net(skb->dev);
int local = skb_rtable(skb)->rt_flags & RTCF_LOCAL;
struct mr_table *mrt;
+ struct net_device *dev;
+
+ /* skb->dev passed in is the loX master dev for vrfs.
+ * As there are no vifs associated with loopback devices,
+ * get the proper interface that does have a vif associated with it.
+ */
+ dev = skb->dev;
+ if (netif_is_l3_master(skb->dev)) {
+ dev = dev_get_by_index_rcu(net, IPCB(skb)->iif);
+ if (!dev) {
+ kfree_skb(skb);
+ return -ENODEV;
+ }
+ }
/* Packet is looped back after forward, it should not be
* forwarded second time, but still can be delivered locally.
/* already under rcu_read_lock() */
cache = ipmr_cache_find(mrt, ip_hdr(skb)->saddr, ip_hdr(skb)->daddr);
if (!cache) {
- int vif = ipmr_find_vif(mrt, skb->dev);
+ int vif = ipmr_find_vif(mrt, dev);
if (vif >= 0)
cache = ipmr_cache_find_any(mrt, ip_hdr(skb)->daddr,
}
read_lock(&mrt_lock);
- vif = ipmr_find_vif(mrt, skb->dev);
+ vif = ipmr_find_vif(mrt, dev);
if (vif >= 0) {
int err2 = ipmr_cache_unresolved(mrt, vif, skb);
read_unlock(&mrt_lock);
static void ipv4_dst_destroy(struct dst_entry *dst)
{
+ struct dst_metrics *p = (struct dst_metrics *)DST_METRICS_PTR(dst);
struct rtable *rt = (struct rtable *) dst;
+ if (p != &dst_default_metrics && atomic_dec_and_test(&p->refcnt))
+ kfree(p);
+
if (!list_empty(&rt->rt_uncached)) {
struct uncached_list *ul = rt->rt_uncached_list;
rt->rt_gateway = nh->nh_gw;
rt->rt_uses_gateway = 1;
}
- dst_init_metrics(&rt->dst, fi->fib_metrics, true);
+ dst_init_metrics(&rt->dst, fi->fib_metrics->metrics, true);
+ if (fi->fib_metrics != &dst_default_metrics) {
+ rt->dst._metrics |= DST_METRICS_REFCOUNTED;
+ atomic_inc(&fi->fib_metrics->refcnt);
+ }
#ifdef CONFIG_IP_ROUTE_CLASSID
rt->dst.tclassid = nh->nh_tclassid;
#endif
{
struct tcp_sock *tp = tcp_sk(sk);
struct inet_sock *inet = inet_sk(sk);
+ struct sockaddr *uaddr = msg->msg_name;
int err, flags;
- if (!(sysctl_tcp_fastopen & TFO_CLIENT_ENABLE))
+ if (!(sysctl_tcp_fastopen & TFO_CLIENT_ENABLE) ||
+ (uaddr && msg->msg_namelen >= sizeof(uaddr->sa_family) &&
+ uaddr->sa_family == AF_UNSPEC))
return -EOPNOTSUPP;
if (tp->fastopen_req)
return -EALREADY; /* Another Fast Open is in progress */
}
}
flags = (msg->msg_flags & MSG_DONTWAIT) ? O_NONBLOCK : 0;
- err = __inet_stream_connect(sk->sk_socket, msg->msg_name,
+ err = __inet_stream_connect(sk->sk_socket, uaddr,
msg->msg_namelen, flags, 1);
/* fastopen_req could already be freed in __inet_stream_connect
* if the connection times out or gets rst
tcp_set_ca_state(sk, TCP_CA_Open);
tcp_clear_retrans(tp);
inet_csk_delack_init(sk);
+ /* Initialize rcv_mss to TCP_MIN_MSS to avoid division by 0
+ * issue in __tcp_select_window()
+ */
+ icsk->icsk_ack.rcv_mss = TCP_MIN_MSS;
tcp_init_send_head(sk);
memset(&tp->rx_opt, 0, sizeof(tp->rx_opt));
__sk_dst_reset(sk);
*/
if (pkt_len > mss) {
unsigned int new_len = (pkt_len / mss) * mss;
- if (!in_sack && new_len < pkt_len) {
+ if (!in_sack && new_len < pkt_len)
new_len += mss;
- if (new_len >= skb->len)
- return 0;
- }
pkt_len = new_len;
}
+
+ if (pkt_len >= skb->len && !in_sack)
+ return 0;
+
err = tcp_fragment(sk, skb, pkt_len, mss, GFP_ATOMIC);
if (err < 0)
return err;
int delta;
/* Non-retransmitted hole got filled? That's reordering */
- if (reord < prior_fackets)
+ if (reord < prior_fackets && reord <= tp->fackets_out)
tcp_update_reordering(sk, tp->fackets_out - reord, 0);
delta = tcp_is_fack(tp) ? pkts_acked :
udp_lib_rehash(sk, new_hash);
}
-int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
+static int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
{
int rc;
* Note that in the success and error cases, the skb is assumed to
* have either been requeued or freed.
*/
-int udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
+static int udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
{
struct udp_sock *up = udp_sk(sk);
int is_udplite = IS_UDPLITE(sk);
int flags, int *addr_len);
int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size,
int flags);
-int __udp_queue_rcv_skb(struct sock *sk, struct sk_buff *skb);
void udp_destroy_sock(struct sock *sk);
#ifdef CONFIG_PROC_FS
INIT_HLIST_NODE(&ifa->addr_lst);
ifa->scope = scope;
ifa->prefix_len = pfxlen;
- ifa->flags = flags | IFA_F_TENTATIVE;
+ ifa->flags = flags;
+ /* No need to add the TENTATIVE flag for addresses with NODAD */
+ if (!(flags & IFA_F_NODAD))
+ ifa->flags |= IFA_F_TENTATIVE;
ifa->valid_lft = valid_lft;
ifa->prefered_lft = prefered_lft;
ifa->cstamp = ifa->tstamp = jiffies;
memcpy(&fl6, &t->fl.u.ip6, sizeof(fl6));
- dsfield = ipv4_get_dsfield(iph);
-
if (t->parms.flags & IP6_TNL_F_USE_ORIG_TCLASS)
- fl6.flowlabel |= htonl((__u32)iph->tos << IPV6_TCLASS_SHIFT)
- & IPV6_TCLASS_MASK;
+ dsfield = ipv4_get_dsfield(iph);
+ else
+ dsfield = ip6_tclass(t->parms.flowinfo);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
fl6.flowi6_mark = skb->mark;
else
memcpy(&fl6, &t->fl.u.ip6, sizeof(fl6));
- dsfield = ipv6_get_dsfield(ipv6h);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_TCLASS)
- fl6.flowlabel |= (*(__be32 *) ipv6h & IPV6_TCLASS_MASK);
+ dsfield = ipv6_get_dsfield(ipv6h);
+ else
+ dsfield = ip6_tclass(t->parms.flowinfo);
+
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FLOWLABEL)
fl6.flowlabel |= ip6_flowlabel(ipv6h);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
const struct net_offload *ops;
int proto;
struct frag_hdr *fptr;
- unsigned int unfrag_ip6hlen;
unsigned int payload_len;
u8 *prevhdr;
int offset = 0;
skb->network_header = (u8 *)ipv6h - skb->head;
if (udpfrag) {
- unfrag_ip6hlen = ip6_find_1stfragopt(skb, &prevhdr);
- fptr = (struct frag_hdr *)((u8 *)ipv6h + unfrag_ip6hlen);
+ int err = ip6_find_1stfragopt(skb, &prevhdr);
+ if (err < 0)
+ return ERR_PTR(err);
+ fptr = (struct frag_hdr *)((u8 *)ipv6h + err);
fptr->frag_off = htons(offset);
if (skb->next)
fptr->frag_off |= htons(IP6_MF);
int ptr, offset = 0, err = 0;
u8 *prevhdr, nexthdr = 0;
- hlen = ip6_find_1stfragopt(skb, &prevhdr);
+ err = ip6_find_1stfragopt(skb, &prevhdr);
+ if (err < 0)
+ goto fail;
+ hlen = err;
nexthdr = *prevhdr;
mtu = ip6_skb_dst_mtu(skb);
*/
alloclen += sizeof(struct frag_hdr);
+ copy = datalen - transhdrlen - fraggap;
+ if (copy < 0) {
+ err = -EINVAL;
+ goto error;
+ }
if (transhdrlen) {
skb = sock_alloc_send_skb(sk,
alloclen + hh_len,
data += fraggap;
pskb_trim_unique(skb_prev, maxfraglen);
}
- copy = datalen - transhdrlen - fraggap;
-
- if (copy < 0) {
- err = -EINVAL;
- kfree_skb(skb);
- goto error;
- } else if (copy > 0 && getfrag(from, data + transhdrlen, offset, copy, fraggap, skb) < 0) {
+ if (copy > 0 &&
+ getfrag(from, data + transhdrlen, offset,
+ copy, fraggap, skb) < 0) {
err = -EFAULT;
kfree_skb(skb);
goto error;
skb_push(skb, sizeof(struct ipv6hdr));
skb_reset_network_header(skb);
ipv6h = ipv6_hdr(skb);
- ip6_flow_hdr(ipv6h, INET_ECN_encapsulate(0, dsfield),
+ ip6_flow_hdr(ipv6h, dsfield,
ip6_make_flowlabel(net, skb, fl6->flowlabel, true, fl6));
ipv6h->hop_limit = hop_limit;
ipv6h->nexthdr = proto;
if (tproto != IPPROTO_IPIP && tproto != 0)
return -1;
- dsfield = ipv4_get_dsfield(iph);
-
if (t->parms.collect_md) {
struct ip_tunnel_info *tun_info;
const struct ip_tunnel_key *key;
fl6.flowi6_proto = IPPROTO_IPIP;
fl6.daddr = key->u.ipv6.dst;
fl6.flowlabel = key->label;
+ dsfield = ip6_tclass(key->label);
} else {
if (!(t->parms.flags & IP6_TNL_F_IGN_ENCAP_LIMIT))
encap_limit = t->parms.encap_limit;
fl6.flowi6_proto = IPPROTO_IPIP;
if (t->parms.flags & IP6_TNL_F_USE_ORIG_TCLASS)
- fl6.flowlabel |= htonl((__u32)iph->tos << IPV6_TCLASS_SHIFT)
- & IPV6_TCLASS_MASK;
+ dsfield = ipv4_get_dsfield(iph);
+ else
+ dsfield = ip6_tclass(t->parms.flowinfo);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
fl6.flowi6_mark = skb->mark;
else
if (iptunnel_handle_offloads(skb, SKB_GSO_IPXIP6))
return -1;
+ dsfield = INET_ECN_encapsulate(dsfield, ipv4_get_dsfield(iph));
+
skb_set_inner_ipproto(skb, IPPROTO_IPIP);
err = ip6_tnl_xmit(skb, dev, dsfield, &fl6, encap_limit, &mtu,
ip6_tnl_addr_conflict(t, ipv6h))
return -1;
- dsfield = ipv6_get_dsfield(ipv6h);
-
if (t->parms.collect_md) {
struct ip_tunnel_info *tun_info;
const struct ip_tunnel_key *key;
fl6.flowi6_proto = IPPROTO_IPV6;
fl6.daddr = key->u.ipv6.dst;
fl6.flowlabel = key->label;
+ dsfield = ip6_tclass(key->label);
} else {
offset = ip6_tnl_parse_tlv_enc_lim(skb, skb_network_header(skb));
/* ip6_tnl_parse_tlv_enc_lim() might have reallocated skb->head */
fl6.flowi6_proto = IPPROTO_IPV6;
if (t->parms.flags & IP6_TNL_F_USE_ORIG_TCLASS)
- fl6.flowlabel |= (*(__be32 *)ipv6h & IPV6_TCLASS_MASK);
+ dsfield = ipv6_get_dsfield(ipv6h);
+ else
+ dsfield = ip6_tclass(t->parms.flowinfo);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FLOWLABEL)
fl6.flowlabel |= ip6_flowlabel(ipv6h);
if (t->parms.flags & IP6_TNL_F_USE_ORIG_FWMARK)
if (iptunnel_handle_offloads(skb, SKB_GSO_IPXIP6))
return -1;
+ dsfield = INET_ECN_encapsulate(dsfield, ipv6_get_dsfield(ipv6h));
+
skb_set_inner_ipproto(skb, IPPROTO_IPV6);
err = ip6_tnl_xmit(skb, dev, dsfield, &fl6, encap_limit, &mtu,
int ip6_find_1stfragopt(struct sk_buff *skb, u8 **nexthdr)
{
u16 offset = sizeof(struct ipv6hdr);
- struct ipv6_opt_hdr *exthdr =
- (struct ipv6_opt_hdr *)(ipv6_hdr(skb) + 1);
unsigned int packet_len = skb_tail_pointer(skb) -
skb_network_header(skb);
int found_rhdr = 0;
*nexthdr = &ipv6_hdr(skb)->nexthdr;
- while (offset + 1 <= packet_len) {
+ while (offset <= packet_len) {
+ struct ipv6_opt_hdr *exthdr;
switch (**nexthdr) {
return offset;
}
- offset += ipv6_optlen(exthdr);
- *nexthdr = &exthdr->nexthdr;
+ if (offset + sizeof(struct ipv6_opt_hdr) > packet_len)
+ return -EINVAL;
+
exthdr = (struct ipv6_opt_hdr *)(skb_network_header(skb) +
offset);
+ offset += ipv6_optlen(exthdr);
+ *nexthdr = &exthdr->nexthdr;
}
- return offset;
+ return -EINVAL;
}
EXPORT_SYMBOL(ip6_find_1stfragopt);
newtp->af_specific = &tcp_sock_ipv6_mapped_specific;
#endif
+ newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
newnp->ipv6_fl_list = NULL;
newnp->pktoptions = NULL;
First: no IPv4 options.
*/
newinet->inet_opt = NULL;
+ newnp->ipv6_mc_list = NULL;
newnp->ipv6_ac_list = NULL;
newnp->ipv6_fl_list = NULL;
return;
}
-int __udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
+static int __udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
{
int rc;
}
EXPORT_SYMBOL(udpv6_encap_enable);
-int udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
+static int udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
{
struct udp_sock *up = udp_sk(sk);
int is_udplite = IS_UDPLITE(sk);
int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len);
int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int noblock,
int flags, int *addr_len);
-int __udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb);
void udpv6_destroy_sock(struct sock *sk);
#ifdef CONFIG_PROC_FS
u8 frag_hdr_sz = sizeof(struct frag_hdr);
__wsum csum;
int tnl_hlen;
+ int err;
mss = skb_shinfo(skb)->gso_size;
if (unlikely(skb->len <= mss))
/* Find the unfragmentable header and shift it left by frag_hdr_sz
* bytes to insert fragment header.
*/
- unfrag_ip6hlen = ip6_find_1stfragopt(skb, &prevhdr);
+ err = ip6_find_1stfragopt(skb, &prevhdr);
+ if (err < 0)
+ return ERR_PTR(err);
+ unfrag_ip6hlen = err;
nexthdr = *prevhdr;
*prevhdr = NEXTHDR_FRAGMENT;
unfrag_len = (skb_network_header(skb) - skb_mac_header(skb)) +
p += pol->sadb_x_policy_len*8;
sec_ctx = (struct sadb_x_sec_ctx *)p;
if (len < pol->sadb_x_policy_len*8 +
- sec_ctx->sadb_x_sec_len) {
+ sec_ctx->sadb_x_sec_len*8) {
*dir = -EINVAL;
goto out;
}
int rc = -EINVAL;
dprintk("%s: binding %02X\n", __func__, addr->sllc_sap);
+
+ lock_sock(sk);
if (unlikely(!sock_flag(sk, SOCK_ZAPPED) || addrlen != sizeof(*addr)))
goto out;
rc = -EAFNOSUPPORT;
out_put:
llc_sap_put(sap);
out:
+ release_sock(sk);
return rc;
}
if (is_multicast_ether_addr(hdr->addr1)) {
mpp_addr = hdr->addr3;
proxied_addr = mesh_hdr->eaddr1;
- } else if (mesh_hdr->flags & MESH_FLAGS_AE_A5_A6) {
+ } else if ((mesh_hdr->flags & MESH_FLAGS_AE) ==
+ MESH_FLAGS_AE_A5_A6) {
/* has_a4 already checked in ieee80211_rx_mesh_check */
mpp_addr = hdr->addr4;
proxied_addr = mesh_hdr->eaddr2;
{
unsigned int verdict = NF_DROP;
- if (IP_VS_FWD_METHOD(cp) != 0) {
- pr_err("shouldn't reach here, because the box is on the "
- "half connection in the tun/dr module.\n");
- }
+ if (IP_VS_FWD_METHOD(cp) != IP_VS_CONN_F_MASQ)
+ goto ignore_cp;
/* Ensure the checksum is correct */
if (!skb_csum_unnecessary(skb) && ip_vs_checksum_complete(skb, ihl)) {
ip_vs_notrack(skb);
else
ip_vs_update_conntrack(skb, cp, 0);
+
+ignore_cp:
verdict = NF_ACCEPT;
out:
*/
cp = pp->conn_out_get(ipvs, af, skb, &iph);
- if (likely(cp))
+ if (likely(cp)) {
+ if (IP_VS_FWD_METHOD(cp) != IP_VS_CONN_F_MASQ)
+ goto ignore_cp;
return handle_response(af, skb, pd, cp, &iph, hooknum);
+ }
/* Check for real-server-started requests */
if (atomic_read(&ipvs->conn_out_counter)) {
}
}
}
+
+out:
IP_VS_DBG_PKT(12, af, pp, skb, iph.off,
"ip_vs_out: packet continues traversal as normal");
return NF_ACCEPT;
+
+ignore_cp:
+ __ip_vs_conn_put(cp);
+ goto out;
}
/*
#endif
if (h != NULL && !try_module_get(h->me))
h = NULL;
+ if (h != NULL && !refcount_inc_not_zero(&h->refcnt)) {
+ module_put(h->me);
+ h = NULL;
+ }
rcu_read_unlock();
}
EXPORT_SYMBOL_GPL(nf_conntrack_helper_try_module_get);
+void nf_conntrack_helper_put(struct nf_conntrack_helper *helper)
+{
+ refcount_dec(&helper->refcnt);
+ module_put(helper->me);
+}
+EXPORT_SYMBOL_GPL(nf_conntrack_helper_put);
+
struct nf_conn_help *
nf_ct_helper_ext_add(struct nf_conn *ct,
struct nf_conntrack_helper *helper, gfp_t gfp)
}
}
}
+ refcount_set(&me->refcnt, 1);
hlist_add_head_rcu(&me->hnode, &nf_ct_helper_hash[h]);
nf_ct_helper_count++;
out:
#include <net/netfilter/nf_conntrack_zones.h>
#include <net/netfilter/nf_conntrack_timestamp.h>
#include <net/netfilter/nf_conntrack_labels.h>
+#include <net/netfilter/nf_conntrack_seqadj.h>
+#include <net/netfilter/nf_conntrack_synproxy.h>
#ifdef CONFIG_NF_NAT_NEEDED
#include <net/netfilter/nf_nat_core.h>
#include <net/netfilter/nf_nat_l4proto.h>
static int
ctnetlink_parse_tuple(const struct nlattr * const cda[],
- struct nf_conntrack_tuple *tuple,
- enum ctattr_type type, u_int8_t l3num,
- struct nf_conntrack_zone *zone)
+ struct nf_conntrack_tuple *tuple, u32 type,
+ u_int8_t l3num, struct nf_conntrack_zone *zone)
{
struct nlattr *tb[CTA_TUPLE_MAX+1];
int err;
nf_ct_tstamp_ext_add(ct, GFP_ATOMIC);
nf_ct_ecache_ext_add(ct, 0, 0, GFP_ATOMIC);
nf_ct_labels_ext_add(ct);
+ nfct_seqadj_ext_add(ct);
+ nfct_synproxy_ext_add(ct);
/* we must add conntrack extensions before confirmation. */
ct->status |= IPS_CONFIRMED;
static int ctnetlink_exp_dump_tuple(struct sk_buff *skb,
const struct nf_conntrack_tuple *tuple,
- enum ctattr_expect type)
+ u32 type)
{
struct nlattr *nest_parms;
{
struct nf_conntrack_tuple curr_tuple, new_tuple;
+ /* Can't setup nat info for confirmed ct. */
+ if (nf_ct_is_confirmed(ct))
+ return NF_ACCEPT;
+
NF_CT_ASSERT(maniptype == NF_NAT_MANIP_SRC ||
maniptype == NF_NAT_MANIP_DST);
BUG_ON(nf_nat_initialized(ct, maniptype));
return nf_tables_fill_setelem(args->skb, set, elem);
}
+struct nft_set_dump_ctx {
+ const struct nft_set *set;
+ struct nft_ctx ctx;
+};
+
static int nf_tables_dump_set(struct sk_buff *skb, struct netlink_callback *cb)
{
+ struct nft_set_dump_ctx *dump_ctx = cb->data;
struct net *net = sock_net(skb->sk);
- u8 genmask = nft_genmask_cur(net);
+ struct nft_af_info *afi;
+ struct nft_table *table;
struct nft_set *set;
struct nft_set_dump_args args;
- struct nft_ctx ctx;
- struct nlattr *nla[NFTA_SET_ELEM_LIST_MAX + 1];
+ bool set_found = false;
struct nfgenmsg *nfmsg;
struct nlmsghdr *nlh;
struct nlattr *nest;
u32 portid, seq;
- int event, err;
+ int event;
- err = nlmsg_parse(cb->nlh, sizeof(struct nfgenmsg), nla,
- NFTA_SET_ELEM_LIST_MAX, nft_set_elem_list_policy,
- NULL);
- if (err < 0)
- return err;
+ rcu_read_lock();
+ list_for_each_entry_rcu(afi, &net->nft.af_info, list) {
+ if (afi != dump_ctx->ctx.afi)
+ continue;
- err = nft_ctx_init_from_elemattr(&ctx, net, cb->skb, cb->nlh,
- (void *)nla, genmask);
- if (err < 0)
- return err;
+ list_for_each_entry_rcu(table, &afi->tables, list) {
+ if (table != dump_ctx->ctx.table)
+ continue;
- set = nf_tables_set_lookup(ctx.table, nla[NFTA_SET_ELEM_LIST_SET],
- genmask);
- if (IS_ERR(set))
- return PTR_ERR(set);
+ list_for_each_entry_rcu(set, &table->sets, list) {
+ if (set == dump_ctx->set) {
+ set_found = true;
+ break;
+ }
+ }
+ break;
+ }
+ break;
+ }
+
+ if (!set_found) {
+ rcu_read_unlock();
+ return -ENOENT;
+ }
event = nfnl_msg_type(NFNL_SUBSYS_NFTABLES, NFT_MSG_NEWSETELEM);
portid = NETLINK_CB(cb->skb).portid;
goto nla_put_failure;
nfmsg = nlmsg_data(nlh);
- nfmsg->nfgen_family = ctx.afi->family;
+ nfmsg->nfgen_family = afi->family;
nfmsg->version = NFNETLINK_V0;
- nfmsg->res_id = htons(ctx.net->nft.base_seq & 0xffff);
+ nfmsg->res_id = htons(net->nft.base_seq & 0xffff);
- if (nla_put_string(skb, NFTA_SET_ELEM_LIST_TABLE, ctx.table->name))
+ if (nla_put_string(skb, NFTA_SET_ELEM_LIST_TABLE, table->name))
goto nla_put_failure;
if (nla_put_string(skb, NFTA_SET_ELEM_LIST_SET, set->name))
goto nla_put_failure;
args.cb = cb;
args.skb = skb;
- args.iter.genmask = nft_genmask_cur(ctx.net);
+ args.iter.genmask = nft_genmask_cur(net);
args.iter.skip = cb->args[0];
args.iter.count = 0;
args.iter.err = 0;
args.iter.fn = nf_tables_dump_setelem;
- set->ops->walk(&ctx, set, &args.iter);
+ set->ops->walk(&dump_ctx->ctx, set, &args.iter);
+ rcu_read_unlock();
nla_nest_end(skb, nest);
nlmsg_end(skb, nlh);
return skb->len;
nla_put_failure:
+ rcu_read_unlock();
return -ENOSPC;
}
+static int nf_tables_dump_set_done(struct netlink_callback *cb)
+{
+ kfree(cb->data);
+ return 0;
+}
+
static int nf_tables_getsetelem(struct net *net, struct sock *nlsk,
struct sk_buff *skb, const struct nlmsghdr *nlh,
const struct nlattr * const nla[])
if (nlh->nlmsg_flags & NLM_F_DUMP) {
struct netlink_dump_control c = {
.dump = nf_tables_dump_set,
+ .done = nf_tables_dump_set_done,
};
+ struct nft_set_dump_ctx *dump_ctx;
+
+ dump_ctx = kmalloc(sizeof(*dump_ctx), GFP_KERNEL);
+ if (!dump_ctx)
+ return -ENOMEM;
+
+ dump_ctx->set = set;
+ dump_ctx->ctx = ctx;
+
+ c.data = dump_ctx;
return netlink_dump_start(nlsk, skb, nlh, &c);
}
return -EOPNOTSUPP;
{
struct nft_set_ext *ext = nft_set_elem_ext(set, elem);
- nft_data_uninit(nft_set_ext_key(ext), NFT_DATA_VALUE);
+ nft_data_release(nft_set_ext_key(ext), NFT_DATA_VALUE);
if (nft_set_ext_exists(ext, NFT_SET_EXT_DATA))
- nft_data_uninit(nft_set_ext_data(ext), set->dtype);
+ nft_data_release(nft_set_ext_data(ext), set->dtype);
if (destroy_expr && nft_set_ext_exists(ext, NFT_SET_EXT_EXPR))
nf_tables_expr_destroy(NULL, nft_set_ext_expr(ext));
if (nft_set_ext_exists(ext, NFT_SET_EXT_OBJREF))
}
EXPORT_SYMBOL_GPL(nft_set_elem_destroy);
+/* Only called from commit path, nft_set_elem_deactivate() already deals with
+ * the refcounting from the preparation phase.
+ */
+static void nf_tables_set_elem_destroy(const struct nft_set *set, void *elem)
+{
+ struct nft_set_ext *ext = nft_set_elem_ext(set, elem);
+
+ if (nft_set_ext_exists(ext, NFT_SET_EXT_EXPR))
+ nf_tables_expr_destroy(NULL, nft_set_ext_expr(ext));
+ kfree(elem);
+}
+
static int nft_setelem_parse_flags(const struct nft_set *set,
const struct nlattr *attr, u32 *flags)
{
kfree(elem.priv);
err3:
if (nla[NFTA_SET_ELEM_DATA] != NULL)
- nft_data_uninit(&data, d2.type);
+ nft_data_release(&data, d2.type);
err2:
- nft_data_uninit(&elem.key.val, d1.type);
+ nft_data_release(&elem.key.val, d1.type);
err1:
return err;
}
return err;
}
+/**
+ * nft_data_hold - hold a nft_data item
+ *
+ * @data: struct nft_data to release
+ * @type: type of data
+ *
+ * Hold a nft_data item. NFT_DATA_VALUE types can be silently discarded,
+ * NFT_DATA_VERDICT bumps the reference to chains in case of NFT_JUMP and
+ * NFT_GOTO verdicts. This function must be called on active data objects
+ * from the second phase of the commit protocol.
+ */
+static void nft_data_hold(const struct nft_data *data, enum nft_data_types type)
+{
+ if (type == NFT_DATA_VERDICT) {
+ switch (data->verdict.code) {
+ case NFT_JUMP:
+ case NFT_GOTO:
+ data->verdict.chain->use++;
+ break;
+ }
+ }
+}
+
+static void nft_set_elem_activate(const struct net *net,
+ const struct nft_set *set,
+ struct nft_set_elem *elem)
+{
+ const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv);
+
+ if (nft_set_ext_exists(ext, NFT_SET_EXT_DATA))
+ nft_data_hold(nft_set_ext_data(ext), set->dtype);
+ if (nft_set_ext_exists(ext, NFT_SET_EXT_OBJREF))
+ (*nft_set_ext_obj(ext))->use++;
+}
+
+static void nft_set_elem_deactivate(const struct net *net,
+ const struct nft_set *set,
+ struct nft_set_elem *elem)
+{
+ const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv);
+
+ if (nft_set_ext_exists(ext, NFT_SET_EXT_DATA))
+ nft_data_release(nft_set_ext_data(ext), set->dtype);
+ if (nft_set_ext_exists(ext, NFT_SET_EXT_OBJREF))
+ (*nft_set_ext_obj(ext))->use--;
+}
+
static int nft_del_setelem(struct nft_ctx *ctx, struct nft_set *set,
const struct nlattr *attr)
{
kfree(elem.priv);
elem.priv = priv;
+ nft_set_elem_deactivate(ctx->net, set, &elem);
+
nft_trans_elem(trans) = elem;
list_add_tail(&trans->list, &ctx->net->nft.commit_list);
return 0;
err3:
kfree(elem.priv);
err2:
- nft_data_uninit(&elem.key.val, desc.type);
+ nft_data_release(&elem.key.val, desc.type);
err1:
return err;
}
nft_set_destroy(nft_trans_set(trans));
break;
case NFT_MSG_DELSETELEM:
- nft_set_elem_destroy(nft_trans_elem_set(trans),
- nft_trans_elem(trans).priv, true);
+ nf_tables_set_elem_destroy(nft_trans_elem_set(trans),
+ nft_trans_elem(trans).priv);
break;
case NFT_MSG_DELOBJ:
nft_obj_destroy(nft_trans_obj(trans));
case NFT_MSG_DELSETELEM:
te = (struct nft_trans_elem *)trans->data;
+ nft_set_elem_activate(net, te->set, &te->elem);
te->set->ops->activate(net, te->set, &te->elem);
te->set->ndeact--;
EXPORT_SYMBOL_GPL(nft_data_init);
/**
- * nft_data_uninit - release a nft_data item
+ * nft_data_release - release a nft_data item
*
* @data: struct nft_data to release
* @type: type of data
* Release a nft_data item. NFT_DATA_VALUE types can be silently discarded,
* all others need to be released by calling this function.
*/
-void nft_data_uninit(const struct nft_data *data, enum nft_data_types type)
+void nft_data_release(const struct nft_data *data, enum nft_data_types type)
{
if (type < NFT_DATA_VERDICT)
return;
WARN_ON(1);
}
}
-EXPORT_SYMBOL_GPL(nft_data_uninit);
+EXPORT_SYMBOL_GPL(nft_data_release);
int nft_data_dump(struct sk_buff *skb, int attr, const struct nft_data *data,
enum nft_data_types type, unsigned int len)
tuple_set = true;
}
+ ret = -ENOENT;
list_for_each_entry_safe(nlcth, n, &nfnl_cthelper_list, list) {
cur = &nlcth->helper;
j++;
tuple.dst.protonum != cur->tuple.dst.protonum))
continue;
- found = true;
- nf_conntrack_helper_unregister(cur);
- kfree(cur->expect_policy);
+ if (refcount_dec_if_one(&cur->refcnt)) {
+ found = true;
+ nf_conntrack_helper_unregister(cur);
+ kfree(cur->expect_policy);
- list_del(&nlcth->list);
- kfree(nlcth);
+ list_del(&nlcth->list);
+ kfree(nlcth);
+ } else {
+ ret = -EBUSY;
+ }
}
/* Make sure we return success if we flush and there is no helpers */
- return (found || j == 0) ? 0 : -ENOENT;
+ return (found || j == 0) ? 0 : ret;
}
static const struct nla_policy nfnl_cthelper_policy[NFCTH_MAX+1] = {
tb[NFTA_BITWISE_MASK]);
if (err < 0)
return err;
- if (d1.len != priv->len)
- return -EINVAL;
+ if (d1.len != priv->len) {
+ err = -EINVAL;
+ goto err1;
+ }
err = nft_data_init(NULL, &priv->xor, sizeof(priv->xor), &d2,
tb[NFTA_BITWISE_XOR]);
if (err < 0)
- return err;
- if (d2.len != priv->len)
- return -EINVAL;
+ goto err1;
+ if (d2.len != priv->len) {
+ err = -EINVAL;
+ goto err2;
+ }
return 0;
+err2:
+ nft_data_release(&priv->xor, d2.type);
+err1:
+ nft_data_release(&priv->mask, d1.type);
+ return err;
}
static int nft_bitwise_dump(struct sk_buff *skb, const struct nft_expr *expr)
if (err < 0)
return ERR_PTR(err);
+ if (desc.type != NFT_DATA_VALUE) {
+ err = -EINVAL;
+ goto err1;
+ }
+
if (desc.len <= sizeof(u32) && op == NFT_CMP_EQ)
return &nft_cmp_fast_ops;
- else
- return &nft_cmp_ops;
+
+ return &nft_cmp_ops;
+err1:
+ nft_data_release(&data, desc.type);
+ return ERR_PTR(-EINVAL);
}
struct nft_expr_type nft_cmp_type __read_mostly = {
struct nft_ct_helper_obj *priv = nft_obj_data(obj);
if (priv->helper4)
- module_put(priv->helper4->me);
+ nf_conntrack_helper_put(priv->helper4);
if (priv->helper6)
- module_put(priv->helper6->me);
+ nf_conntrack_helper_put(priv->helper6);
}
static void nft_ct_helper_obj_eval(struct nft_object *obj,
return 0;
err1:
- nft_data_uninit(&priv->data, desc.type);
+ nft_data_release(&priv->data, desc.type);
return err;
}
const struct nft_expr *expr)
{
const struct nft_immediate_expr *priv = nft_expr_priv(expr);
- return nft_data_uninit(&priv->data, nft_dreg_to_type(priv->dreg));
+
+ return nft_data_release(&priv->data, nft_dreg_to_type(priv->dreg));
}
static int nft_immediate_dump(struct sk_buff *skb, const struct nft_expr *expr)
priv->len = desc_from.len;
return 0;
err2:
- nft_data_uninit(&priv->data_to, desc_to.type);
+ nft_data_release(&priv->data_to, desc_to.type);
err1:
- nft_data_uninit(&priv->data_from, desc_from.type);
+ nft_data_release(&priv->data_from, desc_from.type);
return err;
}
struct nft_set_elem elem;
int err;
- err = rhashtable_walk_init(&priv->ht, &hti, GFP_KERNEL);
+ err = rhashtable_walk_init(&priv->ht, &hti, GFP_ATOMIC);
iter->err = err;
if (err)
return;
&U->u.user.revision, K->u.kernel.TYPE->revision)
int xt_data_to_user(void __user *dst, const void *src,
- int usersize, int size)
+ int usersize, int size, int aligned_size)
{
usersize = usersize ? : size;
if (copy_to_user(dst, src, usersize))
return -EFAULT;
- if (usersize != size && clear_user(dst + usersize, size - usersize))
+ if (usersize != aligned_size &&
+ clear_user(dst + usersize, aligned_size - usersize))
return -EFAULT;
return 0;
}
EXPORT_SYMBOL_GPL(xt_data_to_user);
-#define XT_DATA_TO_USER(U, K, TYPE, C_SIZE) \
+#define XT_DATA_TO_USER(U, K, TYPE) \
xt_data_to_user(U->data, K->data, \
K->u.kernel.TYPE->usersize, \
- C_SIZE ? : K->u.kernel.TYPE->TYPE##size)
+ K->u.kernel.TYPE->TYPE##size, \
+ XT_ALIGN(K->u.kernel.TYPE->TYPE##size))
int xt_match_to_user(const struct xt_entry_match *m,
struct xt_entry_match __user *u)
{
return XT_OBJ_TO_USER(u, m, match, 0) ||
- XT_DATA_TO_USER(u, m, match, 0);
+ XT_DATA_TO_USER(u, m, match);
}
EXPORT_SYMBOL_GPL(xt_match_to_user);
struct xt_entry_target __user *u)
{
return XT_OBJ_TO_USER(u, t, target, 0) ||
- XT_DATA_TO_USER(u, t, target, 0);
+ XT_DATA_TO_USER(u, t, target);
}
EXPORT_SYMBOL_GPL(xt_target_to_user);
}
EXPORT_SYMBOL_GPL(xt_compat_match_from_user);
+#define COMPAT_XT_DATA_TO_USER(U, K, TYPE, C_SIZE) \
+ xt_data_to_user(U->data, K->data, \
+ K->u.kernel.TYPE->usersize, \
+ C_SIZE, \
+ COMPAT_XT_ALIGN(C_SIZE))
+
int xt_compat_match_to_user(const struct xt_entry_match *m,
void __user **dstptr, unsigned int *size)
{
if (match->compat_to_user((void __user *)cm->data, m->data))
return -EFAULT;
} else {
- if (XT_DATA_TO_USER(cm, m, match, msize - sizeof(*cm)))
+ if (COMPAT_XT_DATA_TO_USER(cm, m, match, msize - sizeof(*cm)))
return -EFAULT;
}
if (target->compat_to_user((void __user *)ct->data, t->data))
return -EFAULT;
} else {
- if (XT_DATA_TO_USER(ct, t, target, tsize - sizeof(*ct)))
+ if (COMPAT_XT_DATA_TO_USER(ct, t, target, tsize - sizeof(*ct)))
return -EFAULT;
}
help = nf_ct_helper_ext_add(ct, helper, GFP_KERNEL);
if (help == NULL) {
- module_put(helper->me);
+ nf_conntrack_helper_put(helper);
return -ENOMEM;
}
err4:
help = nfct_help(ct);
if (help)
- module_put(help->helper->me);
+ nf_conntrack_helper_put(help->helper);
err3:
nf_ct_tmpl_free(ct);
err2:
if (ct) {
help = nfct_help(ct);
if (help)
- module_put(help->helper->me);
+ nf_conntrack_helper_put(help->helper);
nf_ct_netns_put(par->net, par->family);
help = nf_ct_helper_ext_add(info->ct, helper, GFP_KERNEL);
if (!help) {
- module_put(helper->me);
+ nf_conntrack_helper_put(helper);
return -ENOMEM;
}
static void __ovs_ct_free_action(struct ovs_conntrack_info *ct_info)
{
if (ct_info->helper)
- module_put(ct_info->helper->me);
+ nf_conntrack_helper_put(ct_info->helper);
if (ct_info->ct)
nf_ct_tmpl_free(ct_info->ct);
}
dev = dev_get_by_index(sock_net(&po->sk), saddr->sll_ifindex);
}
- sockc.tsflags = po->sk.sk_tsflags;
- if (msg->msg_controllen) {
- err = sock_cmsg_send(&po->sk, msg, &sockc);
- if (unlikely(err))
- goto out;
- }
-
err = -ENXIO;
if (unlikely(dev == NULL))
goto out;
if (unlikely(!(dev->flags & IFF_UP)))
goto out_put;
+ sockc.tsflags = po->sk.sk_tsflags;
+ if (msg->msg_controllen) {
+ err = sock_cmsg_send(&po->sk, msg, &sockc);
+ if (unlikely(err))
+ goto out_put;
+ }
+
if (po->sk.sk_socket->type == SOCK_RAW)
reserve = dev->hard_header_len;
size_max = po->tx_ring.frame_size
*arg = (unsigned long) head;
rcu_assign_pointer(tp->root, new);
- call_rcu(&head->rcu, mall_destroy_rcu);
return 0;
err_replace_hw_filter:
if (!qdisc_dev(root))
return 0;
+ if (tcm->tcm_parent) {
+ q = qdisc_match_from_root(root, TC_H_MAJ(tcm->tcm_parent));
+ if (q && tc_dump_tclass_qdisc(q, skb, tcm, cb, t_p, s_t) < 0)
+ return -1;
+ return 0;
+ }
hash_for_each(qdisc_dev(root)->qdisc_hash, b, q, hash) {
if (tc_dump_tclass_qdisc(q, skb, tcm, cb, t_p, s_t) < 0)
return -1;
asoc->ctsn_ack_point = asoc->next_tsn - 1;
asoc->adv_peer_ack_point = asoc->ctsn_ack_point;
- if (!asoc->stream) {
+
+ if (sctp_state(asoc, COOKIE_WAIT)) {
+ sctp_stream_free(asoc->stream);
asoc->stream = new->stream;
new->stream = NULL;
}
struct sctp_association **app,
struct sctp_transport **tpp)
{
+ struct sctp_init_chunk *chunkhdr, _chunkhdr;
union sctp_addr saddr;
union sctp_addr daddr;
struct sctp_af *af;
struct sock *sk = NULL;
struct sctp_association *asoc;
struct sctp_transport *transport = NULL;
- struct sctp_init_chunk *chunkhdr;
__u32 vtag = ntohl(sctphdr->vtag);
- int len = skb->len - ((void *)sctphdr - (void *)skb->data);
*app = NULL; *tpp = NULL;
* discard the packet.
*/
if (vtag == 0) {
- chunkhdr = (void *)sctphdr + sizeof(struct sctphdr);
- if (len < sizeof(struct sctphdr) + sizeof(sctp_chunkhdr_t)
- + sizeof(__be32) ||
+ /* chunk header + first 4 octects of init header */
+ chunkhdr = skb_header_pointer(skb, skb_transport_offset(skb) +
+ sizeof(struct sctphdr),
+ sizeof(struct sctp_chunkhdr) +
+ sizeof(__be32), &_chunkhdr);
+ if (!chunkhdr ||
chunkhdr->chunk_hdr.type != SCTP_CID_INIT ||
- ntohl(chunkhdr->init_hdr.init_tag) != asoc->c.my_vtag) {
+ ntohl(chunkhdr->init_hdr.init_tag) != asoc->c.my_vtag)
goto out;
- }
+
} else if (vtag != asoc->c.peer_vtag) {
goto out;
}
struct sctp_bind_addr *bp;
struct ipv6_pinfo *np = inet6_sk(sk);
struct sctp_sockaddr_entry *laddr;
- union sctp_addr *baddr = NULL;
union sctp_addr *daddr = &t->ipaddr;
union sctp_addr dst_saddr;
struct in6_addr *final_p, final;
__u8 matchlen = 0;
- __u8 bmatchlen;
sctp_scope_t scope;
memset(fl6, 0, sizeof(struct flowi6));
*/
rcu_read_lock();
list_for_each_entry_rcu(laddr, &bp->address_list, list) {
- if (!laddr->valid)
+ struct dst_entry *bdst;
+ __u8 bmatchlen;
+
+ if (!laddr->valid ||
+ laddr->state != SCTP_ADDR_SRC ||
+ laddr->a.sa.sa_family != AF_INET6 ||
+ scope > sctp_scope(&laddr->a))
continue;
- if ((laddr->state == SCTP_ADDR_SRC) &&
- (laddr->a.sa.sa_family == AF_INET6) &&
- (scope <= sctp_scope(&laddr->a))) {
- bmatchlen = sctp_v6_addr_match_len(daddr, &laddr->a);
- if (!baddr || (matchlen < bmatchlen)) {
- baddr = &laddr->a;
- matchlen = bmatchlen;
- }
- }
- }
- if (baddr) {
- fl6->saddr = baddr->v6.sin6_addr;
- fl6->fl6_sport = baddr->v6.sin6_port;
+
+ fl6->saddr = laddr->a.v6.sin6_addr;
+ fl6->fl6_sport = laddr->a.v6.sin6_port;
final_p = fl6_update_dst(fl6, rcu_dereference(np->opt), &final);
- dst = ip6_dst_lookup_flow(sk, fl6, final_p);
+ bdst = ip6_dst_lookup_flow(sk, fl6, final_p);
+
+ if (!IS_ERR(bdst) &&
+ ipv6_chk_addr(dev_net(bdst->dev),
+ &laddr->a.v6.sin6_addr, bdst->dev, 1)) {
+ if (!IS_ERR_OR_NULL(dst))
+ dst_release(dst);
+ dst = bdst;
+ break;
+ }
+
+ bmatchlen = sctp_v6_addr_match_len(daddr, &laddr->a);
+ if (matchlen > bmatchlen)
+ continue;
+
+ if (!IS_ERR_OR_NULL(dst))
+ dst_release(dst);
+ dst = bdst;
+ matchlen = bmatchlen;
}
rcu_read_unlock();
newnp = inet6_sk(newsk);
memcpy(newnp, np, sizeof(struct ipv6_pinfo));
+ newnp->ipv6_mc_list = NULL;
+ newnp->ipv6_ac_list = NULL;
+ newnp->ipv6_fl_list = NULL;
rcu_read_lock();
opt = rcu_dereference(np->opt);
* stream sequence number shall be set to 0.
*/
- /* Allocate storage for the negotiated streams if it is not a temporary
- * association.
- */
- if (!asoc->temp) {
- if (sctp_stream_init(asoc, gfp))
- goto clean_up;
+ if (sctp_stream_init(asoc, gfp))
+ goto clean_up;
- if (sctp_assoc_set_id(asoc, gfp))
- goto clean_up;
- }
+ if (!asoc->temp && sctp_assoc_set_id(asoc, gfp))
+ goto clean_up;
/* ADDIP Section 4.1 ASCONF Chunk Procedures
*
}
}
+ /* Set temp so that it won't be added into hashtable */
+ new_asoc->temp = 1;
+
/* Compare the tie_tag in cookie with the verification tag of
* current association.
*/
The Linux implementation of the SMC-R solution is designed as
a separate socket family SMC.
+ Warning: SMC will expose all memory for remote reads and writes
+ once a connection is established. Don't enable this option except
+ for tightly controlled lab environment.
+
Select this option if you want to run SMC socket applications
config SMC_DIAG
memcpy(&cclc.lcl.mac, &link->smcibdev->mac[link->ibport - 1], ETH_ALEN);
hton24(cclc.qpn, link->roce_qp->qp_num);
cclc.rmb_rkey =
- htonl(conn->rmb_desc->mr_rx[SMC_SINGLE_LINK]->rkey);
+ htonl(conn->rmb_desc->rkey[SMC_SINGLE_LINK]);
cclc.conn_idx = 1; /* for now: 1 RMB = 1 RMBE */
cclc.rmbe_alert_token = htonl(conn->alert_token_local);
cclc.qp_mtu = min(link->path_mtu, link->peer_mtu);
memcpy(&aclc.lcl.mac, link->smcibdev->mac[link->ibport - 1], ETH_ALEN);
hton24(aclc.qpn, link->roce_qp->qp_num);
aclc.rmb_rkey =
- htonl(conn->rmb_desc->mr_rx[SMC_SINGLE_LINK]->rkey);
+ htonl(conn->rmb_desc->rkey[SMC_SINGLE_LINK]);
aclc.conn_idx = 1; /* as long as 1 RMB = 1 RMBE */
aclc.rmbe_alert_token = htonl(conn->alert_token_local);
aclc.qp_mtu = link->path_mtu;
rmb_desc = NULL;
continue; /* if mapping failed, try smaller one */
}
- rc = smc_ib_get_memory_region(lgr->lnk[SMC_SINGLE_LINK].roce_pd,
- IB_ACCESS_REMOTE_WRITE |
- IB_ACCESS_LOCAL_WRITE,
- &rmb_desc->mr_rx[SMC_SINGLE_LINK]);
- if (rc) {
- smc_ib_buf_unmap(lgr->lnk[SMC_SINGLE_LINK].smcibdev,
- tmp_bufsize, rmb_desc,
- DMA_FROM_DEVICE);
- kfree(rmb_desc->cpu_addr);
- kfree(rmb_desc);
- rmb_desc = NULL;
- continue;
- }
+ rmb_desc->rkey[SMC_SINGLE_LINK] =
+ lgr->lnk[SMC_SINGLE_LINK].roce_pd->unsafe_global_rkey;
rmb_desc->used = 1;
write_lock_bh(&lgr->rmbs_lock);
list_add(&rmb_desc->list,
for (i = 0; i < SMC_RMBS_PER_LGR_MAX; i++) {
if ((lgr->rtokens[i][SMC_SINGLE_LINK].rkey == rkey) &&
+ (lgr->rtokens[i][SMC_SINGLE_LINK].dma_addr == dma_addr) &&
test_bit(i, lgr->rtokens_used_mask)) {
conn->rtoken_idx = i;
return 0;
u64 dma_addr[SMC_LINKS_PER_LGR_MAX];
/* mapped address of buffer */
void *cpu_addr; /* virtual address of buffer */
- struct ib_mr *mr_rx[SMC_LINKS_PER_LGR_MAX];
+ u32 rkey[SMC_LINKS_PER_LGR_MAX];
/* for rmb only:
* rkey provided to peer
*/
* identifier
*/
-int smc_ib_get_memory_region(struct ib_pd *pd, int access_flags,
- struct ib_mr **mr)
-{
- int rc;
-
- if (*mr)
- return 0; /* already done */
-
- /* obtain unique key -
- * next invocation of get_dma_mr returns a different key!
- */
- *mr = pd->device->get_dma_mr(pd, access_flags);
- rc = PTR_ERR_OR_ZERO(*mr);
- if (IS_ERR(*mr))
- *mr = NULL;
- return rc;
-}
-
static int smc_ib_modify_qp_init(struct smc_link *lnk)
{
struct ib_qp_attr qp_attr;
{
int rc;
- lnk->roce_pd = ib_alloc_pd(lnk->smcibdev->ibdev, 0);
+ lnk->roce_pd = ib_alloc_pd(lnk->smcibdev->ibdev,
+ IB_PD_UNSAFE_GLOBAL_RKEY);
rc = PTR_ERR_OR_ZERO(lnk->roce_pd);
if (IS_ERR(lnk->roce_pd))
lnk->roce_pd = NULL;
int smc_ib_create_protection_domain(struct smc_link *lnk);
void smc_ib_destroy_queue_pair(struct smc_link *lnk);
int smc_ib_create_queue_pair(struct smc_link *lnk);
-int smc_ib_get_memory_region(struct ib_pd *pd, int access_flags,
- struct ib_mr **mr);
int smc_ib_ready_link(struct smc_link *lnk);
int smc_ib_modify_qp_rts(struct smc_link *lnk);
int smc_ib_modify_qp_reset(struct smc_link *lnk);
return 0;
}
-#define tipc_wait_for_cond(sock_, timeout_, condition_) \
-({ \
- int rc_ = 0; \
- int done_ = 0; \
- \
- while (!(condition_) && !done_) { \
- struct sock *sk_ = sock->sk; \
- DEFINE_WAIT_FUNC(wait_, woken_wake_function); \
- \
- rc_ = tipc_sk_sock_err(sock_, timeout_); \
- if (rc_) \
- break; \
- prepare_to_wait(sk_sleep(sk_), &wait_, \
- TASK_INTERRUPTIBLE); \
- done_ = sk_wait_event(sk_, timeout_, \
- (condition_), &wait_); \
- remove_wait_queue(sk_sleep(sk_), &wait_); \
- } \
- rc_; \
+#define tipc_wait_for_cond(sock_, timeo_, condition_) \
+({ \
+ struct sock *sk_; \
+ int rc_; \
+ \
+ while ((rc_ = !(condition_))) { \
+ DEFINE_WAIT_FUNC(wait_, woken_wake_function); \
+ sk_ = (sock_)->sk; \
+ rc_ = tipc_sk_sock_err((sock_), timeo_); \
+ if (rc_) \
+ break; \
+ prepare_to_wait(sk_sleep(sk_), &wait_, TASK_INTERRUPTIBLE); \
+ release_sock(sk_); \
+ *(timeo_) = wait_woken(&wait_, TASK_INTERRUPTIBLE, *(timeo_)); \
+ sched_annotate_sleep(); \
+ lock_sock(sk_); \
+ remove_wait_queue(sk_sleep(sk_), &wait_); \
+ } \
+ rc_; \
})
/**
long timeout;
int err;
struct vsock_transport_send_notify_data send_data;
-
- DEFINE_WAIT(wait);
+ DEFINE_WAIT_FUNC(wait, woken_wake_function);
sk = sock->sk;
vsk = vsock_sk(sk);
if (err < 0)
goto out;
-
while (total_written < len) {
ssize_t written;
- prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+ add_wait_queue(sk_sleep(sk), &wait);
while (vsock_stream_has_space(vsk) == 0 &&
sk->sk_err == 0 &&
!(sk->sk_shutdown & SEND_SHUTDOWN) &&
/* Don't wait for non-blocking sockets. */
if (timeout == 0) {
err = -EAGAIN;
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
goto out_err;
}
err = transport->notify_send_pre_block(vsk, &send_data);
if (err < 0) {
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
goto out_err;
}
release_sock(sk);
- timeout = schedule_timeout(timeout);
+ timeout = wait_woken(&wait, TASK_INTERRUPTIBLE, timeout);
lock_sock(sk);
if (signal_pending(current)) {
err = sock_intr_errno(timeout);
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
goto out_err;
} else if (timeout == 0) {
err = -EAGAIN;
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
goto out_err;
}
-
- prepare_to_wait(sk_sleep(sk), &wait,
- TASK_INTERRUPTIBLE);
}
- finish_wait(sk_sleep(sk), &wait);
+ remove_wait_queue(sk_sleep(sk), &wait);
/* These checks occur both as part of and after the loop
* conditional since we need to check before and after
{
struct cfg80211_sched_scan_request *pos;
- ASSERT_RTNL();
+ WARN_ON_ONCE(!rcu_read_lock_held() && !lockdep_rtnl_is_held());
- list_for_each_entry(pos, &rdev->sched_scan_req_list, list) {
+ list_for_each_entry_rcu(pos, &rdev->sched_scan_req_list, list) {
if (pos->reqid == reqid)
return pos;
}
trace_cfg80211_sched_scan_results(wiphy, reqid);
/* ignore if we're not scanning */
- rtnl_lock();
+ rcu_read_lock();
request = cfg80211_find_sched_scan_req(rdev, reqid);
if (request) {
request->report_results = true;
queue_work(cfg80211_wq, &rdev->sched_scan_res_wk);
}
- rtnl_unlock();
+ rcu_read_unlock();
}
EXPORT_SYMBOL(cfg80211_sched_scan_results);
if (iftype == NL80211_IFTYPE_MESH_POINT)
skb_copy_bits(skb, hdrlen, &mesh_flags, 1);
+ mesh_flags &= MESH_FLAGS_AE;
+
switch (hdr->frame_control &
cpu_to_le16(IEEE80211_FCTL_TODS | IEEE80211_FCTL_FROMDS)) {
case cpu_to_le16(IEEE80211_FCTL_TODS):
iftype != NL80211_IFTYPE_STATION))
return -1;
if (iftype == NL80211_IFTYPE_MESH_POINT) {
- if (mesh_flags & MESH_FLAGS_AE_A4)
+ if (mesh_flags == MESH_FLAGS_AE_A4)
return -1;
- if (mesh_flags & MESH_FLAGS_AE_A5_A6) {
+ if (mesh_flags == MESH_FLAGS_AE_A5_A6) {
skb_copy_bits(skb, hdrlen +
offsetof(struct ieee80211s_hdr, eaddr1),
tmp.h_dest, 2 * ETH_ALEN);
ether_addr_equal(tmp.h_source, addr)))
return -1;
if (iftype == NL80211_IFTYPE_MESH_POINT) {
- if (mesh_flags & MESH_FLAGS_AE_A5_A6)
+ if (mesh_flags == MESH_FLAGS_AE_A5_A6)
return -1;
- if (mesh_flags & MESH_FLAGS_AE_A4)
+ if (mesh_flags == MESH_FLAGS_AE_A4)
skb_copy_bits(skb, hdrlen +
offsetof(struct ieee80211s_hdr, eaddr1),
tmp.h_source, ETH_ALEN);
static int __init x25_init(void)
{
- int rc = proto_register(&x25_proto, 0);
+ int rc;
- if (rc != 0)
+ rc = proto_register(&x25_proto, 0);
+ if (rc)
goto out;
rc = sock_register(&x25_family_ops);
- if (rc != 0)
+ if (rc)
goto out_proto;
dev_add_pack(&x25_packet_type);
rc = register_netdevice_notifier(&x25_dev_notifier);
- if (rc != 0)
+ if (rc)
goto out_sock;
- pr_info("Linux Version 0.2\n");
+ rc = x25_register_sysctl();
+ if (rc)
+ goto out_dev;
- x25_register_sysctl();
rc = x25_proc_init();
- if (rc != 0)
- goto out_dev;
+ if (rc)
+ goto out_sysctl;
+
+ pr_info("Linux Version 0.2\n");
+
out:
return rc;
+out_sysctl:
+ x25_unregister_sysctl();
out_dev:
unregister_netdevice_notifier(&x25_dev_notifier);
out_sock:
+ dev_remove_pack(&x25_packet_type);
sock_unregister(AF_X25);
out_proto:
proto_unregister(&x25_proto);
{ },
};
-void __init x25_register_sysctl(void)
+int __init x25_register_sysctl(void)
{
x25_table_header = register_net_sysctl(&init_net, "net/x25", x25_table);
+ if (!x25_table_header)
+ return -ENOMEM;
+ return 0;
}
void x25_unregister_sysctl(void)
static int xfrm_dev_down(struct net_device *dev)
{
- if (dev->hw_features & NETIF_F_HW_ESP)
+ if (dev->features & NETIF_F_HW_ESP)
xfrm_dev_state_flush(dev_net(dev), dev, true);
xfrm_garbage_collect(dev_net(dev));
goto out;
}
-#ifdef CONFIG_XFRM_SUB_POLICY
-static int xfrm_dst_alloc_copy(void **target, const void *src, int size)
-{
- if (!*target) {
- *target = kmalloc(size, GFP_ATOMIC);
- if (!*target)
- return -ENOMEM;
- }
-
- memcpy(*target, src, size);
- return 0;
-}
-#endif
-
-static int xfrm_dst_update_parent(struct dst_entry *dst,
- const struct xfrm_selector *sel)
-{
-#ifdef CONFIG_XFRM_SUB_POLICY
- struct xfrm_dst *xdst = (struct xfrm_dst *)dst;
- return xfrm_dst_alloc_copy((void **)&(xdst->partner),
- sel, sizeof(*sel));
-#else
- return 0;
-#endif
-}
-
-static int xfrm_dst_update_origin(struct dst_entry *dst,
- const struct flowi *fl)
-{
-#ifdef CONFIG_XFRM_SUB_POLICY
- struct xfrm_dst *xdst = (struct xfrm_dst *)dst;
- return xfrm_dst_alloc_copy((void **)&(xdst->origin), fl, sizeof(*fl));
-#else
- return 0;
-#endif
-}
-
static int xfrm_expand_policies(const struct flowi *fl, u16 family,
struct xfrm_policy **pols,
int *num_pols, int *num_xfrms)
xdst = (struct xfrm_dst *)dst;
xdst->num_xfrms = err;
- if (num_pols > 1)
- err = xfrm_dst_update_parent(dst, &pols[1]->selector);
- else
- err = xfrm_dst_update_origin(dst, fl);
- if (unlikely(err)) {
- dst_free(dst);
- XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTBUNDLECHECKERROR);
- return ERR_PTR(err);
- }
-
xdst->num_pols = num_pols;
memcpy(xdst->pols, pols, sizeof(struct xfrm_policy *) * num_pols);
xdst->policy_genid = atomic_read(&pols[0]->genid);
x->curlft.add_time = orig->curlft.add_time;
x->km.state = orig->km.state;
x->km.seq = orig->km.seq;
+ x->replay = orig->replay;
+ x->preplay = orig->preplay;
return x;
prog_attach_iptables(argv[2]);
if (cfg_test_traffic) {
if (signal(SIGINT, finish) == SIG_ERR)
- error(1, errno, "register handler failed");
+ error(1, errno, "register SIGINT handler failed");
+ if (signal(SIGTERM, finish) == SIG_ERR)
+ error(1, errno, "register SIGTERM handler failed");
while (!test_finish) {
print_table();
printf("\n");
setrlimit(RLIMIT_MEMLOCK, &r);
signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
if (load_kallsyms()) {
printf("failed to process /proc/kallsyms\n");
return 1;
}
signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
/* do sampling */
printf("Sampling at %d Hertz for %d seconds. Ctrl-C also ends.\n",
setrlimit(RLIMIT_MEMLOCK, &r);
signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
if (load_kallsyms()) {
printf("failed to process /proc/kallsyms\n");
}
signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
/* start 'ping' in the background to have some kfree_skb events */
f = popen("ping -c5 localhost", "r");
fprintf(stderr,
"usage: %s [OPTS] IFINDEX\n\n"
"OPTS:\n"
- " -S use skb-mode\n",
+ " -S use skb-mode\n"
+ " -N enforce native mode\n",
prog);
}
int main(int argc, char **argv)
{
- const char *optstr = "S";
+ const char *optstr = "SN";
char filename[256];
int opt;
case 'S':
xdp_flags |= XDP_FLAGS_SKB_MODE;
break;
+ case 'N':
+ xdp_flags |= XDP_FLAGS_DRV_MODE;
+ break;
default:
usage(basename(argv[0]));
return 1;
}
signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
if (set_link_xdp_fd(ifindex, prog_fd[0], xdp_flags) < 0) {
printf("link set xdp fd failed\n");
printf(" -m <dest-MAC> Used in sending the IP Tunneled pkt\n");
printf(" -T <stop-after-X-seconds> Default: 0 (forever)\n");
printf(" -P <IP-Protocol> Default is TCP\n");
+ printf(" -S use skb-mode\n");
+ printf(" -N enforce native mode\n");
printf(" -h Display this help\n");
}
{
unsigned char opt_flags[256] = {};
unsigned int kill_after_s = 0;
- const char *optstr = "i:a:p:s:d:m:T:P:Sh";
+ const char *optstr = "i:a:p:s:d:m:T:P:SNh";
int min_port = 0, max_port = 0;
struct iptnl_info tnl = {};
struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
case 'S':
xdp_flags |= XDP_FLAGS_SKB_MODE;
break;
+ case 'N':
+ xdp_flags |= XDP_FLAGS_DRV_MODE;
+ break;
default:
usage(argv[0]);
return 1;
}
signal(SIGINT, int_exit);
+ signal(SIGTERM, int_exit);
while (min_port <= max_port) {
vip.dport = htons(min_port++);
#
# ==========================================================================
+PHONY := __headers
+__headers:
+
+include scripts/Kbuild.include
+
+srcdir := $(srctree)/$(obj)
+subdirs := $(patsubst $(srcdir)/%/.,%,$(wildcard $(srcdir)/*/.))
+# caller may set destination dir (when installing to asm/)
+_dst := $(if $(dst),$(dst),$(obj))
+
+# Recursion
+__headers: $(subdirs)
+
+.PHONY: $(subdirs)
+$(subdirs):
+ $(Q)$(MAKE) $(hdr-inst)=$(obj)/$@ dst=$(_dst)/$@
+
+# Skip header install/check for include/uapi and arch/$(hdr-arch)/include/uapi.
+# We have only sub-directories there.
+skip-inst := $(if $(filter %/uapi,$(obj)),1)
+
+ifeq ($(skip-inst),)
+
# generated header directory
gen := $(if $(gen),$(gen),$(subst include/,include/generated/,$(obj)))
kbuild-file := $(srctree)/$(obj)/Kbuild
-include $(kbuild-file)
-# called may set destination dir (when installing to asm/)
-_dst := $(if $(dst),$(dst),$(obj))
-
old-kbuild-file := $(srctree)/$(subst uapi/,,$(obj))/Kbuild
ifneq ($(wildcard $(old-kbuild-file)),)
include $(old-kbuild-file)
endif
-include scripts/Kbuild.include
-
installdir := $(INSTALL_HDR_PATH)/$(subst uapi/,,$(_dst))
-srcdir := $(srctree)/$(obj)
gendir := $(objtree)/$(gen)
-subdirs := $(patsubst $(srcdir)/%/.,%,$(wildcard $(srcdir)/*/.))
header-files := $(notdir $(wildcard $(srcdir)/*.h))
header-files += $(notdir $(wildcard $(srcdir)/*.agh))
header-files := $(filter-out $(no-export-headers), $(header-files))
$(PERL) $< $(INSTALL_HDR_PATH)/include $(SRCARCH); \
touch $@
-PHONY += __headersinst __headerscheck
-
ifndef HDRCHECK
# Rules for installing headers
-__headersinst: $(subdirs) $(install-file)
+__headers: $(install-file)
@:
targets += $(install-file)
$(call if_changed,install)
else
-__headerscheck: $(subdirs) $(check-file)
+__headers: $(check-file)
@:
targets += $(check-file)
endif
-# Recursion
-.PHONY: $(subdirs)
-$(subdirs):
- $(Q)$(MAKE) $(hdr-inst)=$(obj)/$@ dst=$(_dst)/$@
-
targets := $(wildcard $(sort $(targets)))
cmd_files := $(wildcard \
$(foreach f,$(targets),$(dir $(f)).$(notdir $(f)).cmd))
include $(cmd_files)
endif
+endif # skip-inst
+
.PHONY: $(PHONY)
PHONY += FORCE
FORCE: ;
dtc_cpp_flags = -Wp,-MD,$(depfile).pre.tmp -nostdinc \
-I$(srctree)/arch/$(SRCARCH)/boot/dts \
- -I$(srctree)/arch/$(SRCARCH)/boot/dts/include \
+ -I$(srctree)/scripts/dtc/include-prefixes \
-I$(srctree)/drivers/of/testcase-data \
-undef -D__DTS__
while (size--)
reg = (reg << 32) | fdt32_to_cpu(*(cells++));
- snprintf(unit_addr, sizeof(unit_addr), "%lx", reg);
+ snprintf(unit_addr, sizeof(unit_addr), "%zx", reg);
if (!streq(unitname, unit_addr))
FAIL(c, dti, "Node %s simple-bus unit address format error, expected \"%s\"",
node->fullpath, unit_addr);
--- /dev/null
+../../../arch/arc/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/arm/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/arm64/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/c6x/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/cris/boot/dts
\ No newline at end of file
--- /dev/null
+../../../include/dt-bindings
\ No newline at end of file
--- /dev/null
+../../../arch/h8300/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/metag/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/microblaze/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/mips/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/nios2/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/openrisc/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/powerpc/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/sh/boot/dts
\ No newline at end of file
--- /dev/null
+../../../arch/xtensa/boot/dts
\ No newline at end of file
SND_PCI_QUIRK_VENDOR(0x1462, "MSI", ALC882_FIXUP_GPIO3),
SND_PCI_QUIRK(0x1458, 0xa002, "Gigabyte EP45-DS3/Z87X-UD3H", ALC889_FIXUP_FRONT_HP_NO_PRESENCE),
SND_PCI_QUIRK(0x1458, 0xa0b8, "Gigabyte AZ370-Gaming", ALC1220_FIXUP_GB_DUAL_CODECS),
+ SND_PCI_QUIRK(0x1462, 0xda57, "MSI Z270-Gaming", ALC1220_FIXUP_GB_DUAL_CODECS),
SND_PCI_QUIRK(0x147b, 0x107a, "Abit AW9D-MAX", ALC882_FIXUP_ABIT_AW9D_MAX),
SND_PCI_QUIRK_VENDOR(0x1558, "Clevo laptop", ALC882_FIXUP_EAPD),
SND_PCI_QUIRK(0x161f, 0x2054, "Medion laptop", ALC883_FIXUP_EAPD),
{.id = ALC883_FIXUP_ACER_EAPD, .name = "acer-aspire"},
{.id = ALC882_FIXUP_INV_DMIC, .name = "inv-dmic"},
{.id = ALC882_FIXUP_NO_PRIMARY_HP, .name = "no-primary-hp"},
+ {.id = ALC1220_FIXUP_GB_DUAL_CODECS, .name = "dual-codecs"},
{}
};
{.id = ALC292_FIXUP_TPT440_DOCK, .name = "tpt440-dock"},
{.id = ALC292_FIXUP_TPT440, .name = "tpt440"},
{.id = ALC292_FIXUP_TPT460, .name = "tpt460"},
+ {.id = ALC233_FIXUP_LENOVO_MULTI_CODECS, .name = "dual-codecs"},
{}
};
#define ALC225_STANDARD_PINS \
break;
case 0x10ec0225:
case 0x10ec0295:
+ spec->codec_variant = ALC269_TYPE_ALC225;
+ break;
case 0x10ec0299:
spec->codec_variant = ALC269_TYPE_ALC225;
+ spec->gen.mixer_nid = 0; /* no loopback on ALC299 */
break;
case 0x10ec0234:
case 0x10ec0274:
{.id = ALC662_FIXUP_ASUS_MODE8, .name = "asus-mode8"},
{.id = ALC662_FIXUP_INV_DMIC, .name = "inv-dmic"},
{.id = ALC668_FIXUP_DELL_MIC_NO_PRESENCE, .name = "dell-headset-multi"},
+ {.id = ALC662_FIXUP_LENOVO_MULTI_CODECS, .name = "dual-codecs"},
{}
};
"Dell Inspiron 1501", STAC_9200_DELL_M26),
SND_PCI_QUIRK(PCI_VENDOR_ID_DELL, 0x01f6,
"unknown Dell", STAC_9200_DELL_M26),
+ SND_PCI_QUIRK(PCI_VENDOR_ID_DELL, 0x0201,
+ "Dell Latitude D430", STAC_9200_DELL_M22),
/* Panasonic */
SND_PCI_QUIRK(0x10f7, 0x8338, "Panasonic CF-74", STAC_9200_PANASONIC),
/* Gateway machines needs EAPD to be set on resume */
/* Amanero Combo384 USB interface with native DSD support */
case USB_ID(0x16d0, 0x071a):
if (fp->altsetting == 2) {
- switch (chip->dev->descriptor.bcdDevice) {
+ switch (le16_to_cpu(chip->dev->descriptor.bcdDevice)) {
case 0x199:
return SNDRV_PCM_FMTBIT_DSD_U32_LE;
case 0x19b:
#include <drm/intel_lpe_audio.h>
#include "intel_hdmi_audio.h"
+#define for_each_pipe(card_ctx, pipe) \
+ for ((pipe) = 0; (pipe) < (card_ctx)->num_pipes; (pipe)++)
+#define for_each_port(card_ctx, port) \
+ for ((port) = 0; (port) < (card_ctx)->num_ports; (port)++)
+
/*standard module options for ALSA. This module supports only one card*/
static int hdmi_card_index = SNDRV_DEFAULT_IDX1;
static char *hdmi_card_id = SNDRV_DEFAULT_STR1;
spin_unlock_irqrestore(&intelhaddata->had_spinlock, flags);
}
+static u32 had_config_offset(int pipe)
+{
+ switch (pipe) {
+ default:
+ case 0:
+ return AUDIO_HDMI_CONFIG_A;
+ case 1:
+ return AUDIO_HDMI_CONFIG_B;
+ case 2:
+ return AUDIO_HDMI_CONFIG_C;
+ }
+}
+
/* Register access functions */
-static u32 had_read_register_raw(struct snd_intelhad *ctx, u32 reg)
+static u32 had_read_register_raw(struct snd_intelhad_card *card_ctx,
+ int pipe, u32 reg)
{
- return ioread32(ctx->mmio_start + ctx->had_config_offset + reg);
+ return ioread32(card_ctx->mmio_start + had_config_offset(pipe) + reg);
}
-static void had_write_register_raw(struct snd_intelhad *ctx, u32 reg, u32 val)
+static void had_write_register_raw(struct snd_intelhad_card *card_ctx,
+ int pipe, u32 reg, u32 val)
{
- iowrite32(val, ctx->mmio_start + ctx->had_config_offset + reg);
+ iowrite32(val, card_ctx->mmio_start + had_config_offset(pipe) + reg);
}
static void had_read_register(struct snd_intelhad *ctx, u32 reg, u32 *val)
if (!ctx->connected)
*val = 0;
else
- *val = had_read_register_raw(ctx, reg);
+ *val = had_read_register_raw(ctx->card_ctx, ctx->pipe, reg);
}
static void had_write_register(struct snd_intelhad *ctx, u32 reg, u32 val)
{
if (ctx->connected)
- had_write_register_raw(ctx, reg, val);
+ had_write_register_raw(ctx->card_ctx, ctx->pipe, reg, val);
}
/*
return;
}
+ /* Disable Audio */
+ had_enable_audio(intelhaddata, false);
+
intelhaddata->connected = true;
dev_dbg(intelhaddata->dev,
"%s @ %d:DEBUG PLUG/UNPLUG : HAD_DRV_CONNECTED\n",
*/
static irqreturn_t display_pipe_interrupt_handler(int irq, void *dev_id)
{
- struct snd_intelhad *ctx = dev_id;
- u32 audio_stat;
+ struct snd_intelhad_card *card_ctx = dev_id;
+ u32 audio_stat[3] = {};
+ int pipe, port;
+
+ for_each_pipe(card_ctx, pipe) {
+ /* use raw register access to ack IRQs even while disconnected */
+ audio_stat[pipe] = had_read_register_raw(card_ctx, pipe,
+ AUD_HDMI_STATUS) &
+ (HDMI_AUDIO_UNDERRUN | HDMI_AUDIO_BUFFER_DONE);
+
+ if (audio_stat[pipe])
+ had_write_register_raw(card_ctx, pipe,
+ AUD_HDMI_STATUS, audio_stat[pipe]);
+ }
- /* use raw register access to ack IRQs even while disconnected */
- audio_stat = had_read_register_raw(ctx, AUD_HDMI_STATUS);
+ for_each_port(card_ctx, port) {
+ struct snd_intelhad *ctx = &card_ctx->pcm_ctx[port];
+ int pipe = ctx->pipe;
- if (audio_stat & HDMI_AUDIO_UNDERRUN) {
- had_write_register_raw(ctx, AUD_HDMI_STATUS,
- HDMI_AUDIO_UNDERRUN);
- had_process_buffer_underrun(ctx);
- }
+ if (pipe < 0)
+ continue;
- if (audio_stat & HDMI_AUDIO_BUFFER_DONE) {
- had_write_register_raw(ctx, AUD_HDMI_STATUS,
- HDMI_AUDIO_BUFFER_DONE);
- had_process_buffer_done(ctx);
+ if (audio_stat[pipe] & HDMI_AUDIO_BUFFER_DONE)
+ had_process_buffer_done(ctx);
+ if (audio_stat[pipe] & HDMI_AUDIO_UNDERRUN)
+ had_process_buffer_underrun(ctx);
}
return IRQ_HANDLED;
/*
* monitor plug/unplug notification from i915; just kick off the work
*/
-static void notify_audio_lpe(struct platform_device *pdev)
+static void notify_audio_lpe(struct platform_device *pdev, int port)
{
- struct snd_intelhad *ctx = platform_get_drvdata(pdev);
+ struct snd_intelhad_card *card_ctx = platform_get_drvdata(pdev);
+ struct snd_intelhad *ctx = &card_ctx->pcm_ctx[port];
schedule_work(&ctx->hdmi_audio_wq);
}
struct snd_intelhad *ctx =
container_of(work, struct snd_intelhad, hdmi_audio_wq);
struct intel_hdmi_lpe_audio_pdata *pdata = ctx->dev->platform_data;
+ struct intel_hdmi_lpe_audio_port_pdata *ppdata = &pdata->port[ctx->port];
pm_runtime_get_sync(ctx->dev);
mutex_lock(&ctx->mutex);
- if (!pdata->hdmi_connected) {
- dev_dbg(ctx->dev, "%s: Event: HAD_NOTIFY_HOT_UNPLUG\n",
- __func__);
+ if (ppdata->pipe < 0) {
+ dev_dbg(ctx->dev, "%s: Event: HAD_NOTIFY_HOT_UNPLUG : port = %d\n",
+ __func__, ctx->port);
+
memset(ctx->eld, 0, sizeof(ctx->eld)); /* clear the old ELD */
+
+ ctx->dp_output = false;
+ ctx->tmds_clock_speed = 0;
+ ctx->link_rate = 0;
+
+ /* Shut down the stream */
had_process_hot_unplug(ctx);
- } else {
- struct intel_hdmi_lpe_audio_eld *eld = &pdata->eld;
+ ctx->pipe = -1;
+ } else {
dev_dbg(ctx->dev, "%s: HAD_NOTIFY_ELD : port = %d, tmds = %d\n",
- __func__, eld->port_id, pdata->tmds_clock_speed);
+ __func__, ctx->port, ppdata->ls_clock);
- switch (eld->pipe_id) {
- case 0:
- ctx->had_config_offset = AUDIO_HDMI_CONFIG_A;
- break;
- case 1:
- ctx->had_config_offset = AUDIO_HDMI_CONFIG_B;
- break;
- case 2:
- ctx->had_config_offset = AUDIO_HDMI_CONFIG_C;
- break;
- default:
- dev_dbg(ctx->dev, "Invalid pipe %d\n",
- eld->pipe_id);
- break;
- }
-
- memcpy(ctx->eld, eld->eld_data, sizeof(ctx->eld));
+ memcpy(ctx->eld, ppdata->eld, sizeof(ctx->eld));
- ctx->dp_output = pdata->dp_output;
- ctx->tmds_clock_speed = pdata->tmds_clock_speed;
- ctx->link_rate = pdata->link_rate;
+ ctx->dp_output = ppdata->dp_output;
+ if (ctx->dp_output) {
+ ctx->tmds_clock_speed = 0;
+ ctx->link_rate = ppdata->ls_clock;
+ } else {
+ ctx->tmds_clock_speed = ppdata->ls_clock;
+ ctx->link_rate = 0;
+ }
+ /*
+ * Shut down the stream before we change
+ * the pipe assignment for this pcm device
+ */
had_process_hot_plug(ctx);
- /* Process mode change if stream is active */
+ ctx->pipe = ppdata->pipe;
+
+ /* Restart the stream if necessary */
had_process_mode_change(ctx);
}
+
mutex_unlock(&ctx->mutex);
pm_runtime_mark_last_busy(ctx->dev);
pm_runtime_put_autosuspend(ctx->dev);
/*
* Jack interface
*/
-static int had_create_jack(struct snd_intelhad *ctx)
+static int had_create_jack(struct snd_intelhad *ctx,
+ struct snd_pcm *pcm)
{
+ char hdmi_str[32];
int err;
- err = snd_jack_new(ctx->card, "HDMI/DP", SND_JACK_AVOUT, &ctx->jack,
+ snprintf(hdmi_str, sizeof(hdmi_str),
+ "HDMI/DP,pcm=%d", pcm->device);
+
+ err = snd_jack_new(ctx->card_ctx->card, hdmi_str,
+ SND_JACK_AVOUT, &ctx->jack,
true, false);
if (err < 0)
return err;
static int hdmi_lpe_audio_runtime_suspend(struct device *dev)
{
- struct snd_intelhad *ctx = dev_get_drvdata(dev);
- struct snd_pcm_substream *substream;
+ struct snd_intelhad_card *card_ctx = dev_get_drvdata(dev);
+ int port;
- substream = had_substream_get(ctx);
- if (substream) {
- snd_pcm_suspend(substream);
- had_substream_put(ctx);
+ for_each_port(card_ctx, port) {
+ struct snd_intelhad *ctx = &card_ctx->pcm_ctx[port];
+ struct snd_pcm_substream *substream;
+
+ substream = had_substream_get(ctx);
+ if (substream) {
+ snd_pcm_suspend(substream);
+ had_substream_put(ctx);
+ }
}
return 0;
static int __maybe_unused hdmi_lpe_audio_suspend(struct device *dev)
{
- struct snd_intelhad *ctx = dev_get_drvdata(dev);
+ struct snd_intelhad_card *card_ctx = dev_get_drvdata(dev);
int err;
err = hdmi_lpe_audio_runtime_suspend(dev);
if (!err)
- snd_power_change_state(ctx->card, SNDRV_CTL_POWER_D3hot);
+ snd_power_change_state(card_ctx->card, SNDRV_CTL_POWER_D3hot);
return err;
}
static int __maybe_unused hdmi_lpe_audio_resume(struct device *dev)
{
- struct snd_intelhad *ctx = dev_get_drvdata(dev);
+ struct snd_intelhad_card *card_ctx = dev_get_drvdata(dev);
hdmi_lpe_audio_runtime_resume(dev);
- snd_power_change_state(ctx->card, SNDRV_CTL_POWER_D0);
+ snd_power_change_state(card_ctx->card, SNDRV_CTL_POWER_D0);
return 0;
}
/* release resources */
static void hdmi_lpe_audio_free(struct snd_card *card)
{
- struct snd_intelhad *ctx = card->private_data;
+ struct snd_intelhad_card *card_ctx = card->private_data;
+ struct intel_hdmi_lpe_audio_pdata *pdata = card_ctx->dev->platform_data;
+ int port;
- cancel_work_sync(&ctx->hdmi_audio_wq);
+ spin_lock_irq(&pdata->lpe_audio_slock);
+ pdata->notify_audio_lpe = NULL;
+ spin_unlock_irq(&pdata->lpe_audio_slock);
- if (ctx->mmio_start)
- iounmap(ctx->mmio_start);
- if (ctx->irq >= 0)
- free_irq(ctx->irq, ctx);
+ for_each_port(card_ctx, port) {
+ struct snd_intelhad *ctx = &card_ctx->pcm_ctx[port];
+
+ cancel_work_sync(&ctx->hdmi_audio_wq);
+ }
+
+ if (card_ctx->mmio_start)
+ iounmap(card_ctx->mmio_start);
+ if (card_ctx->irq >= 0)
+ free_irq(card_ctx->irq, card_ctx);
}
/*
static int hdmi_lpe_audio_probe(struct platform_device *pdev)
{
struct snd_card *card;
- struct snd_intelhad *ctx;
+ struct snd_intelhad_card *card_ctx;
struct snd_pcm *pcm;
struct intel_hdmi_lpe_audio_pdata *pdata;
int irq;
struct resource *res_mmio;
- int i, ret;
+ int port, ret;
pdata = pdev->dev.platform_data;
if (!pdata) {
/* create a card instance with ALSA framework */
ret = snd_card_new(&pdev->dev, hdmi_card_index, hdmi_card_id,
- THIS_MODULE, sizeof(*ctx), &card);
+ THIS_MODULE, sizeof(*card_ctx), &card);
if (ret)
return ret;
- ctx = card->private_data;
- spin_lock_init(&ctx->had_spinlock);
- mutex_init(&ctx->mutex);
- ctx->connected = false;
- ctx->dev = &pdev->dev;
- ctx->card = card;
- ctx->aes_bits = SNDRV_PCM_DEFAULT_CON_SPDIF;
+ card_ctx = card->private_data;
+ card_ctx->dev = &pdev->dev;
+ card_ctx->card = card;
strcpy(card->driver, INTEL_HAD);
strcpy(card->shortname, "Intel HDMI/DP LPE Audio");
strcpy(card->longname, "Intel HDMI/DP LPE Audio");
- ctx->irq = -1;
- ctx->tmds_clock_speed = DIS_SAMPLE_RATE_148_5;
- INIT_WORK(&ctx->hdmi_audio_wq, had_audio_wq);
+ card_ctx->irq = -1;
card->private_free = hdmi_lpe_audio_free;
- /* assume pipe A as default */
- ctx->had_config_offset = AUDIO_HDMI_CONFIG_A;
-
- platform_set_drvdata(pdev, ctx);
+ platform_set_drvdata(pdev, card_ctx);
dev_dbg(&pdev->dev, "%s: mmio_start = 0x%x, mmio_end = 0x%x\n",
__func__, (unsigned int)res_mmio->start,
(unsigned int)res_mmio->end);
- ctx->mmio_start = ioremap_nocache(res_mmio->start,
- (size_t)(resource_size(res_mmio)));
- if (!ctx->mmio_start) {
+ card_ctx->mmio_start = ioremap_nocache(res_mmio->start,
+ (size_t)(resource_size(res_mmio)));
+ if (!card_ctx->mmio_start) {
dev_err(&pdev->dev, "Could not get ioremap\n");
ret = -EACCES;
goto err;
/* setup interrupt handler */
ret = request_irq(irq, display_pipe_interrupt_handler, 0,
- pdev->name, ctx);
+ pdev->name, card_ctx);
if (ret < 0) {
dev_err(&pdev->dev, "request_irq failed\n");
goto err;
}
- ctx->irq = irq;
-
- ret = snd_pcm_new(card, INTEL_HAD, PCM_INDEX, MAX_PB_STREAMS,
- MAX_CAP_STREAMS, &pcm);
- if (ret)
- goto err;
-
- /* setup private data which can be retrieved when required */
- pcm->private_data = ctx;
- pcm->info_flags = 0;
- strncpy(pcm->name, card->shortname, strlen(card->shortname));
- /* setup the ops for playabck */
- snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &had_pcm_ops);
+ card_ctx->irq = irq;
/* only 32bit addressable */
dma_set_mask(&pdev->dev, DMA_BIT_MASK(32));
dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32));
- /* allocate dma pages;
- * try to allocate 600k buffer as default which is large enough
- */
- snd_pcm_lib_preallocate_pages_for_all(pcm,
- SNDRV_DMA_TYPE_DEV, NULL,
- HAD_DEFAULT_BUFFER, HAD_MAX_BUFFER);
+ init_channel_allocations();
- /* create controls */
- for (i = 0; i < ARRAY_SIZE(had_controls); i++) {
- ret = snd_ctl_add(card, snd_ctl_new1(&had_controls[i], ctx));
- if (ret < 0)
+ card_ctx->num_pipes = pdata->num_pipes;
+ card_ctx->num_ports = pdata->num_ports;
+
+ for_each_port(card_ctx, port) {
+ struct snd_intelhad *ctx = &card_ctx->pcm_ctx[port];
+ int i;
+
+ ctx->card_ctx = card_ctx;
+ ctx->dev = card_ctx->dev;
+ ctx->port = port;
+ ctx->pipe = -1;
+
+ INIT_WORK(&ctx->hdmi_audio_wq, had_audio_wq);
+
+ ret = snd_pcm_new(card, INTEL_HAD, port, MAX_PB_STREAMS,
+ MAX_CAP_STREAMS, &pcm);
+ if (ret)
goto err;
- }
- init_channel_allocations();
+ /* setup private data which can be retrieved when required */
+ pcm->private_data = ctx;
+ pcm->info_flags = 0;
+ strncpy(pcm->name, card->shortname, strlen(card->shortname));
+ /* setup the ops for playabck */
+ snd_pcm_set_ops(pcm, SNDRV_PCM_STREAM_PLAYBACK, &had_pcm_ops);
- /* Register channel map controls */
- ret = had_register_chmap_ctls(ctx, pcm);
- if (ret < 0)
- goto err;
+ /* allocate dma pages;
+ * try to allocate 600k buffer as default which is large enough
+ */
+ snd_pcm_lib_preallocate_pages_for_all(pcm,
+ SNDRV_DMA_TYPE_DEV, NULL,
+ HAD_DEFAULT_BUFFER, HAD_MAX_BUFFER);
+
+ /* create controls */
+ for (i = 0; i < ARRAY_SIZE(had_controls); i++) {
+ struct snd_kcontrol *kctl;
+
+ kctl = snd_ctl_new1(&had_controls[i], ctx);
+ if (!kctl) {
+ ret = -ENOMEM;
+ goto err;
+ }
- ret = had_create_jack(ctx);
- if (ret < 0)
- goto err;
+ kctl->id.device = pcm->device;
+
+ ret = snd_ctl_add(card, kctl);
+ if (ret < 0)
+ goto err;
+ }
+
+ /* Register channel map controls */
+ ret = had_register_chmap_ctls(ctx, pcm);
+ if (ret < 0)
+ goto err;
+
+ ret = had_create_jack(ctx, pcm);
+ if (ret < 0)
+ goto err;
+ }
ret = snd_card_register(card);
if (ret)
spin_lock_irq(&pdata->lpe_audio_slock);
pdata->notify_audio_lpe = notify_audio_lpe;
- pdata->notify_pending = false;
spin_unlock_irq(&pdata->lpe_audio_slock);
- /* runtime PM isn't enabled as default, since it won't save much on
- * BYT/CHT devices; user who want the runtime PM should adjust the
- * power/ontrol and power/autosuspend_delay_ms sysfs entries instead
- */
pm_runtime_use_autosuspend(&pdev->dev);
pm_runtime_mark_last_busy(&pdev->dev);
pm_runtime_set_active(&pdev->dev);
dev_dbg(&pdev->dev, "%s: handle pending notification\n", __func__);
- schedule_work(&ctx->hdmi_audio_wq);
+ for_each_port(card_ctx, port) {
+ struct snd_intelhad *ctx = &card_ctx->pcm_ctx[port];
+
+ schedule_work(&ctx->hdmi_audio_wq);
+ }
return 0;
*/
static int hdmi_lpe_audio_remove(struct platform_device *pdev)
{
- struct snd_intelhad *ctx = platform_get_drvdata(pdev);
+ struct snd_intelhad_card *card_ctx = platform_get_drvdata(pdev);
- snd_card_free(ctx->card);
+ snd_card_free(card_ctx->card);
return 0;
}
#include "intel_hdmi_lpe_audio.h"
-#define PCM_INDEX 0
#define MAX_PB_STREAMS 1
#define MAX_CAP_STREAMS 0
#define BYTES_PER_WORD 0x4
* @chmap: holds channel map info
*/
struct snd_intelhad {
- struct snd_card *card;
+ struct snd_intelhad_card *card_ctx;
bool connected;
struct pcm_stream_info stream_info;
unsigned char eld[HDMI_MAX_ELD_BYTES];
struct snd_pcm_chmap *chmap;
int tmds_clock_speed;
int link_rate;
+ int port; /* fixed */
+ int pipe; /* can change dynamically */
/* ring buffer (BD) position index */
unsigned int bd_head;
unsigned int period_bytes; /* PCM period size in bytes */
/* internal stuff */
- int irq;
- void __iomem *mmio_start;
- unsigned int had_config_offset;
union aud_cfg aud_config; /* AUD_CONFIG reg value cache */
struct work_struct hdmi_audio_wq;
struct mutex mutex; /* for protecting chmap and eld */
struct snd_jack *jack;
};
+struct snd_intelhad_card {
+ struct snd_card *card;
+ struct device *dev;
+
+ /* internal stuff */
+ int irq;
+ void __iomem *mmio_start;
+ int num_pipes;
+ int num_ports;
+ struct snd_intelhad pcm_ctx[3]; /* one for each port */
+};
+
#endif /* _INTEL_HDMI_AUDIO_ */
#define __KVM_HAVE_IRQ_LINE
#define __KVM_HAVE_READONLY_MEM
+#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
+
#define KVM_REG_SIZE(id) \
(1U << (((id) & KVM_REG_SIZE_MASK) >> KVM_REG_SIZE_SHIFT))
};
struct kvm_sync_regs {
+ /* Used with KVM_CAP_ARM_USER_IRQ */
+ __u64 device_irq_level;
};
struct kvm_arch_memory_slot {
#define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7
+#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
#define VGIC_LEVEL_INFO_LINE_LEVEL 0
-#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
+#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
+#define KVM_DEV_ARM_ITS_SAVE_TABLES 1
+#define KVM_DEV_ARM_ITS_RESTORE_TABLES 2
+#define KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES 3
/* KVM_IRQ_LINE irq field index values */
#define KVM_ARM_IRQ_TYPE_SHIFT 24
#define __KVM_HAVE_IRQ_LINE
#define __KVM_HAVE_READONLY_MEM
+#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
+
#define KVM_REG_SIZE(id) \
(1U << (((id) & KVM_REG_SIZE_MASK) >> KVM_REG_SIZE_SHIFT))
#define KVM_GUESTDBG_USE_HW (1 << 17)
struct kvm_sync_regs {
+ /* Used with KVM_CAP_ARM_USER_IRQ */
+ __u64 device_irq_level;
};
struct kvm_arch_memory_slot {
#define KVM_DEV_ARM_VGIC_GRP_REDIST_REGS 5
#define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6
#define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7
+#define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \
(0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT)
#define KVM_DEV_ARM_VGIC_LINE_LEVEL_INTID_MASK 0x3ff
#define VGIC_LEVEL_INFO_LINE_LEVEL 0
-#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
+#define KVM_DEV_ARM_VGIC_CTRL_INIT 0
+#define KVM_DEV_ARM_ITS_SAVE_TABLES 1
+#define KVM_DEV_ARM_ITS_RESTORE_TABLES 2
+#define KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES 3
/* Device Control API on vcpu fd */
#define KVM_ARM_VCPU_PMU_V3_CTRL 0
#define __KVM_HAVE_IRQ_LINE
#define __KVM_HAVE_GUEST_DEBUG
+/* Not always available, but if it is, this is the correct offset. */
+#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
+
struct kvm_regs {
__u64 pc;
__u64 cr;
#define KVM_DEV_FLIC_ADAPTER_REGISTER 6
#define KVM_DEV_FLIC_ADAPTER_MODIFY 7
#define KVM_DEV_FLIC_CLEAR_IO_IRQ 8
+#define KVM_DEV_FLIC_AISM 9
+#define KVM_DEV_FLIC_AIRQ_INJECT 10
/*
* We can have up to 4*64k pending subchannels + 8 adapter interrupts,
* as well as up to ASYNC_PF_PER_VCPU*KVM_MAX_VCPUS pfault done interrupts.
__u8 isc;
__u8 maskable;
__u8 swap;
- __u8 pad;
+ __u8 flags;
+};
+
+#define KVM_S390_ADAPTER_SUPPRESSIBLE 0x01
+
+struct kvm_s390_ais_req {
+ __u8 isc;
+ __u16 mode;
};
#define KVM_S390_IO_ADAPTER_MASK 1
#define KVM_S390_VM_CPU_FEAT_CMMA 10
#define KVM_S390_VM_CPU_FEAT_PFMFI 11
#define KVM_S390_VM_CPU_FEAT_SIGPIF 12
+#define KVM_S390_VM_CPU_FEAT_KSS 13
struct kvm_s390_vm_cpu_feat {
__u64 feat[16];
};
#define KVM_SYNC_VRS (1UL << 6)
#define KVM_SYNC_RICCB (1UL << 7)
#define KVM_SYNC_FPRS (1UL << 8)
+#define KVM_SYNC_GSCB (1UL << 9)
+/* length and alignment of the sdnx as a power of two */
+#define SDNXC 8
+#define SDNXL (1UL << SDNXC)
/* definition of registers in kvm_run */
struct kvm_sync_regs {
__u64 prefix; /* prefix register */
};
__u8 reserved[512]; /* for future vector expansion */
__u32 fpc; /* valid on KVM_SYNC_VRS or KVM_SYNC_FPRS */
- __u8 padding[52]; /* riccb needs to be 64byte aligned */
+ __u8 padding1[52]; /* riccb needs to be 64byte aligned */
__u8 riccb[64]; /* runtime instrumentation controls block */
+ __u8 padding2[192]; /* sdnx needs to be 256byte aligned */
+ union {
+ __u8 sdnx[SDNXL]; /* state description annex */
+ struct {
+ __u64 reserved1[2];
+ __u64 gscb[4];
+ };
+ };
};
#define KVM_REG_S390_TODPR (KVM_REG_S390 | KVM_REG_SIZE_U32 | 0x1)
#define X86_FEATURE_AVX512_4VNNIW (7*32+16) /* AVX-512 Neural Network Instructions */
#define X86_FEATURE_AVX512_4FMAPS (7*32+17) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_MBA ( 7*32+18) /* Memory Bandwidth Allocation */
+
/* Virtualization flags: Linux defined, word 8 */
#define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */
#define X86_FEATURE_VNMI ( 8*32+ 1) /* Intel Virtual NMI */
# define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31))
#endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */
+#ifdef CONFIG_X86_5LEVEL
+# define DISABLE_LA57 0
+#else
+# define DISABLE_LA57 (1<<(X86_FEATURE_LA57 & 31))
+#endif
+
/*
* Make sure to add features to the correct mask
*/
#define DISABLED_MASK13 0
#define DISABLED_MASK14 0
#define DISABLED_MASK15 0
-#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE)
+#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57)
#define DISABLED_MASK17 0
#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
# define NEED_MOVBE 0
#endif
+#ifdef CONFIG_X86_5LEVEL
+# define NEED_LA57 (1<<(X86_FEATURE_LA57 & 31))
+#else
+# define NEED_LA57 0
+#endif
+
#ifdef CONFIG_X86_64
#ifdef CONFIG_PARAVIRT
/* Paravirtualized systems may not have PSE or PGE available */
#define REQUIRED_MASK13 0
#define REQUIRED_MASK14 0
#define REQUIRED_MASK15 0
-#define REQUIRED_MASK16 0
+#define REQUIRED_MASK16 (NEED_LA57)
#define REQUIRED_MASK17 0
#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
#include <linux/types.h>
#include <linux/ioctl.h>
+#define KVM_PIO_PAGE_OFFSET 1
+#define KVM_COALESCED_MMIO_PAGE_OFFSET 2
+
#define DE_VECTOR 0
#define DB_VECTOR 1
#define BP_VECTOR 3
#define EXIT_REASON_WBINVD 54
#define EXIT_REASON_XSETBV 55
#define EXIT_REASON_APIC_WRITE 56
+#define EXIT_REASON_RDRAND 57
#define EXIT_REASON_INVPCID 58
+#define EXIT_REASON_VMFUNC 59
+#define EXIT_REASON_ENCLS 60
+#define EXIT_REASON_RDSEED 61
#define EXIT_REASON_PML_FULL 62
#define EXIT_REASON_XSAVES 63
#define EXIT_REASON_XRSTORS 64
{ EXIT_REASON_TASK_SWITCH, "TASK_SWITCH" }, \
{ EXIT_REASON_CPUID, "CPUID" }, \
{ EXIT_REASON_HLT, "HLT" }, \
+ { EXIT_REASON_INVD, "INVD" }, \
{ EXIT_REASON_INVLPG, "INVLPG" }, \
{ EXIT_REASON_RDPMC, "RDPMC" }, \
{ EXIT_REASON_RDTSC, "RDTSC" }, \
{ EXIT_REASON_IO_INSTRUCTION, "IO_INSTRUCTION" }, \
{ EXIT_REASON_MSR_READ, "MSR_READ" }, \
{ EXIT_REASON_MSR_WRITE, "MSR_WRITE" }, \
+ { EXIT_REASON_INVALID_STATE, "INVALID_STATE" }, \
+ { EXIT_REASON_MSR_LOAD_FAIL, "MSR_LOAD_FAIL" }, \
{ EXIT_REASON_MWAIT_INSTRUCTION, "MWAIT_INSTRUCTION" }, \
{ EXIT_REASON_MONITOR_TRAP_FLAG, "MONITOR_TRAP_FLAG" }, \
{ EXIT_REASON_MONITOR_INSTRUCTION, "MONITOR_INSTRUCTION" }, \
{ EXIT_REASON_MCE_DURING_VMENTRY, "MCE_DURING_VMENTRY" }, \
{ EXIT_REASON_TPR_BELOW_THRESHOLD, "TPR_BELOW_THRESHOLD" }, \
{ EXIT_REASON_APIC_ACCESS, "APIC_ACCESS" }, \
- { EXIT_REASON_GDTR_IDTR, "GDTR_IDTR" }, \
- { EXIT_REASON_LDTR_TR, "LDTR_TR" }, \
+ { EXIT_REASON_EOI_INDUCED, "EOI_INDUCED" }, \
+ { EXIT_REASON_GDTR_IDTR, "GDTR_IDTR" }, \
+ { EXIT_REASON_LDTR_TR, "LDTR_TR" }, \
{ EXIT_REASON_EPT_VIOLATION, "EPT_VIOLATION" }, \
{ EXIT_REASON_EPT_MISCONFIG, "EPT_MISCONFIG" }, \
{ EXIT_REASON_INVEPT, "INVEPT" }, \
+ { EXIT_REASON_RDTSCP, "RDTSCP" }, \
{ EXIT_REASON_PREEMPTION_TIMER, "PREEMPTION_TIMER" }, \
+ { EXIT_REASON_INVVPID, "INVVPID" }, \
{ EXIT_REASON_WBINVD, "WBINVD" }, \
+ { EXIT_REASON_XSETBV, "XSETBV" }, \
{ EXIT_REASON_APIC_WRITE, "APIC_WRITE" }, \
- { EXIT_REASON_EOI_INDUCED, "EOI_INDUCED" }, \
- { EXIT_REASON_INVALID_STATE, "INVALID_STATE" }, \
- { EXIT_REASON_MSR_LOAD_FAIL, "MSR_LOAD_FAIL" }, \
- { EXIT_REASON_INVD, "INVD" }, \
- { EXIT_REASON_INVVPID, "INVVPID" }, \
+ { EXIT_REASON_RDRAND, "RDRAND" }, \
{ EXIT_REASON_INVPCID, "INVPCID" }, \
+ { EXIT_REASON_VMFUNC, "VMFUNC" }, \
+ { EXIT_REASON_ENCLS, "ENCLS" }, \
+ { EXIT_REASON_RDSEED, "RDSEED" }, \
+ { EXIT_REASON_PML_FULL, "PML_FULL" }, \
{ EXIT_REASON_XSAVES, "XSAVES" }, \
{ EXIT_REASON_XRSTORS, "XRSTORS" }
attr.log_size = 0;
attr.log_level = 0;
attr.kern_version = 0;
+ attr.prog_flags = 0;
/*
* Test existence of __NR_bpf and BPF_PROG_LOAD.
.off = OFF, \
.imm = IMM })
+/* Unconditional jumps, goto pc + off16 */
+
+#define BPF_JMP_A(OFF) \
+ ((struct bpf_insn) { \
+ .code = BPF_JMP | BPF_JA, \
+ .dst_reg = 0, \
+ .src_reg = 0, \
+ .off = OFF, \
+ .imm = 0 })
+
/* Function call */
#define BPF_EMIT_CALL(FUNC) \
*/
#define BPF_F_ALLOW_OVERRIDE (1U << 0)
+/* If BPF_F_STRICT_ALIGNMENT is used in BPF_PROG_LOAD command, the
+ * verifier will perform strict alignment checking as if the kernel
+ * has been built with CONFIG_EFFICIENT_UNALIGNED_ACCESS not set,
+ * and NET_IP_ALIGN defined to 2.
+ */
+#define BPF_F_STRICT_ALIGNMENT (1U << 0)
+
#define BPF_PSEUDO_MAP_FD 1
/* flags for BPF_MAP_UPDATE_ELEM command */
__u32 log_size; /* size of user buffer */
__aligned_u64 log_buf; /* user supplied buffer */
__u32 kern_version; /* checked when prog_type=kprobe */
+ __u32 prog_flags;
};
struct { /* anonymous struct used by BPF_OBJ_* commands */
* u32 bpf_get_socket_uid(skb)
* Get the owner uid of the socket stored inside sk_buff.
* @skb: pointer to skb
- * Return: uid of the socket owner on success or 0 if the socket pointer
- * inside sk_buff is NULL
+ * Return: uid of the socket owner on success or overflowuid if failed.
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
* tv_sec holds the number of seconds before (negative) or after (positive)
* 00:00:00 1st January 1970 UTC.
*
- * tv_nsec holds a number of nanoseconds before (0..-999,999,999 if tv_sec is
- * negative) or after (0..999,999,999 if tv_sec is positive) the tv_sec time.
- *
- * Note that if both tv_sec and tv_nsec are non-zero, then the two values must
- * either be both positive or both negative.
+ * tv_nsec holds a number of nanoseconds (0..999,999,999) after the tv_sec time.
*
* __reserved is held in case we need a yet finer resolution.
*/
struct statx_timestamp {
__s64 tv_sec;
- __s32 tv_nsec;
+ __u32 tv_nsec;
__s32 __reserved;
};
return sys_bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
}
+int bpf_verify_program(enum bpf_prog_type type, const struct bpf_insn *insns,
+ size_t insns_cnt, int strict_alignment,
+ const char *license, __u32 kern_version,
+ char *log_buf, size_t log_buf_sz)
+{
+ union bpf_attr attr;
+
+ bzero(&attr, sizeof(attr));
+ attr.prog_type = type;
+ attr.insn_cnt = (__u32)insns_cnt;
+ attr.insns = ptr_to_u64(insns);
+ attr.license = ptr_to_u64(license);
+ attr.log_buf = ptr_to_u64(log_buf);
+ attr.log_size = log_buf_sz;
+ attr.log_level = 2;
+ log_buf[0] = 0;
+ attr.kern_version = kern_version;
+ attr.prog_flags = strict_alignment ? BPF_F_STRICT_ALIGNMENT : 0;
+
+ return sys_bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
+}
+
int bpf_map_update_elem(int fd, const void *key, const void *value,
__u64 flags)
{
size_t insns_cnt, const char *license,
__u32 kern_version, char *log_buf,
size_t log_buf_sz);
+int bpf_verify_program(enum bpf_prog_type type, const struct bpf_insn *insns,
+ size_t insns_cnt, int strict_alignment,
+ const char *license, __u32 kern_version,
+ char *log_buf, size_t log_buf_sz);
int bpf_map_update_elem(int fd, const void *key, const void *value,
__u64 flags);
Set the maximum number of program blocks to print with brstackasm for
each sample.
+--inline::
+ If a callgraph address belongs to an inlined function, the inline stack
+ will be printed. Each entry has function name and file/line.
+
SEE ALSO
--------
linkperf:perf-record[1], linkperf:perf-script-perl[1],
"Enable kernel symbol demangling"),
OPT_STRING(0, "time", &script.time_str, "str",
"Time span of interest (start,stop)"),
+ OPT_BOOLEAN(0, "inline", &symbol_conf.inline_name,
+ "Show inline function"),
OPT_END()
};
const char * const script_subcommands[] = { "record", "report", NULL };
return 0;
ret = b->callchain->max_depth - a->callchain->max_depth;
+ if (callchain_param.order == ORDER_CALLER)
+ ret = -ret;
}
return ret;
}
static enum match_result match_chain_srcline(struct callchain_cursor_node *node,
struct callchain_list *cnode)
{
- char *left = get_srcline(cnode->ms.map->dso,
+ char *left = NULL;
+ char *right = NULL;
+ enum match_result ret = MATCH_EQ;
+ int cmp;
+
+ if (cnode->ms.map)
+ left = get_srcline(cnode->ms.map->dso,
map__rip_2objdump(cnode->ms.map, cnode->ip),
cnode->ms.sym, true, false);
- char *right = get_srcline(node->map->dso,
+ if (node->map)
+ right = get_srcline(node->map->dso,
map__rip_2objdump(node->map, node->ip),
node->sym, true, false);
- enum match_result ret = MATCH_EQ;
- int cmp;
if (left && right)
cmp = strcmp(left, right);
#include "map.h"
#include "strlist.h"
#include "symbol.h"
+#include "srcline.h"
static int comma_fprintf(FILE *fp, bool *first, const char *fmt, ...)
{
if (!print_oneline)
printed += fprintf(fp, "\n");
+ if (symbol_conf.inline_name && node->map) {
+ struct inline_node *inode;
+
+ addr = map__rip_2objdump(node->map, node->ip),
+ inode = dso__parse_addr_inlines(node->map->dso, addr);
+
+ if (inode) {
+ struct inline_list *ilist;
+
+ list_for_each_entry(ilist, &inode->val, list) {
+ if (print_arrow)
+ printed += fprintf(fp, " <-");
+
+ /* IP is same, just skip it */
+ if (print_ip)
+ printed += fprintf(fp, "%c%16s",
+ s, "");
+ if (print_sym)
+ printed += fprintf(fp, " %s",
+ ilist->funcname);
+ if (print_srcline)
+ printed += fprintf(fp, "\n %s:%d",
+ ilist->filename,
+ ilist->line_nr);
+ if (!print_oneline)
+ printed += fprintf(fp, "\n");
+ }
+
+ inline_node__delete(inode);
+ }
+ }
+
if (symbol_conf.bt_stop_list &&
node->sym &&
strlist__has_entry(symbol_conf.bt_stop_list,
}
}
- list_add_tail(&ilist->list, &node->val);
+ if (callchain_param.order == ORDER_CALLEE)
+ list_add_tail(&ilist->list, &node->val);
+ else
+ list_add(&ilist->list, &node->val);
return 0;
}
#define MAX_INLINE_NEST 1024
-static void inline_list__reverse(struct inline_node *node)
+static int inline_list__append_dso_a2l(struct dso *dso,
+ struct inline_node *node)
{
- struct inline_list *ilist, *n;
+ struct a2l_data *a2l = dso->a2l;
+ char *funcname = a2l->funcname ? strdup(a2l->funcname) : NULL;
+ char *filename = a2l->filename ? strdup(a2l->filename) : NULL;
- list_for_each_entry_safe_reverse(ilist, n, &node->val, list)
- list_move_tail(&ilist->list, &node->val);
+ return inline_list__append(filename, funcname, a2l->line, node, dso);
}
static int addr2line(const char *dso_name, u64 addr,
bfd_map_over_sections(a2l->abfd, find_address_in_section, a2l);
- if (a2l->found && unwind_inlines) {
+ if (!a2l->found)
+ return 0;
+
+ if (unwind_inlines) {
int cnt = 0;
+ if (node && inline_list__append_dso_a2l(dso, node))
+ return 0;
+
while (bfd_find_inliner_info(a2l->abfd, &a2l->filename,
&a2l->funcname, &a2l->line) &&
cnt++ < MAX_INLINE_NEST) {
if (node != NULL) {
- if (inline_list__append(strdup(a2l->filename),
- strdup(a2l->funcname),
- a2l->line, node,
- dso) != 0)
+ if (inline_list__append_dso_a2l(dso, node))
return 0;
+ // found at least one inline frame
+ ret = 1;
}
}
+ }
- if ((node != NULL) &&
- (callchain_param.order != ORDER_CALLEE)) {
- inline_list__reverse(node);
- }
+ if (file) {
+ *file = a2l->filename ? strdup(a2l->filename) : NULL;
+ ret = *file ? 1 : 0;
}
- if (a2l->found && a2l->filename) {
- *file = strdup(a2l->filename);
+ if (line)
*line = a2l->line;
- if (*file)
- ret = 1;
- }
-
return ret;
}
static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
struct dso *dso)
{
- char *file = NULL;
- unsigned int line = 0;
struct inline_node *node;
node = zalloc(sizeof(*node));
INIT_LIST_HEAD(&node->val);
node->addr = addr;
- if (!addr2line(dso_name, addr, &file, &line, dso, TRUE, node))
+ if (!addr2line(dso_name, addr, NULL, NULL, dso, TRUE, node))
goto out_free_inline_node;
if (list_empty(&node->val))
{
struct unwind_info *ui = arg;
Dwarf_Addr pc;
+ bool isactivation;
- if (!dwfl_frame_pc(state, &pc, NULL)) {
+ if (!dwfl_frame_pc(state, &pc, &isactivation)) {
pr_err("%s", dwfl_errmsg(-1));
return DWARF_CB_ABORT;
}
+ if (!isactivation)
+ --pc;
+
return entry(pc, ui) || !(--ui->max_stack) ?
DWARF_CB_ABORT : DWARF_CB_OK;
}
while (!ret && (unw_step(&c) > 0) && i < max_stack) {
unw_get_reg(&c, UNW_REG_IP, &ips[i]);
+
+ /*
+ * Decrement the IP for any non-activation frames.
+ * this is required to properly find the srcline
+ * for caller frames.
+ * See also the documentation for dwfl_frame_pc(),
+ * which this code tries to replicate.
+ */
+ if (unw_is_signal_frame(&c) <= 0)
+ --ips[i];
+
++i;
}
--- /dev/null
+acpidbg
+acpidump
+ec
+include
CFLAGS += -Wall -O2 -I$(APIDIR) -I$(LIBDIR) -I$(GENDIR) $(GENFLAGS) -I../../../include
LDLIBS += -lcap -lelf
-TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs
+TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs \
+ test_align
TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o
CLANG ?= clang
%.o: %.c
- $(CLANG) -I. -I../../../include/uapi -I../../../../samples/bpf/ \
+ $(CLANG) -I. -I./include/uapi -I../../../include/uapi \
+ -I../../../../samples/bpf/ \
-Wno-compare-distinct-pointer-types \
-O2 -target bpf -c $< -o $@
--- /dev/null
+#ifndef _UAPI_LINUX_TYPES_H
+#define _UAPI_LINUX_TYPES_H
+
+#include <asm-generic/int-ll64.h>
+
+/* copied from linux:include/uapi/linux/types.h */
+#define __bitwise
+typedef __u16 __bitwise __le16;
+typedef __u16 __bitwise __be16;
+typedef __u32 __bitwise __le32;
+typedef __u32 __bitwise __be32;
+typedef __u64 __bitwise __le64;
+typedef __u64 __bitwise __be64;
+
+typedef __u16 __bitwise __sum16;
+typedef __u32 __bitwise __wsum;
+
+#define __aligned_u64 __u64 __attribute__((aligned(8)))
+#define __aligned_be64 __be64 __attribute__((aligned(8)))
+#define __aligned_le64 __le64 __attribute__((aligned(8)))
+
+#endif /* _UAPI_LINUX_TYPES_H */
--- /dev/null
+#include <asm/types.h>
+#include <linux/types.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <errno.h>
+#include <string.h>
+#include <stddef.h>
+#include <stdbool.h>
+
+#include <linux/unistd.h>
+#include <linux/filter.h>
+#include <linux/bpf_perf_event.h>
+#include <linux/bpf.h>
+
+#include <bpf/bpf.h>
+
+#include "../../../include/linux/filter.h"
+
+#ifndef ARRAY_SIZE
+# define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
+#endif
+
+#define MAX_INSNS 512
+#define MAX_MATCHES 16
+
+struct bpf_align_test {
+ const char *descr;
+ struct bpf_insn insns[MAX_INSNS];
+ enum {
+ UNDEF,
+ ACCEPT,
+ REJECT
+ } result;
+ enum bpf_prog_type prog_type;
+ const char *matches[MAX_MATCHES];
+};
+
+static struct bpf_align_test tests[] = {
+ {
+ .descr = "mov",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_3, 2),
+ BPF_MOV64_IMM(BPF_REG_3, 4),
+ BPF_MOV64_IMM(BPF_REG_3, 8),
+ BPF_MOV64_IMM(BPF_REG_3, 16),
+ BPF_MOV64_IMM(BPF_REG_3, 32),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "1: R1=ctx R3=imm2,min_value=2,max_value=2,min_align=2 R10=fp",
+ "2: R1=ctx R3=imm4,min_value=4,max_value=4,min_align=4 R10=fp",
+ "3: R1=ctx R3=imm8,min_value=8,max_value=8,min_align=8 R10=fp",
+ "4: R1=ctx R3=imm16,min_value=16,max_value=16,min_align=16 R10=fp",
+ "5: R1=ctx R3=imm32,min_value=32,max_value=32,min_align=32 R10=fp",
+ },
+ },
+ {
+ .descr = "shift",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_3, 4),
+ BPF_MOV64_IMM(BPF_REG_4, 32),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "1: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R10=fp",
+ "2: R1=ctx R3=imm2,min_value=2,max_value=2,min_align=2 R10=fp",
+ "3: R1=ctx R3=imm4,min_value=4,max_value=4,min_align=4 R10=fp",
+ "4: R1=ctx R3=imm8,min_value=8,max_value=8,min_align=8 R10=fp",
+ "5: R1=ctx R3=imm16,min_value=16,max_value=16,min_align=16 R10=fp",
+ "6: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R10=fp",
+ "7: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R4=imm32,min_value=32,max_value=32,min_align=32 R10=fp",
+ "8: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R4=imm16,min_value=16,max_value=16,min_align=16 R10=fp",
+ "9: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R4=imm8,min_value=8,max_value=8,min_align=8 R10=fp",
+ "10: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R4=imm4,min_value=4,max_value=4,min_align=4 R10=fp",
+ "11: R1=ctx R3=imm1,min_value=1,max_value=1,min_align=1 R4=imm2,min_value=2,max_value=2,min_align=2 R10=fp",
+ },
+ },
+ {
+ .descr = "addsub",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_3, 4),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 4),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, 2),
+ BPF_MOV64_IMM(BPF_REG_4, 8),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 4),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 2),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "1: R1=ctx R3=imm4,min_value=4,max_value=4,min_align=4 R10=fp",
+ "2: R1=ctx R3=imm8,min_value=8,max_value=8,min_align=4 R10=fp",
+ "3: R1=ctx R3=imm10,min_value=10,max_value=10,min_align=2 R10=fp",
+ "4: R1=ctx R3=imm10,min_value=10,max_value=10,min_align=2 R4=imm8,min_value=8,max_value=8,min_align=8 R10=fp",
+ "5: R1=ctx R3=imm10,min_value=10,max_value=10,min_align=2 R4=imm12,min_value=12,max_value=12,min_align=4 R10=fp",
+ "6: R1=ctx R3=imm10,min_value=10,max_value=10,min_align=2 R4=imm14,min_value=14,max_value=14,min_align=2 R10=fp",
+ },
+ },
+ {
+ .descr = "mul",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_3, 7),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_3, 2),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_3, 4),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "1: R1=ctx R3=imm7,min_value=7,max_value=7,min_align=1 R10=fp",
+ "2: R1=ctx R3=imm7,min_value=7,max_value=7,min_align=1 R10=fp",
+ "3: R1=ctx R3=imm14,min_value=14,max_value=14,min_align=2 R10=fp",
+ "4: R1=ctx R3=imm56,min_value=56,max_value=56,min_align=4 R10=fp",
+ },
+ },
+
+#define PREP_PKT_POINTERS \
+ BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, \
+ offsetof(struct __sk_buff, data)), \
+ BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1, \
+ offsetof(struct __sk_buff, data_end))
+
+#define LOAD_UNKNOWN(DST_REG) \
+ PREP_PKT_POINTERS, \
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2), \
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 8), \
+ BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_0, 1), \
+ BPF_EXIT_INSN(), \
+ BPF_LDX_MEM(BPF_B, DST_REG, BPF_REG_2, 0)
+
+ {
+ .descr = "unknown shift",
+ .insns = {
+ LOAD_UNKNOWN(BPF_REG_3),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_3, 1),
+ LOAD_UNKNOWN(BPF_REG_4),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_4, 5),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_ALU64_IMM(BPF_RSH, BPF_REG_4, 1),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "7: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R10=fp",
+ "8: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv55,min_align=2 R10=fp",
+ "9: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv54,min_align=4 R10=fp",
+ "10: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv53,min_align=8 R10=fp",
+ "11: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv52,min_align=16 R10=fp",
+ "18: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv56 R10=fp",
+ "19: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv51,min_align=32 R10=fp",
+ "20: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv52,min_align=16 R10=fp",
+ "21: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv53,min_align=8 R10=fp",
+ "22: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv54,min_align=4 R10=fp",
+ "23: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv55,min_align=2 R10=fp",
+ },
+ },
+ {
+ .descr = "unknown mul",
+ .insns = {
+ LOAD_UNKNOWN(BPF_REG_3),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_3),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_4, 1),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_3),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_4, 2),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_3),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_4, 4),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_3),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_4, 8),
+ BPF_ALU64_IMM(BPF_MUL, BPF_REG_4, 2),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "7: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R10=fp",
+ "8: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv56 R10=fp",
+ "9: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv55,min_align=1 R10=fp",
+ "10: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv56 R10=fp",
+ "11: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv54,min_align=2 R10=fp",
+ "12: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv56 R10=fp",
+ "13: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv53,min_align=4 R10=fp",
+ "14: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv56 R10=fp",
+ "15: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv52,min_align=8 R10=fp",
+ "16: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=inv56 R4=inv50,min_align=8 R10=fp"
+ },
+ },
+ {
+ .descr = "packet const offset",
+ .insns = {
+ PREP_PKT_POINTERS,
+ BPF_MOV64_REG(BPF_REG_5, BPF_REG_2),
+
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+
+ /* Skip over ethernet header. */
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 14),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_5),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 4),
+ BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_4, 1),
+ BPF_EXIT_INSN(),
+
+ BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_5, 0),
+ BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_5, 1),
+ BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_5, 2),
+ BPF_LDX_MEM(BPF_B, BPF_REG_4, BPF_REG_5, 3),
+ BPF_LDX_MEM(BPF_H, BPF_REG_4, BPF_REG_5, 0),
+ BPF_LDX_MEM(BPF_H, BPF_REG_4, BPF_REG_5, 2),
+ BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_5, 0),
+
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ "4: R0=imm0,min_value=0,max_value=0,min_align=2147483648 R1=ctx R2=pkt(id=0,off=0,r=0) R3=pkt_end R5=pkt(id=0,off=0,r=0) R10=fp",
+ "5: R0=imm0,min_value=0,max_value=0,min_align=2147483648 R1=ctx R2=pkt(id=0,off=0,r=0) R3=pkt_end R5=pkt(id=0,off=14,r=0) R10=fp",
+ "6: R0=imm0,min_value=0,max_value=0,min_align=2147483648 R1=ctx R2=pkt(id=0,off=0,r=0) R3=pkt_end R4=pkt(id=0,off=14,r=0) R5=pkt(id=0,off=14,r=0) R10=fp",
+ "10: R0=imm0,min_value=0,max_value=0,min_align=2147483648 R1=ctx R2=pkt(id=0,off=0,r=18) R3=pkt_end R4=inv56 R5=pkt(id=0,off=14,r=18) R10=fp",
+ "14: R0=imm0,min_value=0,max_value=0,min_align=2147483648 R1=ctx R2=pkt(id=0,off=0,r=18) R3=pkt_end R4=inv48 R5=pkt(id=0,off=14,r=18) R10=fp",
+ "15: R0=imm0,min_value=0,max_value=0,min_align=2147483648 R1=ctx R2=pkt(id=0,off=0,r=18) R3=pkt_end R4=inv48 R5=pkt(id=0,off=14,r=18) R10=fp",
+ },
+ },
+ {
+ .descr = "packet variable offset",
+ .insns = {
+ LOAD_UNKNOWN(BPF_REG_6),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_6, 2),
+
+ /* First, add a constant to the R5 packet pointer,
+ * then a variable with a known alignment.
+ */
+ BPF_MOV64_REG(BPF_REG_5, BPF_REG_2),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 14),
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_5, BPF_REG_6),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_5),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 4),
+ BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_4, 1),
+ BPF_EXIT_INSN(),
+ BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_5, 0),
+
+ /* Now, test in the other direction. Adding first
+ * the variable offset to R5, then the constant.
+ */
+ BPF_MOV64_REG(BPF_REG_5, BPF_REG_2),
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_5, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 14),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_5),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 4),
+ BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_4, 1),
+ BPF_EXIT_INSN(),
+ BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_5, 0),
+
+ /* Test multiple accumulations of unknown values
+ * into a packet pointer.
+ */
+ BPF_MOV64_REG(BPF_REG_5, BPF_REG_2),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 14),
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_5, BPF_REG_6),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_5, 4),
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_5, BPF_REG_6),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_5),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 4),
+ BPF_JMP_REG(BPF_JGE, BPF_REG_3, BPF_REG_4, 1),
+ BPF_EXIT_INSN(),
+ BPF_LDX_MEM(BPF_W, BPF_REG_4, BPF_REG_5, 0),
+
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .matches = {
+ /* Calculated offset in R6 has unknown value, but known
+ * alignment of 4.
+ */
+ "8: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R6=inv54,min_align=4 R10=fp",
+
+ /* Offset is added to packet pointer R5, resulting in known
+ * auxiliary alignment and offset.
+ */
+ "11: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R5=pkt(id=1,off=0,r=0),aux_off=14,aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+
+ /* At the time the word size load is performed from R5,
+ * it's total offset is NET_IP_ALIGN + reg->off (0) +
+ * reg->aux_off (14) which is 16. Then the variable
+ * offset is considered using reg->aux_off_align which
+ * is 4 and meets the load's requirements.
+ */
+ "15: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=pkt(id=1,off=4,r=4),aux_off=14,aux_off_align=4 R5=pkt(id=1,off=0,r=4),aux_off=14,aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+
+
+ /* Variable offset is added to R5 packet pointer,
+ * resulting in auxiliary alignment of 4.
+ */
+ "18: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv,aux_off=14,aux_off_align=4 R5=pkt(id=2,off=0,r=0),aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+
+ /* Constant offset is added to R5, resulting in
+ * reg->off of 14.
+ */
+ "19: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv,aux_off=14,aux_off_align=4 R5=pkt(id=2,off=14,r=0),aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+
+ /* At the time the word size load is performed from R5,
+ * it's total offset is NET_IP_ALIGN + reg->off (14) which
+ * is 16. Then the variable offset is considered using
+ * reg->aux_off_align which is 4 and meets the load's
+ * requirements.
+ */
+ "23: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=pkt(id=2,off=18,r=18),aux_off_align=4 R5=pkt(id=2,off=14,r=18),aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+
+ /* Constant offset is added to R5 packet pointer,
+ * resulting in reg->off value of 14.
+ */
+ "26: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv,aux_off_align=4 R5=pkt(id=0,off=14,r=8) R6=inv54,min_align=4 R10=fp",
+ /* Variable offset is added to R5, resulting in an
+ * auxiliary offset of 14, and an auxiliary alignment of 4.
+ */
+ "27: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv,aux_off_align=4 R5=pkt(id=3,off=0,r=0),aux_off=14,aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+ /* Constant is added to R5 again, setting reg->off to 4. */
+ "28: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv,aux_off_align=4 R5=pkt(id=3,off=4,r=0),aux_off=14,aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+ /* And once more we add a variable, which causes an accumulation
+ * of reg->off into reg->aux_off_align, with resulting value of
+ * 18. The auxiliary alignment stays at 4.
+ */
+ "29: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=inv,aux_off_align=4 R5=pkt(id=4,off=0,r=0),aux_off=18,aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+ /* At the time the word size load is performed from R5,
+ * it's total offset is NET_IP_ALIGN + reg->off (0) +
+ * reg->aux_off (18) which is 20. Then the variable offset
+ * is considered using reg->aux_off_align which is 4 and meets
+ * the load's requirements.
+ */
+ "33: R0=pkt(id=0,off=8,r=8) R1=ctx R2=pkt(id=0,off=0,r=8) R3=pkt_end R4=pkt(id=4,off=4,r=4),aux_off=18,aux_off_align=4 R5=pkt(id=4,off=0,r=4),aux_off=18,aux_off_align=4 R6=inv54,min_align=4 R10=fp",
+ },
+ },
+};
+
+static int probe_filter_length(const struct bpf_insn *fp)
+{
+ int len;
+
+ for (len = MAX_INSNS - 1; len > 0; --len)
+ if (fp[len].code != 0 || fp[len].imm != 0)
+ break;
+ return len + 1;
+}
+
+static char bpf_vlog[32768];
+
+static int do_test_single(struct bpf_align_test *test)
+{
+ struct bpf_insn *prog = test->insns;
+ int prog_type = test->prog_type;
+ int prog_len, i;
+ int fd_prog;
+ int ret;
+
+ prog_len = probe_filter_length(prog);
+ fd_prog = bpf_verify_program(prog_type ? : BPF_PROG_TYPE_SOCKET_FILTER,
+ prog, prog_len, 1, "GPL", 0,
+ bpf_vlog, sizeof(bpf_vlog));
+ if (fd_prog < 0) {
+ printf("Failed to load program.\n");
+ printf("%s", bpf_vlog);
+ ret = 1;
+ } else {
+ ret = 0;
+ for (i = 0; i < MAX_MATCHES; i++) {
+ const char *t, *m = test->matches[i];
+
+ if (!m)
+ break;
+ t = strstr(bpf_vlog, m);
+ if (!t) {
+ printf("Failed to find match: %s\n", m);
+ ret = 1;
+ printf("%s", bpf_vlog);
+ break;
+ }
+ }
+ close(fd_prog);
+ }
+ return ret;
+}
+
+static int do_test(unsigned int from, unsigned int to)
+{
+ int all_pass = 0;
+ int all_fail = 0;
+ unsigned int i;
+
+ for (i = from; i < to; i++) {
+ struct bpf_align_test *test = &tests[i];
+ int fail;
+
+ printf("Test %3d: %s ... ",
+ i, test->descr);
+ fail = do_test_single(test);
+ if (fail) {
+ all_fail++;
+ printf("FAIL\n");
+ } else {
+ all_pass++;
+ printf("PASS\n");
+ }
+ }
+ printf("Results: %d pass %d fail\n",
+ all_pass, all_fail);
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ unsigned int from = 0, to = ARRAY_SIZE(tests);
+
+ if (argc == 3) {
+ unsigned int l = atoi(argv[argc - 2]);
+ unsigned int u = atoi(argv[argc - 1]);
+
+ if (l < to && u < to) {
+ from = l;
+ to = u + 1;
+ }
+ } else if (argc == 2) {
+ unsigned int t = atoi(argv[argc - 1]);
+
+ if (t < to) {
+ from = t;
+ to = t + 1;
+ }
+ }
+ return do_test(from, to);
+}
* License as published by the Free Software Foundation.
*/
#include <stddef.h>
+#include <string.h>
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/if_packet.h>
#define MAX_NR_MAPS 4
#define F_NEEDS_EFFICIENT_UNALIGNED_ACCESS (1 << 0)
+#define F_LOAD_WITH_STRICT_ALIGNMENT (1 << 1)
struct bpf_test {
const char *descr;
.result = REJECT,
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
},
+ {
+ "direct packet access: test17 (pruning, alignment)",
+ .insns = {
+ BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1,
+ offsetof(struct __sk_buff, data)),
+ BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1,
+ offsetof(struct __sk_buff, data_end)),
+ BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_1,
+ offsetof(struct __sk_buff, mark)),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 14),
+ BPF_JMP_IMM(BPF_JGT, BPF_REG_7, 1, 4),
+ BPF_JMP_REG(BPF_JGT, BPF_REG_0, BPF_REG_3, 1),
+ BPF_STX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, -4),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, 1),
+ BPF_JMP_A(-6),
+ },
+ .errstr = "misaligned packet access off 2+15+-4 size 4",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .flags = F_LOAD_WITH_STRICT_ALIGNMENT,
+ },
{
"helper access to packet: test1, valid packet_ptr range",
.insns = {
.result = ACCEPT,
.prog_type = BPF_PROG_TYPE_SCHED_CLS
},
+ {
+ "alu ops on ptr_to_map_value_or_null, 1",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_1, 10),
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_1, -8),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, -2),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 2),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+ BPF_ST_MEM(BPF_DW, BPF_REG_4, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 4 },
+ .errstr = "R4 invalid mem access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS
+ },
+ {
+ "alu ops on ptr_to_map_value_or_null, 2",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_1, 10),
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_1, -8),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_4, -1),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+ BPF_ST_MEM(BPF_DW, BPF_REG_4, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 4 },
+ .errstr = "R4 invalid mem access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS
+ },
+ {
+ "alu ops on ptr_to_map_value_or_null, 3",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_1, 10),
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_1, -8),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,
+ BPF_FUNC_map_lookup_elem),
+ BPF_MOV64_REG(BPF_REG_4, BPF_REG_0),
+ BPF_ALU64_IMM(BPF_LSH, BPF_REG_4, 1),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
+ BPF_ST_MEM(BPF_DW, BPF_REG_4, 0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map1 = { 4 },
+ .errstr = "R4 invalid mem access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS
+ },
{
"invalid memory access with multiple map_lookup_elem calls",
.insns = {
.fixup_map_in_map = { 3 },
.errstr = "R1 type=map_value_or_null expected=map_ptr",
.result = REJECT,
- }
+ },
+ {
+ "ld_abs: check calling conv, r1",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_1, 0),
+ BPF_LD_ABS(BPF_W, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R1 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_abs: check calling conv, r2",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_2, 0),
+ BPF_LD_ABS(BPF_W, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R2 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_abs: check calling conv, r3",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_3, 0),
+ BPF_LD_ABS(BPF_W, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_3),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R3 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_abs: check calling conv, r4",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_4, 0),
+ BPF_LD_ABS(BPF_W, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_4),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R4 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_abs: check calling conv, r5",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_5, 0),
+ BPF_LD_ABS(BPF_W, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_5),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R5 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_abs: check calling conv, r7",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_7, 0),
+ BPF_LD_ABS(BPF_W, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_7),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
+ {
+ "ld_ind: check calling conv, r1",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_1, 1),
+ BPF_LD_IND(BPF_W, BPF_REG_1, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_1),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R1 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_ind: check calling conv, r2",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_2, 1),
+ BPF_LD_IND(BPF_W, BPF_REG_2, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R2 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_ind: check calling conv, r3",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_3, 1),
+ BPF_LD_IND(BPF_W, BPF_REG_3, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_3),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R3 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_ind: check calling conv, r4",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_4, 1),
+ BPF_LD_IND(BPF_W, BPF_REG_4, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_4),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R4 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_ind: check calling conv, r5",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_5, 1),
+ BPF_LD_IND(BPF_W, BPF_REG_5, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_5),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "R5 !read_ok",
+ .result = REJECT,
+ },
+ {
+ "ld_ind: check calling conv, r7",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
+ BPF_MOV64_IMM(BPF_REG_7, 1),
+ BPF_LD_IND(BPF_W, BPF_REG_7, -0x200000),
+ BPF_MOV64_REG(BPF_REG_0, BPF_REG_7),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ },
};
static int probe_filter_length(const struct bpf_insn *fp)
do_test_fixup(test, prog, map_fds);
- fd_prog = bpf_load_program(prog_type ? : BPF_PROG_TYPE_SOCKET_FILTER,
- prog, prog_len, "GPL", 0, bpf_vlog,
- sizeof(bpf_vlog));
+ fd_prog = bpf_verify_program(prog_type ? : BPF_PROG_TYPE_SOCKET_FILTER,
+ prog, prog_len, test->flags & F_LOAD_WITH_STRICT_ALIGNMENT,
+ "GPL", 0, bpf_vlog, sizeof(bpf_vlog));
expected_ret = unpriv && test->result_unpriv != UNDEF ?
test->result_unpriv : test->result;
;;
--verbose|-v|-vv)
VERBOSE=$((VERBOSE + 1))
- [ $1 == '-vv' ] && VERBOSE=$((VERBOSE + 1))
+ [ $1 = '-vv' ] && VERBOSE=$((VERBOSE + 1))
shift 1
;;
--debug|-d)
e=`cat $EVENT_ENABLE`
if [ "$e" != $val ]; then
echo "Expected $val but found $e"
- exit -1
+ exit 1
fi
}
echo > set_ftrace_filter
grep -v '^#' set_ftrace_filter | while read t; do
tr=`echo $t | cut -d: -f2`
- if [ "$tr" == "" ]; then
+ if [ "$tr" = "" ]; then
continue
fi
- if [ $tr == "enable_event" -o $tr == "disable_event" ]; then
+ if [ $tr = "enable_event" -o $tr = "disable_event" ]; then
tr=`echo $t | cut -d: -f1-4`
limit=`echo $t | cut -d: -f5`
else
if [ -d foo ]; then
fail "foo still exists"
fi
-exit 0
-
+mkdir foo
+echo "schedule:enable_event:sched:sched_switch" > foo/set_ftrace_filter
+rmdir foo
+if [ -d foo ]; then
+ fail "foo still exists"
+fi
instance_slam() {
--- /dev/null
+#!/bin/sh
+# description: Register/unregister many kprobe events
+
+# ftrace fentry skip size depends on the machine architecture.
+# Currently HAVE_KPROBES_ON_FTRACE defined on x86 and powerpc
+case `uname -m` in
+ x86_64|i[3456]86) OFFS=5;;
+ ppc*) OFFS=4;;
+ *) OFFS=0;;
+esac
+
+echo "Setup up to 256 kprobes"
+grep t /proc/kallsyms | cut -f3 -d" " | grep -v .*\\..* | \
+head -n 256 | while read i; do echo p ${i}+${OFFS} ; done > kprobe_events ||:
+
+echo 1 > events/kprobes/enable
+echo 0 > events/kprobes/enable
+echo > kprobe_events
+echo "Waiting for unoptimizing & freeing"
+sleep 5
+echo "Done"
tm-signal-context-chk-gpr
tm-signal-context-chk-vmx
tm-signal-context-chk-vsx
+tm-vmx-unavail
tm-signal-context-chk-vmx tm-signal-context-chk-vsx
TEST_GEN_PROGS := tm-resched-dscr tm-syscall tm-signal-msr-resv tm-signal-stack \
- tm-vmxcopy tm-fork tm-tar tm-tmspr $(SIGNAL_CONTEXT_CHK_TESTS)
+ tm-vmxcopy tm-fork tm-tar tm-tmspr tm-vmx-unavail \
+ $(SIGNAL_CONTEXT_CHK_TESTS)
include ../../lib.mk
$(OUTPUT)/tm-syscall: tm-syscall-asm.S
$(OUTPUT)/tm-syscall: CFLAGS += -I../../../../../usr/include
$(OUTPUT)/tm-tmspr: CFLAGS += -pthread
+$(OUTPUT)/tm-vmx-unavail: CFLAGS += -pthread -m64
SIGNAL_CONTEXT_CHK_TESTS := $(patsubst %,$(OUTPUT)/%,$(SIGNAL_CONTEXT_CHK_TESTS))
$(SIGNAL_CONTEXT_CHK_TESTS): tm-signal.S
printf("Check DSCR TM context switch: ");
fflush(stdout);
for (;;) {
- rv = 1;
asm __volatile__ (
/* set a known value into the DSCR */
"ld 3, %[dscr1];"
"mtspr %[sprn_dscr], 3;"
+ "li %[rv], 1;"
/* start and suspend a transaction */
"tbegin.;"
"beq 1f;"
--- /dev/null
+/*
+ * Copyright 2017, Michael Neuling, IBM Corp.
+ * Licensed under GPLv2.
+ * Original: Breno Leitao <brenohl@br.ibm.com> &
+ * Gustavo Bueno Romero <gromero@br.ibm.com>
+ * Edited: Michael Neuling
+ *
+ * Force VMX unavailable during a transaction and see if it corrupts
+ * the checkpointed VMX register state after the abort.
+ */
+
+#include <inttypes.h>
+#include <htmintrin.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <pthread.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <pthread.h>
+
+#include "tm.h"
+#include "utils.h"
+
+int passed;
+
+void *worker(void *unused)
+{
+ __int128 vmx0;
+ uint64_t texasr;
+
+ asm goto (
+ "li 3, 1;" /* Stick non-zero value in VMX0 */
+ "std 3, 0(%[vmx0_ptr]);"
+ "lvx 0, 0, %[vmx0_ptr];"
+
+ /* Wait here a bit so we get scheduled out 255 times */
+ "lis 3, 0x3fff;"
+ "1: ;"
+ "addi 3, 3, -1;"
+ "cmpdi 3, 0;"
+ "bne 1b;"
+
+ /* Kernel will hopefully turn VMX off now */
+
+ "tbegin. ;"
+ "beq failure;"
+
+ /* Cause VMX unavail. Any VMX instruction */
+ "vaddcuw 0,0,0;"
+
+ "tend. ;"
+ "b %l[success];"
+
+ /* Check VMX0 sanity after abort */
+ "failure: ;"
+ "lvx 1, 0, %[vmx0_ptr];"
+ "vcmpequb. 2, 0, 1;"
+ "bc 4, 24, %l[value_mismatch];"
+ "b %l[value_match];"
+ :
+ : [vmx0_ptr] "r"(&vmx0)
+ : "r3"
+ : success, value_match, value_mismatch
+ );
+
+ /* HTM aborted and VMX0 is corrupted */
+value_mismatch:
+ texasr = __builtin_get_texasr();
+
+ printf("\n\n==============\n\n");
+ printf("Failure with error: %lx\n", _TEXASR_FAILURE_CODE(texasr));
+ printf("Summary error : %lx\n", _TEXASR_FAILURE_SUMMARY(texasr));
+ printf("TFIAR exact : %lx\n\n", _TEXASR_TFIAR_EXACT(texasr));
+
+ passed = 0;
+ return NULL;
+
+ /* HTM aborted but VMX0 is correct */
+value_match:
+// printf("!");
+ return NULL;
+
+success:
+// printf(".");
+ return NULL;
+}
+
+int tm_vmx_unavail_test()
+{
+ int threads;
+ pthread_t *thread;
+
+ SKIP_IF(!have_htm());
+
+ passed = 1;
+
+ threads = sysconf(_SC_NPROCESSORS_ONLN) * 4;
+ thread = malloc(sizeof(pthread_t)*threads);
+ if (!thread)
+ return EXIT_FAILURE;
+
+ for (uint64_t i = 0; i < threads; i++)
+ pthread_create(&thread[i], NULL, &worker, NULL);
+
+ for (uint64_t i = 0; i < threads; i++)
+ pthread_join(thread[i], NULL);
+
+ free(thread);
+
+ return passed ? EXIT_SUCCESS : EXIT_FAILURE;
+}
+
+
+int main(int argc, char **argv)
+{
+ return test_harness(tm_vmx_unavail_test, "tm_vmx_unavail_test");
+}
#include <asm/kvm_hyp.h>
#define vtr_to_max_lr_idx(v) ((v) & 0xf)
-#define vtr_to_nr_pri_bits(v) (((u32)(v) >> 29) + 1)
+#define vtr_to_nr_pre_bits(v) (((u32)(v) >> 26) + 1)
static u64 __hyp_text __gic_v3_get_lr(unsigned int lr)
{
if (used_lrs) {
int i;
- u32 nr_pri_bits;
+ u32 nr_pre_bits;
cpu_if->vgic_elrsr = read_gicreg(ICH_ELSR_EL2);
write_gicreg(0, ICH_HCR_EL2);
val = read_gicreg(ICH_VTR_EL2);
- nr_pri_bits = vtr_to_nr_pri_bits(val);
+ nr_pre_bits = vtr_to_nr_pre_bits(val);
for (i = 0; i < used_lrs; i++) {
if (cpu_if->vgic_elrsr & (1 << i))
__gic_v3_set_lr(0, i);
}
- switch (nr_pri_bits) {
+ switch (nr_pre_bits) {
case 7:
cpu_if->vgic_ap0r[3] = read_gicreg(ICH_AP0R3_EL2);
cpu_if->vgic_ap0r[2] = read_gicreg(ICH_AP0R2_EL2);
cpu_if->vgic_ap0r[0] = read_gicreg(ICH_AP0R0_EL2);
}
- switch (nr_pri_bits) {
+ switch (nr_pre_bits) {
case 7:
cpu_if->vgic_ap1r[3] = read_gicreg(ICH_AP1R3_EL2);
cpu_if->vgic_ap1r[2] = read_gicreg(ICH_AP1R2_EL2);
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
u64 val;
- u32 nr_pri_bits;
+ u32 nr_pre_bits;
int i;
/*
}
val = read_gicreg(ICH_VTR_EL2);
- nr_pri_bits = vtr_to_nr_pri_bits(val);
+ nr_pre_bits = vtr_to_nr_pre_bits(val);
if (used_lrs) {
write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2);
- switch (nr_pri_bits) {
+ switch (nr_pre_bits) {
case 7:
write_gicreg(cpu_if->vgic_ap0r[3], ICH_AP0R3_EL2);
write_gicreg(cpu_if->vgic_ap0r[2], ICH_AP0R2_EL2);
write_gicreg(cpu_if->vgic_ap0r[0], ICH_AP0R0_EL2);
}
- switch (nr_pri_bits) {
+ switch (nr_pre_bits) {
case 7:
write_gicreg(cpu_if->vgic_ap1r[3], ICH_AP1R3_EL2);
write_gicreg(cpu_if->vgic_ap1r[2], ICH_AP1R2_EL2);
assert_spin_locked(&kvm->mmu_lock);
pgd = kvm->arch.pgd + stage2_pgd_index(addr);
do {
+ /*
+ * Make sure the page table is still active, as another thread
+ * could have possibly freed the page table, while we released
+ * the lock.
+ */
+ if (!READ_ONCE(kvm->arch.pgd))
+ break;
next = stage2_pgd_addr_end(addr, end);
if (!stage2_pgd_none(*pgd))
unmap_stage2_puds(kvm, pgd, addr, next);
* Walks the level-1 page table pointed to by kvm->arch.pgd and frees all
* underlying level-2 and level-3 tables before freeing the actual level-1 table
* and setting the struct pointer to NULL.
- *
- * Note we don't need locking here as this is only called when the VM is
- * destroyed, which can only be done once.
*/
void kvm_free_stage2_pgd(struct kvm *kvm)
{
- if (kvm->arch.pgd == NULL)
- return;
+ void *pgd = NULL;
spin_lock(&kvm->mmu_lock);
- unmap_stage2_range(kvm, 0, KVM_PHYS_SIZE);
+ if (kvm->arch.pgd) {
+ unmap_stage2_range(kvm, 0, KVM_PHYS_SIZE);
+ pgd = READ_ONCE(kvm->arch.pgd);
+ kvm->arch.pgd = NULL;
+ }
spin_unlock(&kvm->mmu_lock);
/* Free the HW pgd, one page at a time */
- free_pages_exact(kvm->arch.pgd, S2_PGD_SIZE);
- kvm->arch.pgd = NULL;
+ if (pgd)
+ free_pages_exact(pgd, S2_PGD_SIZE);
}
static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
* large. Otherwise, we may see kernel panics with
* CONFIG_DETECT_HUNG_TASK, CONFIG_LOCKUP_DETECTOR,
* CONFIG_LOCKDEP. Additionally, holding the lock too long
- * will also starve other vCPUs.
+ * will also starve other vCPUs. We have to also make sure
+ * that the page tables are not freed while we released
+ * the lock.
*/
- if (need_resched() || spin_needbreak(&kvm->mmu_lock))
- cond_resched_lock(&kvm->mmu_lock);
-
+ cond_resched_lock(&kvm->mmu_lock);
+ if (!READ_ONCE(kvm->arch.pgd))
+ break;
next = stage2_pgd_addr_end(addr, end);
if (stage2_pgd_present(*pgd))
stage2_wp_puds(pgd, addr, next);
* If we are creating a VCPU with a GICv3 we must also register the
* KVM io device for the redistributor that belongs to this VCPU.
*/
- if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3)
+ if (dist->vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
+ mutex_lock(&vcpu->kvm->lock);
ret = vgic_register_redist_iodev(vcpu);
+ mutex_unlock(&vcpu->kvm->lock);
+ }
return ret;
}
if (!vgic_v3_check_base(kvm))
return -EINVAL;
- rd_base = vgic->vgic_redist_base + kvm_vcpu_get_idx(vcpu) * SZ_64K * 2;
+ rd_base = vgic->vgic_redist_base + vgic->vgic_redist_free_offset;
sgi_base = rd_base + SZ_64K;
kvm_iodevice_init(&rd_dev->dev, &kvm_io_gic_ops);
mutex_lock(&kvm->slots_lock);
ret = kvm_io_bus_register_dev(kvm, KVM_MMIO_BUS, sgi_base,
SZ_64K, &sgi_dev->dev);
- mutex_unlock(&kvm->slots_lock);
- if (ret)
+ if (ret) {
kvm_io_bus_unregister_dev(kvm, KVM_MMIO_BUS,
&rd_dev->dev);
+ goto out;
+ }
+ vgic->vgic_redist_free_offset += 2 * SZ_64K;
+out:
+ mutex_unlock(&kvm->slots_lock);
return ret;
}
if (ret) {
/* The current c failed, so we start with the previous one. */
+ mutex_lock(&kvm->slots_lock);
for (c--; c >= 0; c--) {
vcpu = kvm_get_vcpu(kvm, c);
vgic_unregister_redist_iodev(vcpu);
}
+ mutex_unlock(&kvm->slots_lock);
}
return ret;
if (irq->hw) {
val |= GICH_LR_HW;
val |= irq->hwintid << GICH_LR_PHYSID_CPUID_SHIFT;
+ /*
+ * Never set pending+active on a HW interrupt, as the
+ * pending state is kept at the physical distributor
+ * level.
+ */
+ if (irq->active && irq_is_pending(irq))
+ val &= ~GICH_LR_PENDING_BIT;
} else {
if (irq->config == VGIC_CONFIG_LEVEL)
val |= GICH_LR_EOI;
if (irq->hw) {
val |= ICH_LR_HW;
val |= ((u64)irq->hwintid) << ICH_LR_PHYS_ID_SHIFT;
+ /*
+ * Never set pending+active on a HW interrupt, as the
+ * pending state is kept at the physical distributor
+ * level.
+ */
+ if (irq->active && irq_is_pending(irq))
+ val &= ~ICH_LR_PENDING_BIT;
} else {
if (irq->config == VGIC_CONFIG_LEVEL)
val |= ICH_LR_EOI;