PCI/IOV: Reset total_VFs limit after detaching PF driver

PCI/IOV: Reset total_VFs limit after detaching PF driver

The TotalVFs register in the SR-IOV capability is the hardware limit on the
number of VFs.  A PF driver can limit the number of VFs further with
pci_sriov_set_totalvfs().  When the PF driver is removed, reset any VF
limit that was imposed by the driver because that limit may not apply to
other drivers.

Before 8d85a7a4f2c9 ("PCI/IOV: Allow PF drivers to limit total_VFs to 0"),
pci_sriov_set_totalvfs(pdev, 0) meant "we can enable TotalVFs virtual
functions", and the nfp driver used that to remove the VF limit when the
driver unloads.

8d85a7a4f2c9 broke that because instead of removing the VF limit,
pci_sriov_set_totalvfs(pdev, 0) actually sets the limit to zero, and that
limit persists even if another driver is loaded.

We could fix that by making the nfp driver reset the limit when it unloads,
but it seems more robust to do it in the PCI core instead of relying on the
driver.

The regression scenario is:

  nfp_pci_probe (driver 1)
  ...
  nfp_pci_remove
    pci_sriov_set_totalvfs(pf->pdev, 0)   # limits VFs to 0

  ...
  nfp_pci_probe (driver 2)
    nfp_rtsym_read_le("nfd_vf_cfg_max_vfs")
    # no VF limit from firmware

Now driver 2 is broken because the VF limit is still 0 from driver 1.

Fixes: 8d85a7a4f2c9 ("PCI/IOV: Allow PF drivers to limit total_VFs to 0")
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
[bhelgaas: changelog, rename functions]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>