About pcie in linux
Below what I know about pcie working on the example of initialization of the graphics card.
An example of a typical log initialization pcie while booting a linux kernel (in this case based on the platform 5040ds be qoriq от freescale)
/pcie@ffe200000: PCICSRBAR @ 0xdf000000 /pcie@ffe200000: Setup 64-bit PCI DMA window /pcie@ffe200000: DMA window size is 0xdf000000 Found FSL PCI host bridge at 0x0000000ffe201000. Firmware bus number: 0->0 PCI host bridge /pcie@ffe201000 ranges: MEM 0x0000000c20000000..0x0000000c3fffffff -> 0x00000000e0000000 IO 0x0000000ff8010000..0x0000000ff801ffff -> 0x0000000000000000 /pcie@ffe201000: PCICSRBAR @ 0xdf000000 /pcie@ffe201000: Setup 64-bit PCI DMA window /pcie@ffe201000: DMA window size is 0xdf000000 software IO TLB [mem 0x0bdce000-0x0fdce000] (64MB) mapped at [c00000000bdce000-c00000000fdcdfff] PCI: Probing PCI hardware fsl-pci ffe200000.pcie: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff]) pci_bus 0000:00: root bus resource [mem 0xc00000000-0xc1fffffff] (bus address [0xe0000000-0xffffffff]) pci_bus 0000:00: root bus resource [bus 00-01] pci_bus 0000:00: busn_res: [bus 00-01] end is updated to ff pci 0000:00:00.0: [1957:0450] type 01 class 0x0b2000 pci 0000:00:00.0: ignoring class 0x0b2000 (doesn't match header type 01) pci 0000:00:00.0: supports D1 D2 pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold pci 0000:01:00.0: [10de:0141] type 00 class 0x030000 pci 0000:01:00.0: reg 10: [mem 0xc00000000-0xc00ffffff] pci 0000:01:00.0: reg 14: [mem 0xc10000000-0xc1fffffff 64bit pref] pci 0000:01:00.0: reg 1c: [mem 0x1000ff000000-0x1000ffffffff 64bit] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it with 'pcie_aspm=force' pci 0000:00:00.0: PCI bridge to [bus 01-ff] pci 0000:00:00.0: bridge window [io 0x11000-0x11fff] pci 0000:00:00.0: bridge window [mem 0xc00000000-0xc1fffffff] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01 fsl-pci ffe201000.pcie: PCI host bridge to bus 0001:00 pci_bus 0001:00: root bus resource [io 0x21000-0x30fff] (bus address [0x0000-0xffff]) pci_bus 0001:00: root bus resource [mem 0xc20000000-0xc3fffffff] (bus address [0xe0000000-0xffffffff]) pci_bus 0001:00: root bus resource [bus 00] pci_bus 0001:00: busn_res: [bus 00] end is updated to ff pci 0001:00:00.0: [1957:0450] type 01 class 0x0b2000 pci 0001:00:00.0: ignoring class 0x0b2000 (doesn't match header type 01) pci 0001:00:00.0: supports D1 D2 pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0001:00:00.0: PCI bridge to [bus 01-ff] pci 0001:00:00.0: bridge window [io 0x21000-0x21fff] pci 0001:00:00.0: bridge window [mem 0x00000000-0x000fffff] pci 0001:00:00.0: bridge window [mem 0x00000000-0x000fffff 64bit pref] pci_bus 0001:01: busn_res: [bus 01-ff] end is updated to 01 pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to 01 PCI: Cannot allocate resource region 3 of device 0000:01:00.0, will remap PCI 0000:00 Cannot reserve Legacy IO [io 0x10000-0x10fff] PCI 0001:00 Cannot reserve Legacy IO [io 0x21000-0x21fff] pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x100000) pci 0000:01:00.0: BAR 3: assigned [mem 0xc01000000-0xc01ffffff 64bit] pci 0000:01:00.0: BAR 6: assigned [mem 0xc02000000-0xc0201ffff pref] pci 0000:00:00.0: PCI bridge to [bus 01] pci 0000:00:00.0: bridge window [io 0x10000-0x1ffff] pci 0000:00:00.0: bridge window [mem 0xc00000000-0xc1fffffff] pci 0001:00:00.0: PCI bridge to [bus 01] pci 0001:00:00.0: bridge window [io 0x21000-0x30fff] pci 0001:00:00.0: bridge window [mem 0xc20000000-0xc3fffffff] pci_bus 0000:00: resource 4 [io 0x10000-0x1ffff] pci_bus 0000:00: resource 5 [mem 0xc00000000-0xc1fffffff] pci_bus 0000:01: resource 0 [io 0x10000-0x1ffff] pci_bus 0000:01: resource 1 [mem 0xc00000000-0xc1fffffff] pci_bus 0001:00: resource 4 [io 0x21000-0x30fff] pci_bus 0001:00: resource 5 [mem 0xc20000000-0xc3fffffff] pci_bus 0001:01: resource 0 [io 0x21000-0x30fff] pci_bus 0001:01: resource 1 [mem 0xc20000000-0xc3fffffff]
The most important is the output beginning from detecting of the video card:
pci 0000:01:00.0: [10de:0141] type 00 class 0x030000
Next is the initialization algorithm of BAR registers of the pcie device. Very interesting to read the documentation under the link: resources.in fosecinstitute.com/system-address-map-initialization-in-x86x64-architecture-part-1-pci-based-systems/
The initialization algorithm (described in the documentation under that link) looks in uboot like this:
In the file corenet_ds.h of uboot there are the following defines:
define CONFIG_SYS_PCIE1_MEM_SIZE 0x20000000 /* 512M */ #define CONFIG_SYS_PCIE1_IO_VIRT 0xf8000000 #define CONFIG_SYS_PCIE1_IO_BUS 0x00000000
In the file pci_auto.c:
pci_hose_read_config_word(hose, dev, PCI_COMMAND, &cmdstat);
cmdstat = (cmdstat & ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)) | PCI_COMMAND_MASTER;
for (bar = PCI_BASE_ADDRESS_0;
bar < PCI_BASE_ADDRESS_0 + (bars_num * 4); bar += 4) {
/* Tickle the BAR and get the response */
#ifndef CONFIG_PCI_ENUM_ONLY
pci_hose_write_config_dword(hose, dev, bar, 0xffffffff);
debugPrintAllBarValue(__FUNCTION__,__LINE__,hose);
#endif
pci_hose_read_config_dword(hose, dev, bar, &bar_response);
/* If BAR is not implemented go to the next BAR */
if (!bar_response)
continue;
1. the value 0xffffffff is being written in the bar register
2. the value is being read from the same register.
The same algorithm in the kernel is presented in the function __pci_read_base (файл probe.c) and looks like this:
pci_read_config_dword(dev, pos, &l);
pci_write_config_dword(dev, pos, l | mask);
pci_read_config_dword(dev, pos, &sz);
pci_write_config_dword(dev, pos, l);
/*
* All bits set in sz means the device isn't working properly.
* If the BAR isn't implemented, all bits must be 0. If it's a
* memory BAR or a ROM, bit 0 must be clear; if it's an io BAR, bit
* 1 must be clear.
*/
if (!sz || sz == 0xffffffff)
goto fail;
As a result, the algorithm indicates what memory size should to be displayed in the address space of the cpu for a specific pci-device. The very important part in the log are the values associated with the content of the registers BAR directly:
pci 0000:01:00.0: reg 10: [mem 0xc00000000-0xc00ffffff] pci 0000:01:00.0: reg 14: [mem 0xc10000000-0xc1fffffff 64bit pref] pci 0000:01:00.0: reg 1c: [mem 0x1000ff000000-0x1000ffffffff 64bit] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
These address values can be different for various platforms, where this pcie device will be connected, but they will always display the memory size which must be allocated in the address space for this device. For example BAR1(reg 14) contains information about 256мбт in this case :
0xc1fffffff−0xc10000000 +1 = 0x10000000; 0x10000000 / 0x400 / 0x400 = 256МБт
We also need to pay attention to messages directly related to the initialization of BAR
pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x100000) pci 0000:01:00.0: BAR 3: assigned [mem 0xc01000000-0xc01ffffff 64bit] pci 0000:01:00.0: BAR 6: assigned [mem 0xc02000000-0xc0201ffff pref]
And to the messages while video card initialization, since there is also information about the required size of memory. For example, I had a situation that the size of memory for pcie device 128Мib was determinated in uboot but it was required about 512Mib and therefore the video was not raised in the kernel, i.e. the device /dev/fb0 had been not created:
nouveau [ PFB][0000:01:00.0] RAM type: DDR2 nouveau [ PFB][0000:01:00.0] RAM size: 512 MiB nouveau [ PFB][0000:01:00.0] ZCOMP: 378880 tags [TTM] Zone kernel: Available graphics memory: 4063564 kiB [TTM] Zone dma32: Available graphics memory: 2097152 kiB [TTM] Initializing pool allocator [TTM] Initializing DMA pool allocator nouveau [ DRM] VRAM: 508 MiB nouveau [ DRM] GART: 512 MiB nouveau [ DRM] BIT BIOS found nouveau [ DRM] Bios version 05.43.02.88 nouveau [ DRM] TMDS table version 1.1 nouveau [ DRM] DCB version 3.0 nouveau [ DRM] DCB outp 00: 01000300 00000028 nouveau [ DRM] DCB outp 01: 04011320 00000028
Besides in the dts file of the kernel the required memory size was determinate for the graphics card, but the dts file just generate declaration files and does not configurate anything. All settings are set up by uboot in the role of the BIOS on this platform.
Below (without any comments) is the fragment from a file p5040ds.dts where is shown how to inform the kernel about the memory size. As I wrote above these data are received by the kernel and it begins to work with them, but it don't realize any hardware restructuring (it should be done by uboot)):
pcie@ffe200000 {
reg = <0xf 0xfe200000 0x0 0x1000>;
/* ranges = <0x2000000 0x0 0xe0000000 0xc 0x0 0x0
0x20000000 0x1000000 0x0 0x0 0xf 0xf8000000
0x0 0x10000>;
*/
ranges = <0x2000000 0x0 0xe0000000 0xc 0x0 0x0
0x40000000 0x1000000 0x0 0x0 0xf 0xf8000000
0x0 0x10000>;
compatible = "fsl,p5040-pcie", "fsl,qoriq-pcie-v2.4", "fsl,qoriq-pcie";
device_type = "pci";
#size-cells = <0x2>;
#address-cells = <0x3>;
bus-range = <0x0 0xff>;
clock-frequency = <0x1fca055>;
interrupts = <0x10 0x2 0x1 0xf>;
fsl,iommu-parent = <0x3c>;
pcie@0 {
/* ranges = <0x2000000 0x0 0xe0000000 0x2000000 0x0 0xe0000000
0x0 0x20000000 0x1000000 0x0 0x0 0x1000000
0x0 0x0 0x0 0x10000>;
*/
ranges = <0x2000000 0x0 0xe0000000 0x2000000 0x0 0xe0000000
0x0 0x40000000 0x1000000 0x0 0x0 0x1000000
0x0 0x0 0x0 0x10000>;
