About pcie in linux

Below what I know about pcie working on the example of initialization of the graphics card.

An example of a typical log initialization pcie while booting a linux kernel (in this case based on the platform 5040ds be qoriq от freescale)

/pcie@ffe200000: PCICSRBAR @ 0xdf000000 
/pcie@ffe200000: Setup 64-bit PCI DMA window 
/pcie@ffe200000: DMA window size is 0xdf000000 
Found FSL PCI host bridge at 0x0000000ffe201000. Firmware bus number: 0->0 
PCI host bridge /pcie@ffe201000  ranges: 
 MEM 0x0000000c20000000..0x0000000c3fffffff -> 0x00000000e0000000 
  IO 0x0000000ff8010000..0x0000000ff801ffff -> 0x0000000000000000 
/pcie@ffe201000: PCICSRBAR @ 0xdf000000 
/pcie@ffe201000: Setup 64-bit PCI DMA window 
/pcie@ffe201000: DMA window size is 0xdf000000 
software IO TLB [mem 0x0bdce000-0x0fdce000] (64MB) mapped at [c00000000bdce000-c00000000fdcdfff] 
PCI: Probing PCI hardware 
fsl-pci ffe200000.pcie: PCI host bridge to bus 0000:00 
pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff]) 
pci_bus 0000:00: root bus resource [mem 0xc00000000-0xc1fffffff] (bus address [0xe0000000-0xffffffff]) 
pci_bus 0000:00: root bus resource [bus 00-01] 
pci_bus 0000:00: busn_res: [bus 00-01] end is updated to ff 
pci 0000:00:00.0: [1957:0450] type 01 class 0x0b2000 
pci 0000:00:00.0: ignoring class 0x0b2000 (doesn't match header type 01) 
pci 0000:00:00.0: supports D1 D2 
pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold 
pci 0000:01:00.0: [10de:0141] type 00 class 0x030000 
pci 0000:01:00.0: reg 10: [mem 0xc00000000-0xc00ffffff] 
pci 0000:01:00.0: reg 14: [mem 0xc10000000-0xc1fffffff 64bit pref] 
pci 0000:01:00.0: reg 1c: [mem 0x1000ff000000-0x1000ffffffff 64bit] 
pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref] 
pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force' 
pci 0000:00:00.0: PCI bridge to [bus 01-ff] 
pci 0000:00:00.0:   bridge window [io  0x11000-0x11fff] 
pci 0000:00:00.0:   bridge window [mem 0xc00000000-0xc1fffffff] 
pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01 
pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01 
fsl-pci ffe201000.pcie: PCI host bridge to bus 0001:00 
pci_bus 0001:00: root bus resource [io  0x21000-0x30fff] (bus address [0x0000-0xffff]) 
pci_bus 0001:00: root bus resource [mem 0xc20000000-0xc3fffffff] (bus address [0xe0000000-0xffffffff]) 
pci_bus 0001:00: root bus resource [bus 00] 
pci_bus 0001:00: busn_res: [bus 00] end is updated to ff 
pci 0001:00:00.0: [1957:0450] type 01 class 0x0b2000 
pci 0001:00:00.0: ignoring class 0x0b2000 (doesn't match header type 01) 
pci 0001:00:00.0: supports D1 D2 
pci 0001:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold 
pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring 
pci 0001:00:00.0: PCI bridge to [bus 01-ff] 
pci 0001:00:00.0:   bridge window [io  0x21000-0x21fff] 
pci 0001:00:00.0:   bridge window [mem 0x00000000-0x000fffff] 
pci 0001:00:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref] 
pci_bus 0001:01: busn_res: [bus 01-ff] end is updated to 01 
pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to 01 
PCI: Cannot allocate resource region 3 of device 0000:01:00.0, will remap 
PCI 0000:00 Cannot reserve Legacy IO [io  0x10000-0x10fff] 
PCI 0001:00 Cannot reserve Legacy IO [io  0x21000-0x21fff] 
pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x100000) 
pci 0000:01:00.0: BAR 3: assigned [mem 0xc01000000-0xc01ffffff 64bit] 
pci 0000:01:00.0: BAR 6: assigned [mem 0xc02000000-0xc0201ffff pref] 
pci 0000:00:00.0: PCI bridge to [bus 01] 
pci 0000:00:00.0:   bridge window [io  0x10000-0x1ffff] 
pci 0000:00:00.0:   bridge window [mem 0xc00000000-0xc1fffffff] 
pci 0001:00:00.0: PCI bridge to [bus 01] 
pci 0001:00:00.0:   bridge window [io  0x21000-0x30fff] 
pci 0001:00:00.0:   bridge window [mem 0xc20000000-0xc3fffffff] 
pci_bus 0000:00: resource 4 [io  0x10000-0x1ffff] 
pci_bus 0000:00: resource 5 [mem 0xc00000000-0xc1fffffff] 
pci_bus 0000:01: resource 0 [io  0x10000-0x1ffff] 
pci_bus 0000:01: resource 1 [mem 0xc00000000-0xc1fffffff] 
pci_bus 0001:00: resource 4 [io  0x21000-0x30fff] 
pci_bus 0001:00: resource 5 [mem 0xc20000000-0xc3fffffff] 
pci_bus 0001:01: resource 0 [io  0x21000-0x30fff] 
pci_bus 0001:01: resource 1 [mem 0xc20000000-0xc3fffffff]

The most important is the output beginning from detecting of the video card:

pci 0000:01:00.0: [10de:0141] type 00 class 0x030000 

Next is the initialization algorithm of BAR registers of the pcie device. Very interesting to read the documentation under the link: resources.in fosecinstitute.com/system-address-map-initialization-in-x86x64-architecture-part-1-pci-based-systems/

The initialization algorithm (described in the documentation under that link) looks in uboot like this:

In the file corenet_ds.h of uboot there are the following defines:

define CONFIG_SYS_PCIE1_MEM_SIZE	0x20000000	/* 512M */
#define CONFIG_SYS_PCIE1_IO_VIRT	0xf8000000
#define CONFIG_SYS_PCIE1_IO_BUS		0x00000000

In the file pci_auto.c:

	pci_hose_read_config_word(hose, dev, PCI_COMMAND, &cmdstat);
	cmdstat = (cmdstat & ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY)) | PCI_COMMAND_MASTER;

	for (bar = PCI_BASE_ADDRESS_0;
		bar < PCI_BASE_ADDRESS_0 + (bars_num * 4); bar += 4) {


		/* Tickle the BAR and get the response */
#ifndef CONFIG_PCI_ENUM_ONLY
		pci_hose_write_config_dword(hose, dev, bar, 0xffffffff);
		debugPrintAllBarValue(__FUNCTION__,__LINE__,hose);
#endif
		pci_hose_read_config_dword(hose, dev, bar, &bar_response);

		/* If BAR is not implemented go to the next BAR */
		if (!bar_response)
			continue;

1. the value 0xffffffff is being written in the bar register

2. the value is being read from the same register.

The same algorithm in the kernel is presented in the function __pci_read_base (файл probe.c) and looks like this:

      pci_read_config_dword(dev, pos, &l); 
        pci_write_config_dword(dev, pos, l | mask); 
        pci_read_config_dword(dev, pos, &sz); 
        pci_write_config_dword(dev, pos, l); 

        /* 
         * All bits set in sz means the device isn't working properly. 
         * If the BAR isn't implemented, all bits must be 0.  If it's a 
         * memory BAR or a ROM, bit 0 must be clear; if it's an io BAR, bit 
         * 1 must be clear. 
         */ 
        if (!sz || sz == 0xffffffff) 
                goto fail; 

As a result, the algorithm indicates what memory size should to be displayed in the address space of the cpu for a specific pci-device. The very important part in the log are the values associated with the content of the registers BAR directly:

pci 0000:01:00.0: reg 10: [mem 0xc00000000-0xc00ffffff] 
pci 0000:01:00.0: reg 14: [mem 0xc10000000-0xc1fffffff 64bit pref] 
pci 0000:01:00.0: reg 1c: [mem 0x1000ff000000-0x1000ffffffff 64bit] 
pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref] 

These address values can be different for various platforms, where this pcie device will be connected, but they will always display the memory size which must be allocated in the address space for this device. For example BAR1(reg 14) contains information about 256мбт in this case :

0xc1fffffff−0xc10000000 +1 = 0x10000000;
0x10000000 / 0x400 / 0x400 = 256МБт

We also need to pay attention to messages directly related to the initialization of BAR

pci 0000:00:00.0: BAR 9: can't assign mem pref (size 0x100000) 
pci 0000:01:00.0: BAR 3: assigned [mem 0xc01000000-0xc01ffffff 64bit] 
pci 0000:01:00.0: BAR 6: assigned [mem 0xc02000000-0xc0201ffff pref]

And to the messages while video card initialization, since there is also information about the required size of memory. For example, I had a situation that the size of memory for pcie device 128Мib was determinated in uboot but it was required about 512Mib and therefore the video was not raised in the kernel, i.e. the device /dev/fb0 had been not created:

nouveau  [     PFB][0000:01:00.0] RAM type: DDR2
nouveau  [     PFB][0000:01:00.0] RAM size: 512 MiB
nouveau  [     PFB][0000:01:00.0]    ZCOMP: 378880 tags
[TTM] Zone  kernel: Available graphics memory: 4063564 kiB
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[TTM] Initializing pool allocator
[TTM] Initializing DMA pool allocator
nouveau  [     DRM] VRAM: 508 MiB
nouveau  [     DRM] GART: 512 MiB
nouveau  [     DRM] BIT BIOS found
nouveau  [     DRM] Bios version 05.43.02.88
nouveau  [     DRM] TMDS table version 1.1
nouveau  [     DRM] DCB version 3.0
nouveau  [     DRM] DCB outp 00: 01000300 00000028
nouveau  [     DRM] DCB outp 01: 04011320 00000028

Besides in the dts file of the kernel the required memory size was determinate for the graphics card, but the dts file just generate declaration files and does not configurate anything. All settings are set up by uboot in the role of the BIOS on this platform.

Below (without any comments) is the fragment from a file p5040ds.dts where is shown how to inform the kernel about the memory size. As I wrote above these data are received by the kernel and it begins to work with them, but it don't realize any hardware restructuring (it should be done by uboot)):

        pcie@ffe200000 {
                reg = <0xf 0xfe200000 0x0 0x1000>;
/*              ranges = <0x2000000   0x0       0xe0000000 0xc         0x0       0x0
              0x20000000  0x1000000 0x0        0x0         0xf       0xf8000000
              0x0         0x10000>;
*/
                        
                ranges = <0x2000000   0x0       0xe0000000 0xc         0x0       0x0
              0x40000000  0x1000000 0x0        0x0         0xf       0xf8000000
              0x0         0x10000>;
                
                                                           
                
                compatible = "fsl,p5040-pcie", "fsl,qoriq-pcie-v2.4", "fsl,qoriq-pcie";
                device_type = "pci";
                #size-cells = <0x2>;
                #address-cells = <0x3>;
                bus-range = <0x0 0xff>;
                clock-frequency = <0x1fca055>;
                interrupts = <0x10 0x2 0x1 0xf>;
                fsl,iommu-parent = <0x3c>;
                  
                pcie@0 {
/*                      ranges = <0x2000000 0x0         0xe0000000  0x2000000 0x0 0xe0000000
                0x0       0x20000000  0x1000000   0x0       0x0 0x1000000
                  0x0       0x0         0x0         0x10000>;
*/
                        ranges = <0x2000000 0x0         0xe0000000  0x2000000 0x0 0xe0000000
                0x0       0x40000000  0x1000000   0x0       0x0 0x1000000
                  0x0       0x0         0x0         0x10000>;