Love your stuff! This is really handy tutorials. And no need to dive in f@#*ing documentation. At least at start. Thank you very much! You doing really Great Job! Please don't stop!
1st off all Thank You for this Informative video series. Please explain why calculation of the offset is required @11:27 ? In Kernel module also is it required ?
You are welcome. The Offset is needed because of mmap. Mmap returns a page align Pointer. In case the BAR is smaller then the pagesize (4kByte) it is possible the BAR doesn't start at the start of the page and you have to add an Offset. If they BAR is bigger then a page, you don't need an Offset. Hope this helps ;)
Well, the offsets are from the datasheet of the PCI GPIO card I use. Behind the addresses are just the registers to set the io pins to input or output and the other one should be for setting the output value. Sadly the company homepage is no longer available, so I can't sent you a link to the datasheet... But this are the offsets in the BAR: #define PCITTL32IO_GPIO_STATE 0xFC #define PCITTL32IO_DIRECTION 0xF8 #define PCITTL32IO_OFFSET_IRQ 0xF9
Thank you. I work as a hardware near software developer. I do bringups of new prototypes (including timing measurements of interfaces) and write low level and testing software.
Hi, thanks for your video. I wonder if I can use blocking `read` to read the resource file, so that I can get data when another device send packet to me.
Hey mate, thanks for this, it was all very informative, but I think I got a bit lost on the offset part. You have a 256 bit BAR, so mmap will give you a single page of memory and you'll have that BAR mapped at the beginning of the page. Why would it matter where it's mapped in the kernel and why does it stop being relevant if the BAR is larger than 4k?
You are welcome. check out the man page of mmap. We passed a NULL pointer as the first argument of mmap (addr). "If addr is NULL, then the kernel chooses the (page-aligned) address at which to create the mapping;" As we passed a NULL here, our pointer is page-aligned and on a x86 system a page is normally 4kB. But as the size of our BAR is smaller then 4kB, it is possible that our BAR is not on the start of the 4kB page but has an offset. So, we need to calculate the offset by doing (Physical BAR address % 4kB). We get the BAR address from the PCI config header and can then calculate the offset. If the page is bigger then 4kB, of course the pointer we got is page aligned and so we can skip this step. I hope this helps you.
@@johannes4gnu_linux96 Thank you, this does make sense. I expected the address to be page aligned regardless of the size, but I guess it makes sense that that's not a requirement.
@@johannes4gnu_linux96 Hi Johannes. Thanks for your video, it was interesting to watch. I did read the man page for mmap(). But I don't follow your explanation here. Can you check it please ? (In your C program, do you ever get a value for offset which is not 0 ?) I'm learning about PCIe now - if I've misunderstood, I apologise in advance. Thanks.
@johannes4gnu_linux96 Why do you say that if the BAR region is > 4KB that we are guaranteed page alignment? IOW are we guaranteed that by the kernel / firmware that it would not be at an offset (understand that that's likely the case)? And have you found instances in practice where for sizes < 4kB that the BAR actually appears at an offset less than 4kB?
very informative, giving insights to beginners. thank you again! a question. you showed an example using mmap for resource file, but can we use fread or fwrite for the resource file too? (combined with fseek) just curious..
I haven't tried fread and fwrite, but I think this is a bad idea, because these functions are made for the manipulation of real (text) files and this time the file represents the memory from the PCI card. So, I think working with mmap and pointers is better here, because on a write you could accidentally overwrite something. Let me give you an example: Let's say behind BAR0 address 0x10 is a FIFO, which writes out data and on 0x14 is a setting register, you want to change. Now you write something out to the whole BAR0 file. Now this will update the setting register as expected, but it will also write something into the FIFO and will change its state. With the pointer you can directly and only write to the 0x14 settings register.
When i write similar code to read and write my pcie endpoint, i couldnt write any value to pcie register and also when i read some register of pcie endpoint it will return all ones. Why? Mmap works well, there is no any error. İ know i didnt do any mistake. When i search this someones says there must be enable flag, what is this?
You can try to add iomem=relaxed to the kernel command line. See the following post how to do it: askubuntu.com/questions/1120578/how-do-i-edit-grub-to-add-iomem-relaxed#1162794 If it doesn't work after this, you can check if the kernel config CONFIG_IO_STRICT_DEVMEM is set. This could also prevent you from accessing the PCIe BARs from userspace.
Hi my brother, How do a access an address located out of memory bar? Basically, I have a PCIe device that's mapped its RCRB to address 0xd14000000 indicated by PCIe trace. But the PCIe header mapped the standard bar0 to 0xd1600000. I can just access this 0xd1600000 addr by using your method, namely file resource0 in the folder. But how do I access 0xd1400000?
Hi, good question. Before you asked, I didn't know RCRB exists :p Maybe this post on Stack overflow helps: stackoverflow.com/questions/12040303/how-to-access-physical-addresses-from-user-space-in-linux
Love your stuff! This is really handy tutorials. And no need to dive in f@#*ing documentation. At least at start. Thank you very much! You doing really Great Job!
Please don't stop!
11:32 5th byte of configuration space - Did you mean 5th dword? (config is type uint32_t)
always love to watch your videos, thanks for uploading!
1st off all Thank You for this Informative video series.
Please explain why calculation of the offset is required @11:27 ? In Kernel module also is it required ?
You are welcome. The Offset is needed because of mmap. Mmap returns a page align Pointer. In case the BAR is smaller then the pagesize (4kByte) it is possible the BAR doesn't start at the start of the page and you have to add an Offset. If they BAR is bigger then a page, you don't need an Offset. Hope this helps ;)
Really nice, why do we need uio-pci to access pci device memory if your method is so fast and easy?
sorry, why did you put 0xf8 and 0xfc for io config / io byte ? It doesn make sense :( for me
Well, the offsets are from the datasheet of the PCI GPIO card I use. Behind the addresses are just the registers to set the io pins to input or output and the other one should be for setting the output value. Sadly the company homepage is no longer available, so I can't sent you a link to the datasheet...
But this are the offsets in the BAR:
#define PCITTL32IO_GPIO_STATE 0xFC
#define PCITTL32IO_DIRECTION 0xF8
#define PCITTL32IO_OFFSET_IRQ 0xF9
Kindly make videos every now and then. Your video are so informative. I would love to know what u do for a living?
Thank you.
I work as a hardware near software developer. I do bringups of new prototypes (including timing measurements of interfaces) and write low level and testing software.
Hi, thanks for your video. I wonder if I can use blocking `read` to read the resource file, so that I can get data when another device send packet to me.
You are welcome. Reading should be possible. But I wouldn't do a write on the resource files as maybe some registers in the PCIe Card are read only.
Hey mate, thanks for this, it was all very informative, but I think I got a bit lost on the offset part. You have a 256 bit BAR, so mmap will give you a single page of memory and you'll have that BAR mapped at the beginning of the page. Why would it matter where it's mapped in the kernel and why does it stop being relevant if the BAR is larger than 4k?
You are welcome.
check out the man page of mmap. We passed a NULL pointer as the first argument of mmap (addr).
"If addr is NULL, then the kernel chooses the (page-aligned) address at which to create the mapping;" As we passed a NULL here, our pointer is page-aligned and on a x86 system a page is normally 4kB. But as the size of our BAR is smaller then 4kB, it is possible that our BAR is not on the start of the 4kB page but has an offset.
So, we need to calculate the offset by doing (Physical BAR address % 4kB). We get the BAR address from the PCI config header and can then calculate the offset.
If the page is bigger then 4kB, of course the pointer we got is page aligned and so we can skip this step.
I hope this helps you.
@@johannes4gnu_linux96 Thank you, this does make sense. I expected the address to be page aligned regardless of the size, but I guess it makes sense that that's not a requirement.
@@johannes4gnu_linux96 Hi Johannes. Thanks for your video, it was interesting to watch. I did read the man page for mmap(). But I don't follow your explanation here. Can you check it please ? (In your C program, do you ever get a value for offset which is not 0 ?) I'm learning about PCIe now - if I've misunderstood, I apologise in advance. Thanks.
@johannes4gnu_linux96 Why do you say that if the BAR region is > 4KB that we are guaranteed page alignment? IOW are we guaranteed that by the kernel / firmware that it would not be at an offset (understand that that's likely the case)? And have you found instances in practice where for sizes < 4kB that the BAR actually appears at an offset less than 4kB?
very informative, giving insights to beginners. thank you again! a question. you showed an example using mmap for resource file, but can we use fread or fwrite for the resource file too? (combined with fseek) just curious..
I haven't tried fread and fwrite, but I think this is a bad idea, because these functions are made for the manipulation of real (text) files and this time the file represents the memory from the PCI card. So, I think working with mmap and pointers is better here, because on a write you could accidentally overwrite something.
Let me give you an example:
Let's say behind BAR0 address 0x10 is a FIFO, which writes out data and on 0x14 is a setting register, you want to change. Now you write something out to the whole BAR0 file. Now this will update the setting register as expected, but it will also write something into the FIFO and will change its state. With the pointer you can directly and only write to the 0x14 settings register.
When i write similar code to read and write my pcie endpoint, i couldnt write any value to pcie register and also when i read some register of pcie endpoint it will return all ones. Why? Mmap works well, there is no any error. İ know i didnt do any mistake. When i search this someones says there must be enable flag, what is this?
You can try to add iomem=relaxed to the kernel command line. See the following post how to do it:
askubuntu.com/questions/1120578/how-do-i-edit-grub-to-add-iomem-relaxed#1162794
If it doesn't work after this, you can check if the kernel config CONFIG_IO_STRICT_DEVMEM is set. This could also prevent you from accessing the PCIe BARs from userspace.
you have everything i wanted to learn thank you
Hi my brother,
How do a access an address located out of memory bar?
Basically, I have a PCIe device that's mapped its RCRB to address 0xd14000000 indicated by PCIe trace. But the PCIe header mapped the standard bar0 to 0xd1600000. I can just access this 0xd1600000 addr by using your method, namely file resource0 in the folder. But how do I access 0xd1400000?
Hi, good question. Before you asked, I didn't know RCRB exists :p Maybe this post on Stack overflow helps: stackoverflow.com/questions/12040303/how-to-access-physical-addresses-from-user-space-in-linux
@@johannes4gnu_linux96 thank you my man! it worked
great work