Just thought I would revise my virtual memory understanding for some work I got to do.
Each address has its own address space (private to it). Page table keeps the mapping of the physical address to the virtual address(TLBs are cache to the page table). The extra amount of the virtual space is maintained by treating some area of the hard disk as RAM. This is called "swap space" in linux and "paging file" in windows. One can go to "Advanced System Settings->Advanced->Settings->Advanced" to view the size of paging file in windows(Did you notice the hierarchy.. windows is a funny os ;) ). This page summarises the physical memory limit for different windows platforms.
A default 32 bit windows has 4GB virtual space. The lower 2GB (0 to 0x7FFFFFFF) is used by the process and the higher 2GB (0x80000000 to 0xFFFFFFFF) is used by the system. One can also increase the process memory limit to 3GB by 4 gigabyte tuning(called 4GT). Actually this limit is configurable within a 2GB-3GB window and can be done by the command BCDEdit /set increaseuserva Megabyte. The virtual to physical mapping is managed in terms of pages of size 4KB. So, the pages which are not there in the physical RAM are in the pagefile. A valid page fault occurs if a page is not in RAM and in the swap space(or page file) and the page is brough back. An invalid page fault occurs if the page is not even in the page file. This is one of the reasons you see BSOD ;).
So, a RAM consists of a) Non-paged area which is basically non paged i.e. kernel modules have been loaded in that space and the pages are never swapped out, and b) page pool which basically contains the code, the data, the file cache. One thing to note is there always will be very little free RAM. Free RAM is wasted RAM. So, do not think that if you 8GB RAM the free RAM will increase. Your OS will always find ways to fill it. Manually emptying the RAM is thus an overkill and the programs which do this are actually causing performance degrade.
This page file is pagefile.sys. Obviously, it is kept in C: drive by default. Seek time for C: will be generally less than other drives as this drive handles most of the action.
One interesting aspect I have come across is "you can not access all the memory of your process, even when some parts seem to be free". But, first, lets discuss "memory fragmentation". So, if you have lots of physical RAM free,low committed memory but still getting a "no memory available" then it is most probably a case of memory fragmentation. So, if multiple chunks are malloced and then freed, then this creates obvious fragments in the virtual memory. and after some time it may happen that the virtual memory is so fragmented that there is no contiguous memory available for n bytes although in total a lot of memory is free. So, huge softwares generally precreate memory blocks and cache it for runtime usage.
But, the first line of the previous paragraph mentions something even more interesting, right ! So, even if you have enough memory with enough big contiguous memory blocks, you can still get a "cant alloc" problem. ntdll!ZwAllocateVirtualMemory is the preliminary function which is used to allocate memory. It is never used directly even via its user stub called VirtualAlloc. Because, we mostly use the heap manager provided by the OS. Now, VirtualAlloc has a granularity of 64KB with 4KB minimum allocation(default page size). (Note that the first 64KB and the last 64KB cant be used by a process). So, if I keep VirtualAlloc'ing at fresh addresses I while exhaust the memory space with <2G/64K iterations (as some amount of memory is already used by the process, for example the default heap, some dlls, stack of the thread etc). Now, if you start allocating space from heap such that you overflow the heap space then the heap manager will try to increase the heap as expected using VirtualAlloc and will fail as all 64KB boundaries have been exhausted. If you use a memory footprint s/w you will see lots of 60K empty blocks (as 4KB was allocated due to VirtualAlloc). Thus even though you are allocating much less than 60K and have so much of free memory (60K * no of iterations of VirtualAlloc'ing) you will still get "out of memory". So, if such a case ever occurs it means ntdll!ZwAllocateVirtualMemory is being directly used in the program and is the source of problem.
Besides, this is a useful tool to monitor the address space of a process.
I plan to cover exception handling in windows very soon.
Each address has its own address space (private to it). Page table keeps the mapping of the physical address to the virtual address(TLBs are cache to the page table). The extra amount of the virtual space is maintained by treating some area of the hard disk as RAM. This is called "swap space" in linux and "paging file" in windows. One can go to "Advanced System Settings->Advanced->Settings->Advanced" to view the size of paging file in windows(Did you notice the hierarchy.. windows is a funny os ;) ). This page summarises the physical memory limit for different windows platforms.
A default 32 bit windows has 4GB virtual space. The lower 2GB (0 to 0x7FFFFFFF) is used by the process and the higher 2GB (0x80000000 to 0xFFFFFFFF) is used by the system. One can also increase the process memory limit to 3GB by 4 gigabyte tuning(called 4GT). Actually this limit is configurable within a 2GB-3GB window and can be done by the command BCDEdit /set increaseuserva Megabyte. The virtual to physical mapping is managed in terms of pages of size 4KB. So, the pages which are not there in the physical RAM are in the pagefile. A valid page fault occurs if a page is not in RAM and in the swap space(or page file) and the page is brough back. An invalid page fault occurs if the page is not even in the page file. This is one of the reasons you see BSOD ;).
So, a RAM consists of a) Non-paged area which is basically non paged i.e. kernel modules have been loaded in that space and the pages are never swapped out, and b) page pool which basically contains the code, the data, the file cache. One thing to note is there always will be very little free RAM. Free RAM is wasted RAM. So, do not think that if you 8GB RAM the free RAM will increase. Your OS will always find ways to fill it. Manually emptying the RAM is thus an overkill and the programs which do this are actually causing performance degrade.
This page file is pagefile.sys. Obviously, it is kept in C: drive by default. Seek time for C: will be generally less than other drives as this drive handles most of the action.
One interesting aspect I have come across is "you can not access all the memory of your process, even when some parts seem to be free". But, first, lets discuss "memory fragmentation". So, if you have lots of physical RAM free,low committed memory but still getting a "no memory available" then it is most probably a case of memory fragmentation. So, if multiple chunks are malloced and then freed, then this creates obvious fragments in the virtual memory. and after some time it may happen that the virtual memory is so fragmented that there is no contiguous memory available for n bytes although in total a lot of memory is free. So, huge softwares generally precreate memory blocks and cache it for runtime usage.
But, the first line of the previous paragraph mentions something even more interesting, right ! So, even if you have enough memory with enough big contiguous memory blocks, you can still get a "cant alloc" problem. ntdll!ZwAllocateVirtualMemory is the preliminary function which is used to allocate memory. It is never used directly even via its user stub called VirtualAlloc. Because, we mostly use the heap manager provided by the OS. Now, VirtualAlloc has a granularity of 64KB with 4KB minimum allocation(default page size). (Note that the first 64KB and the last 64KB cant be used by a process). So, if I keep VirtualAlloc'ing at fresh addresses I while exhaust the memory space with <2G/64K iterations (as some amount of memory is already used by the process, for example the default heap, some dlls, stack of the thread etc). Now, if you start allocating space from heap such that you overflow the heap space then the heap manager will try to increase the heap as expected using VirtualAlloc and will fail as all 64KB boundaries have been exhausted. If you use a memory footprint s/w you will see lots of 60K empty blocks (as 4KB was allocated due to VirtualAlloc). Thus even though you are allocating much less than 60K and have so much of free memory (60K * no of iterations of VirtualAlloc'ing) you will still get "out of memory". So, if such a case ever occurs it means ntdll!ZwAllocateVirtualMemory is being directly used in the program and is the source of problem.
Besides, this is a useful tool to monitor the address space of a process.
I plan to cover exception handling in windows very soon.
No comments:
Post a Comment