There are always problems with a huge binary generation with a linker. That’s the reason why, there is always a necessity to split even the libraries. I guess people understand how a linker works. 
Linker can generate library from a set of objects either dynamic or static. It can also generate a executable which is what known by everyone. But whatever output format you want from a linker, executable or library, linker uses all the system memory to extract the objects supplied to it. There are some optimized linkers such as Microsoft Incremental linker, which basically does this only once. But all the open source linkers available, catch up full RAM + Swap space to put all the objects linked. If the number of objects or libraries or huge, you might even get out-of-memory!!
How to solve this problems? Even microsoft’s incremental linking technology is buggy when the executable cannot be constructed sequentially. If you note in visual studio, it skips incremental linking lots of times and uses full RAM.
The best way to solve the problem is to split up the libraries and objects. But nowadays you get binaries of size 2 GB and even FAT filesystem supports a file of size upto 4 GB. So, splitting up is a big and useless headache other than avoiding this heavy usage of RAM.
I love the quote “Algorithms are for people who don’t know how to buy RAM”. Even this problem is for people who don’t know how to buy RAM or virtual memory space. But strangely while building Webkit, I got a issue from gold linker under linux qtwebkit build.
I am using Ubuntu 10.4 and I got this out of file descriptors error. I have huge enough RAM and also SWAP. But this error is because, the kernel restricts user space programs to not open files more than 1024. Check what is the value you have for the opened files,
$ ulimit –a
$ ulimit –Sn
ulimit –a will list all the limitations for the user and ulimit –Sn gives as just the software file open limit. On my machine, it is set to 1024 and it is very less for Webkit.
I tuned it to 4096 for all users by doing the following steps:
$ sudo vi /etc/security/limits.conf
Add the following lines,
#@faculty hard nproc 50
#ftp hard nproc 0
#ftp - chroot /ftp
#@student - maxlogins 4* soft nofile 4096
* hard nofile 65536
# End of file$ sudo sysctl -w fs.file-max=100000
The above lines in file means, For all users( *) soft open file is 4096 and hard open file is 65536. ulimit –n output is nothing put the soft open file. By default, it is 1024. Now you have changed it to 4096. sysctl will set FS limit. limits.conf is for kernel, sysctl is for filesystem.
You need to logout to get this effected. Just logout the user and login back.
After that, just give
# ulimit –Sn
# ulimit –Hn
You will get both hard and soft limits changed. Now do the webkit build
The above steps will avoid gold linker from throwing out of file descriptors. I think even 4096 is not enough for webkit project soon!
Please split the projects and educate the developers!!
References:
[1] http://sourceware.org/bugzilla/show_bug.cgi?id=10708
[2] http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
No comments:
Post a Comment