Login
User Name:

Password:



Register
Forgot your password?
Vote for Us!
tintin++ ogg sound player script for linux
Author: Robert Smith
Submitted by: Vladaar
6Dragons ogg Soundpack
Author: Vladaar
Submitted by: Vladaar
6Dragons 4.4
Author: Vladaar
Submitted by: Vladaar
LoP 1.46
Author: Remcon
Submitted by: Remcon
LOP 1.45
Author: Remcon
Submitted by: Remcon
Users Online
CommonCrawl, Yandex, Sogou, Majestic-12

Members: 0
Guests: 3
Stats
Files
Topics
Posts
Members
Newest Member
481
3,734
19,366
618
Micheal64X
Today's Birthdays
There are no member birthdays today.
Related Links
» SmaugMuds.org » General » General Discussions » Faster compiles? How?!?
Forum Rules | Mark all | Recent Posts

Faster compiles? How?!?
< Newer Topic :: Older Topic > ccache - Compiler Cache

Pages:<< prev 1 next >>
Post is unread #1 Nov 25, 2005, 9:11 am   Last edited Nov 25, 2005, 9:12 am by Samson
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,643
JoinedJan 1, 2002

Tired of those slow compiles? Thanks to Kiasyn, I was made aware of a new tool that should help with this problem immensely. It's called ccache. What it does basically is caches the results of compiling source code so that the next time it's called on it gets processed alot faster. Compile times are reduced dramatically. More details available here: http://www.debian-administration.org/articles/129

Obviously in order for this to be effective, your server must have the ccache package installed ( mine do ).

Best illustrated with an example. This is done on the AFKMud 2.0 C++ code which is still being worked on, but I think it clearly demonstrates the power of this tool:

Test run on single CPU system running Fedora Core 4 with 768MB RAM
Before
time make clean

[Compiler spam removed]

real 2m51.428s
user 2m36.798s
sys 0m10.317s

After
time make clean

[Compiler spam removed]

real 0m11.418s
user 0m8.641s
sys 0m2.260s

How to use it
So how do you use this? It's very simple. Somewhere in your Makefile there should be a line like this:
CC      = g++


It may specify gcc instead if you're compiling C. To use the cache, simply change it to this:
CC      = ccache g++


The first time you run a clean make, it populates the cache so the compile time will be about normal. But after that is when you can expect to see huge speed boosts from it.
       
Post is unread #2 Nov 25, 2005, 2:38 pm
Go to the top of the page
Go to the bottom of the page

Conner
Sorcerer
GroupMembers
Posts870
JoinedMay 8, 2005

I guess ccache isn't in my path.. and I don't think that it's installed, you've tested this and it's the best way to go? Nick Gammon seemed to think it'd result in caching all of yuor code so that you would compile on old code instead of your changes each time. He was saying something about precompiled headers being better. :(

This is what I get when I tried to see if I had ccache:
$ ccache
bash: ccache: command not found
$ locate ccache
/usr/lib/xemacs-21.4.17/i386-redhat-linux/include/xgccache.h
/usr/lib/xemacs-21.4.17/i386-redhat-linux/include/gccache-gtk.h
/usr/include/freetype2/freetype/cache/ftccache.h
/usr/share/doc/krb5-devel-1.3.6/api/ccache.tex
/usr/share/doc/krb5-devel-1.3.6/implement/ccache-i.tex
       
Post is unread #3 Nov 25, 2005, 3:29 pm
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,643
JoinedJan 1, 2002

No C files altered, using makedepend locally
[samson@boralis: ~/Alsherok/src] touch mud.h
[samson@boralis: ~/Alsherok/src] time make
Building AFKMud....
make -s afkmud

[Compiler spam]

Done building AFKMud.
Buidling DNS Resolver...
make -s resolver

real 0m11.219s
user 0m8.389s
sys 0m2.392s

Deliberate non-useful modifications made to act_wiz.c, and imc.c
[samson@boralis: ~/Alsherok/src] touch mud.h
[samson@boralis: ~/Alsherok/src] time make
Building AFKMud....
make -s afkmud

[Compiler spam]

Done building AFKMud.
Buidling DNS Resolver...
make -s resolver

real 0m26.020s
user 0m22.365s
sys 0m2.848s

Conclusion
Seems to have made an obvious difference to me. I chose act_wiz.c and imc.c since they are two files I know take longer to compile than others.
       
Post is unread #4 Nov 27, 2005, 8:14 am   Last edited Nov 27, 2005, 1:22 pm by Zeno
Go to the top of the page
Go to the bottom of the page

Zeno
Sorcerer
GroupMembers
Posts723
JoinedMar 5, 2005

Wow indeed, I just tried it myself. Very nice; much faster.

[EDIT] Gah, I just hit my quota. The .ccache dir was getting over 20mb.
       
Post is unread #5 Nov 27, 2005, 2:04 pm
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,643
JoinedJan 1, 2002

Yeah, that's the heavy price paid for using it. It has also been pointed out by a couple of MUD client developers that the benefit isn't as good as it could be in the long run for all the space it consumes.
       
Post is unread #6 Nov 27, 2005, 2:38 pm
Go to the top of the page
Go to the bottom of the page

Zeno
Sorcerer
GroupMembers
Posts723
JoinedMar 5, 2005

Okay this is odd. I removed the ccache in the Makefile and the folder, but it keeps coming back.
       
Post is unread #7 Nov 27, 2005, 7:37 pm
Go to the top of the page
Go to the bottom of the page

Conner
Sorcerer
GroupMembers
Posts870
JoinedMay 8, 2005

It seems to me, based on my own strange configuration, that if you have multiple muds (a dev and a live for example, or a code, build, and live) that the ccache is trying to use the same cache for all sets since they're all compiling under the same user, so you're losing any benefit gained by it since it can't seem to determine which set is being compiled, is there a way to point the cache to a different location so the conflict will go away?
       
Post is unread #8 Nov 28, 2005, 6:54 pm
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,643
JoinedJan 1, 2002

Well if there are cache conflicts being generated from this, I'm not seeing them, but as someone with many different muds being compiled I can say now that I almost wish we'd never been told about this. The space being devoured by this thing isn't ultimately worth it. I ended up taking the cache use out of all of my Makefiles because it took no time at all for the directory to balloon up to several hundred megabytes. I can only assume it was never really meant for us little guys.

Nick Gammon's dependency generation mods to the Makefile do work, they help cut things down a bit but the dependency isn't quite as solidly dependable as an installed copy of makedepend. His method won't pick up modified system headers your code is dependent on. Only headers actually in your mud source. Better than nothing and universally available even on Cygwin if you install Perl.

If your system has it available, Makedepend is the best option. This cache thing only turned out to be a silver-lined thundercloud :/
       
Post is unread #9 Nov 29, 2005, 12:34 pm
Go to the top of the page
Go to the bottom of the page

Conner
Sorcerer
GroupMembers
Posts870
JoinedMay 8, 2005

So, for different overall reasons, my initial hesitation was merited. *sigh* I hate finding out that I was right in not trusting something, even if it's not for the reasons I started with.. oh well, at least the makedepend still works for me so it's already improved over what it had been.
       
Post is unread #10 Nov 29, 2005, 1:02 pm
Go to the top of the page
Go to the bottom of the page

Keberus
Conjurer
GroupFUSS Project Team
Posts341
JoinedJun 4, 2005

I have been watching this thread since it started. I even got dune to install ccache. What about having some internal mud cleanup code run on the .ccache directory. I know you can "m -fr *" then run through 2 makes and it will be fast again....so why not have to do a slower make every once in a while but still have the advantages of the fast make....especailly if you work on the code quite often. I don't know how to ...but if straight coding it woulndn't be easy how about writing code that calls a shell script that just cleans the directory...and maybe has the time logged in some file...then overwrites the file with the new time...maybe once every mudday or something the code could automatically run.

Just a thought,
KeB
       
Post is unread #11 Dec 3, 2005, 12:37 pm   Last edited Dec 3, 2005, 12:40 pm by Zeno
Go to the top of the page
Go to the bottom of the page

Zeno
Sorcerer
GroupMembers
Posts723
JoinedMar 5, 2005

I still can't get the .ccache dir to stop coming back. I've removed the "ccache" from the Makefile, and yet it still creates this directory. Anyone have any idea? It comes back everytime I do a compile.
       
Post is unread #12 Dec 3, 2005, 1:05 pm
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,643
JoinedJan 1, 2002

Hrm, ick, I have no idea why it's regenerating even when you delete the cache dir and stop using it. I just tried this myself and it decided to do the same thing. I guess in light of this it would be best to uninstall it :/
       
Post is unread #13 Dec 3, 2005, 1:10 pm
Go to the top of the page
Go to the bottom of the page

Zeno
Sorcerer
GroupMembers
Posts723
JoinedMar 5, 2005

Yeah, I really don't think this is useful for MUDs.

So we had the wear_obj bug that wouldn't go away, and now this. I'm sure we'll look back on this and laugh. ;)
       
Post is unread #14 Dec 3, 2005, 3:16 pm
Go to the top of the page
Go to the bottom of the page

Conner
Sorcerer
GroupMembers
Posts870
JoinedMay 8, 2005

Makes you wonder what sort of coding situation it would be best suited for though, doesn't it?

The big drawback for a mud is the space being used by the cache which is impractical for most muds and unreasonable for any mud that's on a paid host with space limits.. so what sort of application would make that space for the cache worthwhile? Any thoughts?
       
Post is unread #15 Mar 16, 2006, 9:42 am
Go to the top of the page
Go to the bottom of the page

Keberus
Conjurer
GroupFUSS Project Team
Posts341
JoinedJun 4, 2005

Conner said:

Makes you wonder what sort of coding situation it would be best suited for though, doesn't it?

The big drawback for a mud is the space being used by the cache which is impractical for most muds and unreasonable for any mud that's on a paid host with space limits.. so what sort of application would make that space for the cache worthwhile? Any thoughts?


I know this is an old topic but I decided to play around with ccache some more on our server and noticed that you can actually set your own size limits.

if you type ccache by itself you should get something like:

ccache, a compiler cache. Version 2.3
Copyright Andrew Tridgell, 2002

Usage:
ccache [options]
ccache compiler [compile options]
compiler [compile options] (via symbolic link)

Options:
-s show statistics summary
-z zero statistics
-c run a cache cleanup
-C clear the cache completely
-F <maxfiles> set maximum files in cache
-M <maxsize> set maximum size of cache (use G, M or K)
-h this help page
-V print version number


so you can set the max space by
ccache -M 20M
and that will set the max ccache size to 20 megs...I can also say that we have been using ccache on the dune server since my post about it being installed (above) and have not had any noticible problems even when changing quite a bit of code.

Anyways just my two cents on the matter.

Hope this helps someone,
KeB
       
Post is unread #16 Mar 16, 2006, 9:55 am
Go to the top of the page
Go to the bottom of the page

Conner
Sorcerer
GroupMembers
Posts870
JoinedMay 8, 2005

Hmm, now that might make it worthwhile afterall. Once you've set a ccache size limit for it, that would eliminate the biggest drawback.. in your experiences to date, Keberus, have you encountered the problem I'd mentioned about it seeming to want to use the same cache set for each port under your user id? If so, have you found a solution to that?
       
Post is unread #17 Mar 16, 2006, 12:16 pm
Go to the top of the page
Go to the bottom of the page

Keberus
Conjurer
GroupFUSS Project Team
Posts341
JoinedJun 4, 2005

Conner said:

Hmm, now that might make it worthwhile afterall. Once you've set a ccache size limit for it, that would eliminate the biggest drawback.. in your experiences to date, Keberus, have you encountered the problem I'd mentioned about it seeming to want to use the same cache set for each port under your user id? If so, have you found a solution to that?


Okay, I can tell you what I did as a test. I use a fote deriv as the codebase and the folder is swfote and of course the code is in source ....i went ahead and cleaned the cache with "ccache -c" then i made a folder called swfotecode and extracted a copy of my running codebase into there....i then changed a function to ouput another string in keb.c and did a make. I switched directories to the normal game and did a make clean and a make to see if it would use the cached keb.c from the other mud(which is what I think you wanted to test). I copyover'ed and it didnt change it so it must have used the right one...and not the other cached keb.c....Sorry If this isnt what you wanted to know. This is only an idea but maybe the ccache keeps track of the directory the file came from so a keb.c in swfotecode/swfote/src isnt the same as keb.c in swfote/src. But I have no clue if thats what really is going on or not.

Hope this helps,
KeB
       
Post is unread #18 Mar 16, 2006, 4:13 pm
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,643
JoinedJan 1, 2002

The new information certainly makes it more useful for personal use, but I still don't think it holds much value for a shared hosting solution. Too many users would either ignore the settings, or not know how to adjust them, and hosts won't have the time to babysit them through it.
       
Post is unread #19 Mar 16, 2006, 7:08 pm
Go to the top of the page
Go to the bottom of the page

Conner
Sorcerer
GroupMembers
Posts870
JoinedMay 8, 2005

Keberus: I'm not certain myself if that's what I meant, but if the cache appears to know which directory the cached contents came from it would mean that my 'issue' must have either been a fluke or a hallucination, either of which has equal possibility at this point. ;)

Samson: I can understand your point there, most hosts have no reason to inspect a customer's code (even just the Makefile) let alone keep tabs on it to make sure they've got their ccache settings correct and not all hosts are going to allocate a specific amount of disk space, even as a standard minimum so it'd be unreasonable to set a default value for the stock package, but I think I also agree that for individual users it sounds like it may be a worthwhile idea as long as you already know what sort of limits you've got to work inside of.
       
Pages:<< prev 1 next >>