Login
User Name:

Password:



Register
Forgot your password?
Vote for Us!
 Couple bugs
Today, 5:42 pm
By Remcon
Bug in disarm( )
Nov 12, 2017, 6:54 pm
By GatewaySysop
Bug in will_fall( )
Oct 23, 2017, 1:35 am
By GatewaySysop
Bug in do_zap( ), do_brandish( )
Oct 18, 2017, 1:52 pm
By GatewaySysop
Bug in get_exp_worth( )
Oct 10, 2017, 1:26 am
By GatewaySysop
LOP 1.45
Author: Remcon
Submitted by: Remcon
LOP Heroes Edition
Author: Vladaar
Submitted by: Vladaar
Heroes sound extras
Author: Vladaar
Submitted by: Vladaar
6Dragons 4.3
Author: Vladaar
Submitted by: Vladaar
Memwatch
Author: Johan Lindh
Submitted by: Vladaar
Users Online
CommonCrawl, Yandex, Sogou, Yahoo!

Members: 0
Guests: 17
Stats
Files
Topics
Posts
Members
Newest Member
477
3,705
19,232
608
LAntorcha
Today's Birthdays
There are no member birthdays today.
Related Links
» SmaugMuds.org » Codebases » SmaugFUSS » Suggested fixes
Forum Rules | Mark all | Recent Posts

Suggested fixes
< Newer Topic :: Older Topic > Reduce disk space usage

Pages:<< prev 1 next >>
Post is unread #1 Dec 29, 2002, 9:29 pm
Go to the top of the page
Go to the bottom of the page

Samson
Black Hand
GroupAdministrators
Posts3,639
JoinedJan 1, 2002

Couple of non-fatal suggestions for reducing the amount of wasted disk space Smaug logging takes up. Should have been done this way to begin with.

In mud.h, find:

#define BUG_FILE SYSTEM_DIR "bugs.txt" /* For bug( ) */

Remove it.

In db.c, in the bug() function, find:

    fclose( fpLOG );
    if ( ( fp = fopen( BUG_FILE, "a" ) ) != NULL )
    {
	fprintf( fp, "%s\n", buf );
	fclose( fp );
    }
    fpLOG = fopen( NULL_FILE, "r" );


Remove that.

This removes the use of the bugs.txt file in your system directory. The file often fills up with tons of spam generated by bug messages, and can grow quickly out of control if left unattended. It's often not even known to be there until a problem comes up and admins find it there.

Next fix - while you have db.c open, go to funtion area_update and find:

     fprintf( stderr, "Resetting: %s\n", pArea->filename );
     reset_area( pArea );
     if ( reset_age == -1 )
  pArea->age = -1;
     else
  pArea->age = number_range( 0, reset_age / 5 );


Remove the fprintf call. That stops it from logging every last area reset that occurs as a natural part of the game. You'd be surprised how huge your logs get if this isn't stopped.
       
Post is unread #2 Jan 1, 2003, 8:21 am
Go to the top of the page
Go to the bottom of the page

Ddruid
Member
GroupMembers
Posts11
JoinedNov 17, 2002

One of the things I have done is add a verbosity level to the logging, configurable by a command line option and a system configuration option (commandline overides). In addition I have a single 'log' function that handles all logging to files and to the admin channel.
       
Post is unread #3 Jan 1, 2003, 1:13 pm
Go to the top of the page
Go to the bottom of the page

Orion
Master Member
GroupMembers
Posts35
JoinedNov 12, 2002

These fixes are now in the main bug fixes list. Also, Druid, would you care to elaborate on what you're speaking of? I got the general gist of it, but it is a bit vague and may leave a few people quite clueless. ;)
       
Post is unread #4 Jan 1, 2003, 6:13 pm
Go to the top of the page
Go to the bottom of the page

Ddruid
Member
GroupMembers
Posts11
JoinedNov 17, 2002

Mainly I replaced log_string & bug and all references to stdout/stderr (also got rid of typo, idea, etc.) with a single logging function. I seperated log messages by their intent/importance and assigned different levels of messages, things like reset messages got LOG_INFO, minor bug messages (stuff that is self correcting) got LOG_WARN, more important messages got LOG_CRITICAL, etc. a typical call would be 'log_message( LOG_INFO, "%s has connected.", player->name );' Within the configuration (or command line) I can change the verbosity level of the messages:
0 - LOG_CRITICAL to stdout & file. all rest ignored.
1 - LOG_CRITICAL to file + level 0.
2 - LOG_WARN to stdout+ level 1 & 0.
etc.
At the highest level I get full debug messages, typically I run at level 3 which is all bug messages to stdout and to files, also in my config I can define which files to write to (again command line option overides system config). In essence I wanted something similiar to *nix syslog capabilities, it still needs a lot of work, I'm not satisfied with how I laid out the levels...

This may help a little bit, this is an excerpt from my system configuration:

<logging>
  <verbosity_level = '3' />
  <log_file log_level = 'LOG_INFO'>info.log</log_file>
  <log_file log_level = 'LOG_WARN'>warn.log</log_file>
  <log_file log_level = 'LOG_CRITICAL'>critical.log</log_file>
  <log_file log_level = 'LOG_DEBUG_GEN'>gen_debug.log</log_file>
 ....edited....
</logging>
       
Post is unread #5 Jan 11, 2003, 8:11 am
Go to the top of the page
Go to the bottom of the page

Ddruid
Member
GroupMembers
Posts11
JoinedNov 17, 2002

Since the overall subject is reducing disk space, one of the other things that I had done in the past is compressed my area files using the 'zlib' library. It required re-writting/duplicating of most of the file reading functions, but was fairly simple to do other that the fact that 'zlib' does not have a 'ungetc' function and required file pointer manipulation instead. Th ebiggest downfall was the increase in boot time, my estimate that it increased the amount of time to boot by atleast tenfold..
       
Post is unread #6 Jan 11, 2003, 7:35 pm
Go to the top of the page
Go to the bottom of the page

Orion
Master Member
GroupMembers
Posts35
JoinedNov 12, 2002

Garil of DotDII did this, as well. He had the same problem with the massive boot time, but he said that by doing some working with it, he got it down to just 3x the original loading speed. We might get lucky, and maybe he'll drop in on this thread, or start his own, and elaborate on what he did. :)
       
Post is unread #7 Jan 11, 2003, 9:08 pm
Go to the top of the page
Go to the bottom of the page

Garil

GroupMembers
Posts9
JoinedDec 24, 2003

Here are the results of my tests with zlib (in seconds of boot time):
Athlon XP 1800+     uncompressed: 5     compressed: 8      1.6x slower
UltraSparc IIe-400   uncompressed: 9     compressed: 23     2.5x slower
Pentium II 300         uncompressed: 11   compressed: 32     2.9x slower

zlib will read uncompressed files transparently, I found stdio to be a tad bit faster than the above uncompressed times, but not by much. As a result DOTDII uses the compressed area files routines, but I don't compress my area files because it's just too slow (my production server isn't the Athlon).

My first iteration resulted in worse than 10x speed boots, I never did let it finish booting, it was going on 10 minutes. The problem as Druid mentioned is there is no replacement for 'ungetc', so you have to use 'gzseek' which is EXTREMELY slow.

So the obvious thing is to remove as many gzseeks as possible. I did this by slightly changing the area file format. In SMAUG, records like mobs/objs don't have a terminating character, the end of record is determined when the next record (starting with #) is found. So I created an end of record character 'S' and resaved all the area files. During loading the code can skip the ungetc of '#' and thus save some time.

Another speedup is the line number display in low_bug, which uses fseek/ftell. If you sacrifice that feature you gain a little more speed.

Finally, if you cut down on getc's and use fgets instead, you will improve performance of both stdio and zlib.

On DOTDII, before compressing my area files are 8740kb, after 2320kb, so I would save 6420kb disk space.
       
Post is unread #8 Jan 12, 2003, 6:43 am
Go to the top of the page
Go to the bottom of the page

Ddruid
Member
GroupMembers
Posts11
JoinedNov 17, 2002

I never went as far as trying to tune it once I got it working, I figured that even with improving the time to half that was of no real advantage to me. I did try compressing a few area files once I had my file formats switched to XML, and I found that the boot times were only slightly increased since libxml2 handles gzip format natively.
       
Post is unread #9 Jan 12, 2003, 9:30 am
Go to the top of the page
Go to the bottom of the page

Garil

GroupMembers
Posts9
JoinedDec 24, 2003

How much faster/slower is your mud using libxml2 than the old stock stdio stuff?
       
Post is unread #10 Jan 12, 2003, 2:12 pm
Go to the top of the page
Go to the bottom of the page

Ddruid
Member
GroupMembers
Posts11
JoinedNov 17, 2002

I would guess it was about 1-2 times slower than stock stdio calls for initial booting. I didn't notice too much of it affecting player file loading, but by comparison to an area file a player file is tiny. The first time I started working with XML I wrote the parser(s) in a DOM context, which is by far much easier to work with, but it is an ungodly memory hog. This last time I wrote the parser(s) it was in a SAX context which, while a bit more difficult, is better suited to the large area files. I have been told that Expat is a faster parser, but I couldn't get any good documentation on the API, not that xmlsoft.org are the documentation kings.... ;) *IF* (and it is not likely) I do decide to try to starting over I will once again convert everything over to XML, but the thought of trying to re-create over three years worth of work is not at all appealing to me..
       
Pages:<< prev 1 next >>