Re: Memory-Mapped I/O

From: Peter Ajamian (peter@pajamian.dhs.org)
Date: 03/02/01


"Daniel A. Koepke" wrote:
>
> The output is heavily dependent on your platform.  Not just between
> different machine architectures or operating system or C library, but
> between different versions of your C library and operating system.  This
> means that you should not rely on any important data to be stored in a
> binary memory map (if you're mmap'ing to read the file like one long
> string, that's different) if you think you're ever going to have to switch
> platforms.  You should not rely upon the correctness of the mapped data
> after hardware/software upgrades.

Agreed.

> As with longer term persistence, the idea of using this to have data
> persist over reboots introduces several problems that I've detailed
> before.  You cannot and should not trust the state of an unsync'd, open
> memory map at the time of a crash.

I was thinking of the possibility of using some kind of validation check
at bootup time to ensure the integrity of the memory.  If the integrity
test fails then the MUD discards the memory and does a normal full boot
and initialization (it can still use rudimentary copyover-recover at
that time to keep players in the game and to load them back in at thier
current locations).  I can think of at least a few ways to do integrity
tests of varying levels of complexity.  Here are four ordered from the
simplest and least accurate to the most complex and most accurate:

1.  Magic numbers.  Something similar to the way that a majic number is
stashed on the end of the global buffers (buf, buf1, buf2, var).  This
requires only placing the magic number at the time that the memory is
initialized.  The CREATE and RECREATE macros could be expanded upon to
automatically allocate an extra byte or two of memory and then place the
magic at the end (or better yet, at the beginning).  This should provide
more than adequate protection against buffer overruns, but will not
provide great protection against random memory writes.

2.  Data-reliant magics.  The first one or two bytes of the allocated
memory could be stashed in a different area of memory and updated
whenever the memory is changed.  This would involve changing the way
that the variable is accessed.  This Also provides protection against
buffer overruns and still does not adequately protect against random
writes.

3.  Checksums.  Like Data-reliant magics, only a checksum of the entire
chunk of memory is stashed.  The implementation of this should not be
any harder than that of data-reliant magics, but it could slow down
performance significantly.  This would protect against buffer overruns
and random writes as either would alter the checksum of the affected
data.

4.  Mirroring.  By far the most rock solid method of ensuring the data
quality.  It is also the most costly.

>  You should not sync or close a memory
> map after a crash by catching SIGSEGV (you are not given any reasonable
> guarantees about the purity of the data and your process at this time).

It's the data we're concerned about, we'll reload the process.

<snip the rest of the post, good stuff, but I don't have a reply for it>

Regards, Peter

--
   +---------------------------------------------------------------+
   | FAQ: http://qsilver.queensu.ca/~fletchra/Circle/list-faq.html |
   | Archives: http://post.queensu.ca/listserv/wwwarch/circle.html |
   +---------------------------------------------------------------+



This archive was generated by hypermail 2b30 : 12/04/01 PST