Add a writefile_atomic() routine that is only used for writing complete
files, not appending or piping, and is atomic on a POSIX filesystem,
and have timed-build use that.

Reorder the locking in the classic writefile() to return the race
condition vulnerability window back to the much smaller time it was before
the introduction of UTF-8 support, to make it very unlikely to occur.

Detailed explanation of the problem and fix:

Recently I was surprised to run into a bug in Interchange's timed-build
generation routine on a client's production system.

First see the source of &Vend::Interpolate::timed_build and then look
at &Vend::File::writefile, especially this:

    open(MVLOGDATA, $file) or die "open\n";
    if ($encoding) {
        local $PerlIO::encoding::fallback = $fallback;
        binmode(MVLOGDATA, ":encoding($encoding)");
    }

    lockfile(\*MVLOGDATA, 1, 1) or die "lock\n";
    seek(MVLOGDATA, 0, 2) or die "seek\n";
    if(ref $data) {
        print(MVLOGDATA $$data) or die "write to\n";
    }
    else {
        print(MVLOGDATA $data) or die "write to\n";
    }
    unlockfile(\*MVLOGDATA) or die "unlock\n";

Note that $file will contain ">/path/to/timed_build_file" so it causes
the file to be opened in write mode and truncated.

That alone is a race condition: When a file is truncated, there's a
brief time before it's written to when it's empty and any other process
reading it will get an empty file.

The introduction of UTF-8 support has made that race window longer because
right after truncation the binmode() call is made with :encoding(UTF-8)
or whatever, which can induce dynamic Encode package loads, which is
relatively slow.

That is one bug, but the bug I encountered is different and much weirder:
Two timed build files generated by Interchange each had their contents
doubled.

That is, if the file should have contained "HORSES\n", it instead
contained "HORSES\nHORSES\n".

I've never seen such a thing before, but it happened in two separate
timed-build files that both get used right after each other in ITL,
so if it's caused by a race condition, it's believable that both would
happen in the same run because they're sequential, so the same thing
that caused it one place could cause it another.

Now let's walk through the above code as run by two concurrent
processes to see the race as it happens.

Process A runs this:

    open(MVLOGDATA, $file) or die "open\n";

And then process B runs it:

    open(MVLOGDATA, $file) or die "open\n";

Process A runs this:

    if ($encoding) {
        local $PerlIO::encoding::fallback = $fallback;
        binmode(MVLOGDATA, ":encoding($encoding)");
    }

Then process B runs it:

    if ($encoding) {
        local $PerlIO::encoding::fallback = $fallback;
        binmode(MVLOGDATA, ":encoding($encoding)");
    }

Process A runs this:

    lockfile(\*MVLOGDATA, 1, 1) or die "lock\n";

Then process B runs it and blocks on the lock:

    lockfile(\*MVLOGDATA, 1, 1) or die "lock\n";

Process A runs this:

    seek(MVLOGDATA, 0, 2) or die "seek\n";
    if(ref $data) {
        print(MVLOGDATA $$data) or die "write to\n";
    }
    else {
        print(MVLOGDATA $data) or die "write to\n";
    }
    unlockfile(\*MVLOGDATA) or die "unlock\n";

and the timed-build file is written correctly and unlocked.

That unblocks process B which now runs this:

    seek(MVLOGDATA, 0, 2) or die "seek\n";

which, oddly for this situation where we are writing to a truncated file,
seeks to the end of the file! That appears to be a recipe gotten from
the Perl flock() documentation:

    flock($fh, LOCK_EX) or die "Cannot lock mailbox - $!\n";
    # and, in case we're running on a very old UNIX
    # variant without the modern O_APPEND semantics...
    seek($fh, 0, SEEK_END) or die "Cannot seek - $!\n";

and would be harmless if the file is still truncated, but now that the
file has been written to, means we seek to the end of the file and start
writing there!

Now process B runs this:

    if(ref $data) {
        print(MVLOGDATA $$data) or die "write to\n";
    }
    else {
        print(MVLOGDATA $data) or die "write to\n";
    }
    unlockfile(\*MVLOGDATA) or die "unlock\n";

And we have two copies of the same timed-build output, one after the
other, in the same file.

This bug has always existed in this code, but I think it became many
times more likely to be encountered once the binmode() with encoding
was added because that can be so slow.

There are a few ways to try to solve the doubling problem:

1. Just remove the seek() call which appears to have been only to cope
with Unix systems that were already old and rare at the time the Perl
flock() docs were written. Then when the doubling problem hits in the
future, it'll write the same timed-build output once, then over itself
again.

If the timed-build runs return output of differing length, it could
cause the tail end of the longer one to appear after the shorter one,
which might be an even more confusing bug than this one.  But we could
call truncate() to empty the rest of the file and remove the risk of
the longer pieces remaining from before.

This isn't attractive, though, because I don't know for sure that removing
the seek() is actually safe, and on which systems.

2. Call the binmode() to set the encoding *after* taking the lock. This
seems like the right thing to do anyway, but it only narrows the window
for the race condition, and doesn't remove it entirely.

3. Overhaul the routine to use standard practice in writing a shared
file atomically.

It seems only (3) really solves both of the bugs. We can achieve our
goal with a procedure like this:

a. Open a new randomly-named temporary file in the same directory.

b. Write to it.

c. Close it.

d. Rename it over the original file.

Renaming is an atomic operation when done within a single POSIX
filesystem. It isn't atomic for reads over NFS, but writefile() didn't
support fcntl() locking for NFS anyway, so we're not making that problem
any worse.

This doesn't stop concurrent timed-build writes from happening, but they
would harmlessly replace each other in a way that would never leave an
empty or partially-written file, would keep concurrent reads from ever
getting a stale timed-build file, and doesn't need locking at all.

Thanks to Mark Johnson for review.