Questions about E.openFile

[MercurialPony]

Hi there! Currently working on a couple of apps for the TWC Pip-Boy. Since there's quite a tight limit on memory (and therefore on file reads/writes), I've had to move over to stream file IO to avoid memory fragmentation when accessing files. (seemingly related to this issue https://github.com/orgs/espruino/discussions/7114 - looks like it hasn't been fixed)

I've been wondering though:

1. Why is it that you can't E.openFile for both reading and writing at the same time? (normally r+ or w+ would give you that behavior, but seems like that's not the case here). So far I've had to resort to opening two files, with r and w+ modes, to get around this
2. Is it possible to fully overwrite a file using the E.openFile methods? i.e. the equivalent of fs.writeFile, without having to close the file every time (and run into memory issues)

[gfwilliams]

That filesystem issue should really have been fixed. Can you provide me with some code that causes the problem just to be sure? You do have to close files (afaik) when you're done so if you're not doing that you'd run out of memory.

Maybe you could catch the file open error and call E.defrag()? If that works potentially I could add that inside the firmware.

So what happens when you try to read a file opened with w+? I'd be a bit concerned that having two copies of the same file opened at once might cause problems (as changes to the writable one may not immediately appear on the readable one).

Under the hood we use a library called FatFS - it feels like read+write should work. Looking at https://github.com/espruino/Espruino/blob/master/libs/filesystem/jswrap_file.c#L311 and https://elm-chan.org/fsw/ff/doc/open.html it looks like it'd be trivial to implement a r+ which is probably what you needed? But obviously that means a new firmware.

Is it possible to fully overwrite a file using the E.openFile methods?

I don't think so, no - not on the same file.

[MercurialPony]

Can you provide me with some code that causes the problem just to be sure?

Sure, here's a very simplified example of what my app does:

const g_arr = [];

for (let i = 0; i < 400; ++i) {
    try {
        g_arr[i % 7] = JSON.parse(fs.readFile("TEST/test.json"));
    } catch(e) {
        console.log(i);
    }
}

Where TEST/test.json is a valid json about 300 characters in length.

1. I then check the current memory before running the code: free: 1669, usage: 3331, total: 5000
2. Running the code, fs.readFile starts occasionally returning undefined.
3. Re-running the loop multiple times increases the amount of errors, and at some point it seems like the file system completely refuses to open any new files, returning undefined for every single file read.
4. Memory after re-running the loop multiple times: free: 1334, usage: 3666, total: 5000. At no point does the app/device crash with an OOM.

It's worth noting that this only happens when more memory is in use. For instance this doesn't happen in the case when none of my app code is loaded free: 3681, usage: 1319, total: 5000.

Adding E.defrag(); after console.log(i); does indeed seem to help, with only a couple errors popping up during the whole loop, even with repeated re-runs. And there's no longer a case when the file system completely freezes up as well. But I think it also slows down the execution time? (not sure)

I've already restructured my app to use stream IO to keep a single larger file open for reading, instead of reading many smaller files, so this is no longer an issue (except for cases where I need to completely rewrite a file, but those are rarer and haven't caused any issues so far).

So what happens when you try to read a file opened with w+?

It seems to work as expected - any written changes appear in the readable file right away. I haven't encountered any issues so far.

it looks like it'd be trivial to implement a r+
I don't think so, no - not on the same file.

Thanks for letting me know! It'd be convenient if we could have these features implemented :>

[gfwilliams]

Great, thanks! I guess the issue is the readFile is allocating memory and by keeping it around we're ending up with these fragmented areas which is making it unable to find a contiguous area of the 700ish bytes it needs.

Adding E.defrag(); after console.log(i); does indeed seem to help ... But I think it also slows down the execution time? (not sure)

It shouldn't, as it'll only run when required. But if you reach a point where the memory is fragmented and every time it's having to defrag then I guess that could make it slower - but you'd hope after a defrag things would be a lot better for a while.

I've just added r+ and an auto-defrag but that won't make it into the Pip-boy for a while afaik.

[MercurialPony]

we're ending up with these fragmented areas which is making it unable to find a contiguous area of the 700ish bytes it needs

It's just strange to me that we're clearly not running out of memory (since we never have more than 7 objects in memory at once), but due to the supposed fragmentation, readFile isn't able to allocate enough memory even though process.memory() shows that we have 1334 blocks free after running over 1200 cycles!

you'd hope after a defrag things would be a lot better for a while.

From a few quick tests I observed defrags happening anywhere from 0 to 5 times per loop. And, I might be wrong, but I felt like the execution time of the loop was noticeably longer when more defrags happened. Also I do think this workaround does feel a little sketchy, because if you wanted to make sure that file is never missed, then you'd write something like:

function readWithRetry(path) {
    let file = fs.readFile(path);
    while (!file) {
        E.defrag();
        file = fs.readFile(path);
    }
    return file;
}

but couldn't that possibly cause an infinite loop if there IS enough memory to read the file, but it's not possible to defragment it enough?

I've just added r+ and an auto-defrag but that won't make it into the Pip-boy for a while afaik.

Thanks! That's no problem, they're mostly QOL features anyways. I'll be keeping an eye on when they're pushed out so I can update my app. (I only need r+ at this point)

Oh, and now that I think about it, fs.writeFile probably doesn't have the same limitations as fs.readFile (as in, it won't suddenly start failing after repeated calls), so truncating a file using stream IO probably isn't necessary, like what I was originally asking for.