Rick's b.log - entry 2025/10/23

A little more Tea

so this will be - I hope - the final "work in progress" release. The next one can have a normal version bump instead of a letter suffix.

Does that sound like tempting fate to you? Cthulhu thought it did, and in return struck me down with a program that crashed in the early hours.

It wasn't hard to track this down, and as always it was a one-line fix, a missing line. A rather important line, not that there are many lines in a computer program that aren't important...

Let's do a deep dive here. I'll keep it simple so non-nerdy people can follow along too.

When you click on a programme in the EPG, Tea will request the programme information from the server. If the programme has an associated picture, this will also be requested.
These files are put into a cache, so that the next time you click on that programme, the information is available instantly.

However, these are lots of small files with a limited validity. Thus it was decided that they should be automatically tidied by periodically looking at everything in the cache and getting rid of anything that is over twenty four hours old.

It's here that things blew up.

In order to work out what to actually do, I had to scan all the files in the cache five times.

The first time, I look for files beginning "i_" and count them. These are the programme images.
Then I look for files beginning "p_" and count them. These are the programme information files.

At this point I allocate some memory that is large enough to hold a reference to all of those files.

Then I go back and look again for all files beginning "i_", but this time instead of counting them I make a note of their filenames and add them to the memory that I just allocated myself.
This is repeated with all of the "p_" files.

The end result is a bit of memory that contains a list of all of the programme cache files, first the images and afterwards the programme data.

Now this list is looked at, every file one after the other, to get how old the files are (in hours). If the age is more than twenty four (hours) then the file is deleted.

Sounds simple, right?

Well, there's a little trick. When you ask for a list of files, there's a value that keeps track of where you are. You are supposed to set this to zero (begin at the beginning) and then you read a file. That system call will hand back an opaque value guaranteed not to be -1 that you hand over for the next file, and so on. When it is -1, this means there are no more files, you've read them all.

The problem was that for the second scan, I forgot to reset this to zero.

So let's look at what was going on...

I scanned the image files. Let's say there are fourteen of them.
I then scanned the programme information files. Let's say there are eighteen of them, but because I forgot to reset the opaque value, it only told me there were two.

I would then allocate some memory in order to store sixteen file references.
Then I would carefully read in the fourteen images, and then the eighteen programme files.

Now, thirty two of anything isn't going to fit into the space of sixteen of them. It doesn't matter if it's files in a bit of memory, or ceramic tea mugs in little boxes - sixteen is sixteen and thirty two isn't going to fit.

However since Tea is written in C, it won't stop and say "Dude! No!". It will instead dutifully keep on adding stuff to that bit of allocated memory, wandering right off the end of the street and ending up in the sorts of dark back alleys where nice people never go. Or in software terms, dumping those additional filenames over top of other bits of memory, other bits of memory that are important to the software, like perhaps the call stack (if you call a function that calls a function that... well when those functions end you need to know where to go back to). It doesn't really matter. What matters is that in extremely short order the program experienced an "OhShit!" moment.

This, obviously, should not have happened. ☺

A total aside: Secondly, that opaque value. The programmer's documentation makes it quite clear that one should not make any assumptions as to what the value is or what it means. All you need to know is that the filing system can use it to identify the next file in the list.
Traditional RISC OS filing systems (based on FileCore/FileSwitch) do indeed use sequential numbering, but then they shuffle the directory every time it is updated in order that everything is in nice alphabetical order.
Foreign filing systems, such as DOSFS, not only do not use sequential numbering but they also don't return files in alphabetical order, because they aren't held as such in the directory list.

Anyway, allow me to touch my wood-effect chipboard desk while I say that hopefully this will be the final v0.25 release. ;)

Oh, and the cache expiry is now also done as and when the program starts. This is so your cache will get tidied if you only run Tea when you want it, rather than just leaving it running.

Your comments:

Please note that while I check this page every so often, I am not able to control what users write; therefore I disclaim all liability for unpleasant and/or infringing and/or defamatory material. Undesired content will be removed as soon as it is noticed. By leaving a comment, you agree not to post material that is illegal or in bad taste, and you should be aware that the time and your IP address are both recorded, should it be necessary to find out who you are. Oh, and don't bother trying to inline HTML. I'm not that stupid! ☺
As of February 2025, commenting is no longer available to UK residents, following the implementation of the vague and overly broad Online Safety Act. You must tick the box below to verify that you are not a UK resident, and you expressly agree if you are in fact a UK resident that you will indemnify me (Richard Murray), as well as the person maintaining my site (Rob O'Donnell), the hosting providers, and so on. It's a shitty law, complain to your MP.
It's not that I don't want to hear from my British friends, it's because your country makes stupid laws.
 
You can now follow comment additions with the comment RSS feed. This is distinct from the b.log RSS feed, so you can subscribe to one or both as you wish.

Rob, 25th October 2025, 23:22

Why read through counting i_s, then again counting p_s? Read through once, incrementing the relevant counter depending on the filename you find. Ditto the storing of the names. You'd only need to run through twice.  If there's a way of getting the number of files in a folder without counting them, you could just use that as a potential maximum-possible for each list, and blow the extra memory usage.

Add a comment (v0.12) [help?] . . . try the comment feed!
		Your name
		Your email	(optional)
		Validation	Are you real? Please type 34791 backwards.
		UK resident	Check to verify that you are NOT a UK resident.
		Your comment