Acorn files on other platforms

Discuss all aspects of programming here. From 8-bit through to modern architectures.
User avatar
pau1ie
Posts: 512
Joined: Thu May 10, 2012 9:48 pm
Location: Bedford
Contact:

Re: Acorn files on other platforms

Post by pau1ie » Sat Apr 07, 2018 10:37 pm

davidb wrote:I'll upload my DFS-related scripts to the stardot GitHub organisation and we can take things from there. :)
Thanks David, I would appreciate that. The bbcmicro desktop application I was writing (Stalled at present) is written in python, so it would be nice to have some python utilities.

At the Cambridge Centre for Computing History last summer, a child sat down at my BBC Master and started typing in print statements that worked. It transpired that he was using python, which obviously stopped working as he started to use other language features! It seems popular with the kids!
I'm working on http://bbcmicro.co.uk

User avatar
davidb
Posts: 2070
Joined: Sun Nov 11, 2007 10:11 pm
Contact:

Re: Acorn files on other platforms

Post by davidb » Sat Apr 07, 2018 11:04 pm

I'm not sure about the eventual structure of the repository, so I've created the repository in my user area: Acorn-Format-Tools. This means that we can discuss what should be in there and I can rearrange/redo things before we create a repo under the stardot organisation.

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Sun Apr 08, 2018 1:25 am

For what it's worth, I'm currently pursuing the idea of emulated 6502 code in a restricted environment as a way to implement filing systems.

In the process, I'm trying to formulate a clean C++ emulator implementation based loosely on the one in PiTubeDirect, but taking advantage of generic coding techniques so it can be tuned efficiently. That should mean that, as well as running on desktop hardware, Raspberry Pi and similar, it could also squeeze into about 8K of RAM on a microcontroller (albeit rather more ROM). Which might be useful some day.

Let's see how this pans out...

User avatar
davidb
Posts: 2070
Joined: Sun Nov 11, 2007 10:11 pm
Contact:

Re: Acorn files on other platforms

Post by davidb » Sun Apr 08, 2018 11:40 am

crj wrote:For what it's worth, I'm currently pursuing the idea of emulated 6502 code in a restricted environment as a way to implement filing systems.
If you haven't already seen it, you might want to take a look at py65.
crj wrote:In the process, I'm trying to formulate a clean C++ emulator implementation based loosely on the one in PiTubeDirect, but taking advantage of generic coding techniques so it can be tuned efficiently. That should mean that, as well as running on desktop hardware, Raspberry Pi and similar, it could also squeeze into about 8K of RAM on a microcontroller (albeit rather more ROM). Which might be useful some day.
That would also be useful. I could imagine something like that running on an ATmega microcontroller.

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Sun Apr 08, 2018 10:30 pm

davidb wrote:If you haven't already seen it, you might want to take a look at py65.
That's one of the things I'm aware of, but confess I've not investigated thoroughly. I'm not certain it's an especially good fit for what I'm up to, though it could be (yet) another thing to look at when trying to determine correct behaviour in corner cases.
In the process, I'm trying to formulate a clean C++ emulator implementation based loosely on the one in PiTubeDirect, but taking advantage of generic coding techniques so it can be tuned efficiently.
That would also be useful. I could imagine something like that running on an ATmega microcontroller.
Indeed. ATmega, STM32, various Cortex-M0-based controllers embedded in WiFi/Bluetooth chipsets, and so on.

It doesn't solve anything like the whole problem, but it does feel like a useful thing to have.

User avatar
jgharston
Posts: 3055
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield
Contact:

Re: Acorn files on other platforms

Post by jgharston » Tue Apr 10, 2018 4:21 pm

crj wrote:Besides, if metadata was left as .INF files in the ZIP, the RISC OS client would have to deal with that. A reasonable client would likely hide the .INF files anyway as part of that handling!

Thinking about it, I'd argue that putting .INF files in the Zip file is actually a superior solution, because it involves one fewer standard and less translation. RISC OS has to translate between something and its native FS whatever you do; if you used INF-inside-ZIP then you could use a stock ZIP tool on the foreign OS.
No, 'cos ZIP tools for RISC OS already use the pre-existing 'Acorn' metadata field, and have done for almost 30 years, so you'd be breaking existing systems. Leaving the .inf files out of a ZIP file is involving one fewer standard, and the standard is already that Acorn metadata is stored in the Acorn metadata ZIP field. And dragging everthing out of a ZIP file then leaves you having to trawl the results and delete loads of INF files. And the ZIP standard is very strong on saying Thou Shalt Put Thy Metadata In The Metadata Field And Not In Additional File Entries. We've created the Metadata field for you to put stuff like this in, use it dammit! ;)

Also, the problem of extracting metadata from a ZiI Pfile that is not native to the platform you are unzipping in is use a tool specifically for getting the non-native metadata, for whatever non-native metadata it is. No stock UnZip tool can understand metadata that it doesn't understand, you use a tool built for that specific purpose, such as ZipToInf for extracting Acorn metadata as INF files, various tools for extracting NTFS file permissions on non-NTFS platforms, various tools for extracting Unix user settings on non-Unix systems, etc. Use the tool for the job.
crj wrote:
Character set is any 8-bit character as the pathnames are length specified.
Provided at least the more common tools which handle ZIP files support that, yes.
Loads of ZIP tools can only deal with 8-bit filename lengths even though it is a 16-bit field in the ZIP file. :(
crj wrote:
No, the attribute is defined as %-ewrLEWR. It's just the normal Acorn attribute, just as the load and exec addresses are just the normal Acorn load and exec addresses.
...unless the file came from an SJ Research fileserver and uses b2 for P rather than E? And the high-order attributes might be an Econet timestamp. But they are "undefined" under ADFS and presumably there are other filesystems which put useful stuff there? (ADFS really ought to have provided the sequence number!)
Most ADFSs store the top access bits, but just ignore them. The Acorn datestamp is stored in the pre-existing ZIP datestamp field. All ZIP tools I've come across ignore the datestamp in the Acorn attributes word and expect it to have been duplicated to the datastamp in the core ZIP datestamp metadata field. And you only see the MDFS P bit in bit 2 if you examine the object on RISC OS, it is invisible on other systems. I've debated with myself for years how to deal with that, and have ended up just ignoring it on the grounds that I have very few files on my MDFS with the P bit set that I ZIP up with RISC OS. ;)

I'd forgotten I'd already typed most of this up ages ago: Metadata.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Tue Apr 10, 2018 11:35 pm

This is all a confusing mess.

I'll start by making a few general observations, then get more specific.

Firstly, and most importantly, as a general principle an archiving tool's main job is to preserve as much information as possible. If, during the path from source filing system, through archiving, transport and restoration, to destination filing system, there must be any translation or discarding of information, that should occur in the dearchiving step, not the archiving one. That way, you're not stuck if you find a bug, change your mind, or want to use the data in a different environment.

Secondly, hosting files for one OS under another OS's filesystem presents an additional complexity: you have two sets of metadata: host and guest. When archiving such files:
  • (To the extent that they differ) both sets of metadata need to be preserved.
  • Only one subtree might contain guest-OS files, with the rest for the host OS.
  • There might be multiple guest subtrees for different guest-OSes.
  • There might be guest-of-a-guest nesting for three sets of metadata. Or worse.
  • It might not be apparent which files are intended for a guest OS. It might even be impossible to determine accurately.
Thirdly, several aspects of file metadata including hierarchy, naming, date and time representations, file typing, access permissions, forks/ADSes/user attributes, DSORG/sparseness, text encoding, order of files, volume, compression, etc. are each individually a monstrous can of worms. In terms of which precise data items exist, what range of values is possible and how they are represented.

Fourthly, there are many other kinds of filing system objects besides files: directories, image files, symbolic links, hard links, devices, pipes, sockets, mount points... which may need to be represented. The difficulties may be the same as for files, or they may be differently complicated.

Fifthly, not all filesystem metadata is associated with individual filesystem objects.

Sixthly, although things like SparkFS prove it can be done at a pinch, most archival formats are not suitable for use as a live filesystem. Significant problems include:
  • The lack of indexing structrures for better than O(n) access to anything
  • The difficulty and inefficiency of extending existing files
  • The difficulty of modifying data when it's been compressed without regard to that need
  • How to be robust in the face of failures during modifications
Seventhly, the world has moved on in past quarter of a century. On the kinds of data volumes we're discussing, seeking has become relatively quicker in comparison with reading and writing. Fragmentation is cheap; moving data is expensive.


I'm realistic enough to recognise that I'm not going to set the world to rights. And I'm aware of the problem that Myelin mentioned, that creating a shiny new format means ending up with N+1 competing formats where before there were only N. But I'm also uncomfortable that things feel somewhat messed-up at the moment.


So...
jgharston wrote:No, 'cos ZIP tools for RISC OS already use the pre-existing 'Acorn' metadata field, and have done for almost 30 years, so you'd be breaking existing systems. Leaving the .inf files out of a ZIP file is involving one fewer standard, and the standard is already that Acorn metadata is stored in the Acorn metadata ZIP field.
Suppose I have some files on an Archimedes. I wish to archive them, transport that archive to a Windows/MacOS/Linux box and unpack that archive. I want those unpacked files to be usable natively, and from a remote/virtualised/emulated RISC OS box. Then I want to archive them up again, transport them back to an Archimedes, unpack them and get back something equivalent to what I started with.

So far as I can tell, nothing approaching that is possible. The Archimedes ZIP tools work with the Acorn extra-data format, but I'm not aware of any ZIP tool for any other platform that honours them. I see your ZipToInf tool, but the documentation is sparse, there's no indication how robust it might be in various corner cases, the Windows version isn't open-source, the BBC BASIC version is impenetrable, there's no Linux/MacOS/portable version and there's no corresponding InfToZip tool.

For BBC Micros the situation is even worse. There's an UnZip tool, but no Zip tool.

Meanwhile, if you've managed to get a filesystem image from an 8-bit Acorn or Archimedes machine onto some other platform, I don't believe any of the tools for picking them apart support the Acorn extensions to ZIP.
the ZIP standard is very strong on saying Thou Shalt Put Thy Metadata In The Metadata Field And Not In Additional File Entries
For one specific OS, if generating files on that OS either for restoring on that OS or for specialist manipulation on other platforms, that's fair enough. But it's completely impractical to expect WinZip, say, to do something meaningful with Acorn extra data.
you use [...] various tools for extracting Unix user settings on non-Unix systems, etc.
A complicating factor is that ZIP is show-stoppingly deficient as a way of storing Unix files. Someone would have fixed it by now, except that everyone uses tar instead so they've never bothered.
The Acorn datestamp is stored in the pre-existing ZIP datestamp field.
Which Acorn datestamp is stored in which pre-existing ZIP datestamp field? There's the RISC OS 40-bit value in files with type+timestamp files, and there's sometimes the Econet datestamp (anyone know what happened when that wrapped in 1997? How about 2013?). What if one or the other is absent; what if they're inconsistent? On the ZIP side, there's the basic DOS mdate and mtime, but also the 0x5455 extended timestamp ctime/mtime/atime. Note that neither is of sufficient resolution to hold an Acorn centisecond-accurate timestamp. Also, Acorn timestamps are local time where POSIX ones are UTC; what mapping do you use? If given load+exec attributes for an Acorn file, how do you distinguish a type+timestamp from a legacy 8-bit host processor address when deciding whether or not to impart the Acorn timestamp to the ZIP file? What timestamp do you use in the ZIP file if there is no timestamp at all on the Acorn side, and how do you annotate that this is what has happened?

And then there's the question of what to do with timestamps when unpacking the ZIP file. On various platforms...

User avatar
Elminster
Posts: 2021
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Acorn files on other platforms

Post by Elminster » Wed Apr 11, 2018 10:16 am

pau1ie wrote:
At the Cambridge Centre for Computing History last summer, a child sat down at my BBC Master and started typing in print statements that worked. It transpired that he was using python, which obviously stopped working as he started to use other language features! It seems popular with the kids!
In some ways Python is probably the way to go if we want future people to take over the Retro scene, the people who are to come after us.

If you look at any resource for computing in schools it will basically be scratch and then Python.

e.g.
https://coderdojo.com/resources/
https://www.codeclub.org.uk/projects
https://curriculum.raspberrypi.org/

So we might all need to learn Python so that anyone (who is currently) under 18 understands us. For the heavy lifting coding everyone seems to be being go'ing now (on my to list to learn).

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Wed Apr 11, 2018 1:14 pm

I agree that Python is a pretty good choice for anything amenable to being coded in it.

But, although MicroPython is available for embedded systems, I'm not sure it's up to running a 6502 emulation at a sensible speed with small RAM footprint.

Meanwhile, C has been around for almost half a century and isn't going anywhere soon, given how much important stuff is written in it.

Post Reply