Acorn files on other platforms

Discuss all aspects of programming here. From 8-bit through to modern architectures.
crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Acorn files on other platforms

Post by crj » Fri Mar 30, 2018 10:03 pm

I'd been thinking about this anyway, without reaching any firm conclusions. However, this thread has motivated me to think out loud for a bit and see how other people feel.

I'm liking the idea of basing something around sqlite. Specifically, I'm liking the idea of interoperating with SQLAR which is a trivial archive format layered on top of sqlite by the authors of sqlite themselves.

SQLite is well known for its portability, speed, compactness and robustness. Now, by retrocomputing standards "compactness" is a relative term - we're talking half a megabyte or so of object code, but for modern platforms that's negligible.

One way sqlite wins over things like TAR and ZIP is the versatility of being able to augment the data structure with almost anything else you care to add - whether in new tables, or as extra columns in an existing table. Another is that it has good support for changing the content rather than merely creating and reading archives.

Also, the sqlite flavour of SQL is a standardised and mature textual representation for data.

I can see a few interesting ways such techniques could be used:
  • Create an sqlar file as a repository for an entire hierarchy of Acorn files with all their metadata
  • The same, but also include a file server's user database, or similar stuff.
  • Create an sqlar file as an metadata-and-optional-compression wrapper around a single Acorn file.
  • Create an sqlite database under a standardised (probably hidden) name in a directory, to store metadata for that directory's files
  • Traverse the directory hierarchy upwards until you find such a file, which can store metadata for an entire subtree
  • Associate a single textual SQL statement with a file, either by putting it in an adjoining file or by putting it in an extended attribute
As well as what .INF files provide, an sqlite database could optionally contain all sorts of other goodies such as:
  • Disc titles
  • *OPT4 settings
  • Information about where files came from (DFS, ADFS, Econet fileserver, Linux, etc.) and sub-categorisations (Atom v. Beeb DFS, SJ Research v. Acorn fileserver, Linux foo.c => foo/c v. foo.c => c.foo, etc.) to allow more informed strategies for dealing with the other metadata. For example, being aware that the middle-order bits of a load address are inferred for a DFS file, or that bit 3 of the attributes means Private not Locked on an SJ Research fileserver.
  • The capacity to represent RO3+ image files as either the binary object or the unpacked directory structure they contain.
While thinking about this, I realised the world has moved to Unicode and, although top-bit-clear characters map back and forth relatively seamlessly, it's hard to know how to map between Unicode characters and top-bit-set Acorn ones. It occurs to me that a reasonable convention would be to map between &80 to &FF and, say, U+EA80 to U+EAFF in the Private Use Area. That would avoid the risk of damage to Acorn filenames, while preserving collation order.


I also realised one minor wrinkle in SQLAR: for each file it has a blob of data and a size field. If the size is greater than the length of the blob, it is assumed that the file is compressed. However, it actually assumes this if the size is unequal. It would be good if it were possible to represent a file by an uncompressed blob that is longer than the file, to leave extra extent for in-place expansion. Fixing this was a trivial one-line change to the SQLAR source code, but I'd probably want to get that change accepted upstream before letting the practice out into the wild.


So: good idea? Sucky idea? Worth thinking about some more?

User avatar
myelin
Posts: 460
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Acorn files on other platforms

Post by myelin » Fri Mar 30, 2018 11:50 pm

Feels like the sort of thing that could be awesome, but also could result in a "there are 15 competing standards" situation :)

Are you thinking of this as an exchange format? Something for embedded systems to understand?

How stable is the SQLite/SQLAR format? Is there a risk that a new version of SQLite will be unable to read existing SQLAR files?

An interesting exercise would be to document all the existing exchange/usage formats (disk images, .MMB files, plain files + associated .inf metadata files, ...) because they presumably represent what developers have considered necessary so far. Then think about the pain points currently being experienced.

Personally, annoyances include: interchange between different formats (I couldn't tell you off the top of my head how to extract files from a .ssd or an ADFS image, unless the 'beeb' tool for .MMB files can do this directly); platform-specific editors (Windows-only .MMB editors)
SW/EE from New Zealand, now in Mountain View, CA, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

User avatar
davidb
Posts: 2231
Joined: Sun Nov 11, 2007 10:11 pm
Contact:

Re: Acorn files on other platforms

Post by davidb » Sat Mar 31, 2018 5:05 pm

One place to look for (and write up) informationas part of a preliminary information gathering exercise is the Just Solve the File Format Problem site. There are, of course, other Wikis out there that gather this sort of information, including BeebWiki for Acorn 8-bit file formats.

I started to write up various Acorn 8-bit game data structures and level formats in the Retro Software Wiki. That's not very relevant for general interoperability, but the tools that process those formats could have been written to rely on a general conversion tool rather than a collection of libraries. I'm not sure it would have made it so much easier but it's another use case to bear in mind.

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Sun Apr 01, 2018 3:00 am

myelin wrote:Feels like the sort of thing that could be awesome, but also could result in a "there are 15 competing standards" situation :)
Indeed. I'm aware of that risk. Hence my reluctance to plough in and do something if there's not a useful amount of buy-in by anyone else.
Are you thinking of this as an exchange format? Something for embedded systems to understand?
Not for small embedded systems. The smallest system that could cope would be an OpenWRT box or a Raspberry Pi.

I'm also contemplating filing systems for more deeply embedded systems, but that would be a rather more vertical and less portable thing. (Well, unless I put some effort into making that more generally useful, which isn't an awful idea now I saytype it out loud.)
How stable is the SQLite/SQLAR format? Is there a risk that a new version of SQLite will be unable to read existing SQLAR files?
The stability of SQLite's format is exemplary, amongst the best. I've been using it professionally for over a decade in multiple jobs on everything from embedded to enterprise, and they've not broken forward or backward compatability in that time (provided, obviously, that you don't actually use newer features).

SQLAR is a more open question. However, note that SQLite does most of the hard work and the "Storage" section in the webpage I linked to is a complete specification, despite being just a few short paragraphs. That's relatively confidence-inspiring.

What's more, provided you have SQLite, the tools for reading SQLAR aren't strictly necessary:

Code: Select all

-- Print a shell script that creates directories
SELECT printf("mkdir %s",name) FROM sqlar WHERE data IS NULL;
-- Print a shell script that creates symlinks
SELECT printf("ln -s %s %s", data, name) FROM sqlar WHERE sz<0;
-- Unpack the contents of all the files
SELECT writefile(name,data) FROM sqlar WHERE data IS NOT NULL AND sz>=0;
-- Produce a list of the files that need zlib-inflation
SELECT name FROM sqlar WHERE length(blob)<sz;
-- Produce a shell script to fix up permissions and mtimes
SELECT printf("touch -m --date=@%i %s; chmod %o %s", mtime, name, mode, name) FROM sqlar WHERE sz>=0;
Honestly, I'd have preferred it if they'd chosen a cleaner technique for denoting directories and symlinks - a single-character type field, for example. But even so, it's pretty clear that the data wouldn't become unusable for so long as sqlite continued to thrive.
An interesting exercise would be to document all the existing exchange/usage formats (disk images, .MMB files, plain files + associated .inf metadata files, ...) because they presumably represent what developers have considered necessary so far. Then think about the pain points currently being experienced.
By my current understanding:
Feature/limitDFS .ssdVintage ADFSModern ADFS.MMBFile, .INF(Augmented) SQLAR
Overall size200K512MB<<256GB?128MBn/a128TB
File size200K512MB4GB?128MBOS1GB
Filename charset7-bit8-bit restricted8-bit restricted7-bit8-bit no spacesUnicode
Leafname length710256?7OS/undefined1GB
Hierarchy depth2256 chars?3OS, 1
Attribute bits166?13232+
Attribute meaningsLWREL/wrWREL/wr ?Lunspecifiedflexible
Load/exec addr bits183232183232+
Dynamic sizeNoNoNoNoOS(probably!)Yes
Augmentable data structureNoNoNoNoIshYes
File compression optionNoNoNoNoNoYes
ACID?NoNoNoNoOS, noYes
Portable implementationYesNoNoYesYesYes
Open-sourceYesNoYesYesYesYes
LightweightYesYes?YesYesMedium

User avatar
BigEd
Posts: 2139
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Acorn files on other platforms

Post by BigEd » Sun Apr 01, 2018 9:10 am

I don't think binary is a good idea for an archival format. Dare I say it, Microsoft's "new" office format has some very good ideas: it's a zip archive, and (at least) one of the files inside is in XML. It seems to me XML is an ideal self-descriptive format for metadata: names, timestamps, protections and so on. File payloads themselves should be raw bytestreams, so long as that's what a file is (no resource forks.) Zip is an old and stable compressed archive format with very wide support.

Is there a danger of confusing different kinds of things though? There are files, there are filesystem images, and there are lower level media images. I think each of these needs some different handling. The lowest level disk or tape image is useful for preserving the intricacies of a copy-protected work, but most of the time a higher level description is much more convenient and entirely appropriate.

User avatar
pau1ie
Posts: 586
Joined: Thu May 10, 2012 9:48 pm
Location: Bedford
Contact:

Re: Acorn files on other platforms

Post by pau1ie » Sun Apr 01, 2018 11:01 am

As a database administrator I approve of your using a database format!

BigEd has a good point here. What problem are you trying to solve? While .ssd and mmb files have limitations, the great advantage is that they can be read by a real bbc micro. The disadvantage with a more flexible format is that it will need processing into a simpler format that can be read by the micro.

From your other posts it seems that you are thinking about archiving, and storing metadata with the discs. Your suggestions would be useful for that, but it would be a lot of work to create the formats, the programs to be able to convert them, and to create and input the data.
I'm working on http://bbcmicro.co.uk

Coeus
Posts: 1077
Joined: Mon Jul 25, 2016 11:05 am
Contact:

Re: Acorn files on other platforms

Post by Coeus » Sun Apr 01, 2018 11:55 am

It seems to me there are plenty of cases in technology where the first defacto standard established tends to prevail even if hindsight shows weaknesses and a redesign would be better.

It seems to me that SSD is just such a defacto standard for distributing Acorn software whether published or simply a collection of files such as a test case for an emulator bug. It is well known that it is not sufficient for much copy protected software but it also lacks embedded metadata - it would be good if a class of similar disc images files contained information on the disc geometry, the filing system used as well as title, etc. For archiving it would be good to include extra information such as when, who, where the source media came from etc. For all that, it prevails because it has good support from emulators and can be used on real hardware via the SD/MMC options or written to floppy either on a suitability equipped PC or via UPURS.

So I think one could invent something better but it would be a struggle to get it widely adopted.

When it comes to storing individual Acorn files on another OS's filesystem the .inf file seems to be the defacto standard. It is not ideal - the ideal would be store the Acorn-specific metadata in such a way that it always stays with the file but does not impede the use of that file from the OS on whose filesystem it is stored. Multi-fork files would be a good solution but are not supported everywhere. Something that either puts the Acorn metadata into a hidden file, or collects it together in a single file for a whole directory, or both would not, in my opinion, be an improvement because it means using the file management commands/ui on the host OS now risks leaving the metadata behind when moving files around.

User avatar
1024MAK
Posts: 7988
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...
Contact:

Re: Acorn files on other platforms

Post by 1024MAK » Sun Apr 01, 2018 5:57 pm

Coeus wrote:It seems to me there are plenty of cases in technology where the first defacto standard established tends to prevail even if hindsight shows weaknesses and a redesign would be better.
Yep, very common for a 'popular' system, standard or specification to prevail long after it could or should have been replaced/updated. However, the reason(s) it became popular vary. Sometimes it is because it was the least expensive, sometimes because it was the easiest to support, plus various other reasons.

As to where we go from here, I'm not sure. But the point that it will be hard to displace the existing systems is a very valid one.

Mark

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Mon Apr 02, 2018 12:15 am

BigEd wrote:I don't think binary is a good idea for an archival format. Dare I say it, Microsoft's "new" office format has some very good ideas: it's a zip archive, and (at least) one of the files inside is in XML.
Hmm. That seems self-contradictory. ZIP is every bit as much a binary file format as sqlite!
It seems to me XML is an ideal self-descriptive format for metadata: names, timestamps, protections and so on.
Ugh. XML isn't extensible and is barely a language. Worse, although intended for markup, it's very seldom used for that, and this is yet another example of a potential misapplication.

Given the other advantages SQLite would bring, SQL seems fine for such purposes. If it was deemed unacceptable, plan B would be JSON not XML.
File payloads themselves should be raw bytestreams, so long as that's what a file is (no resource forks.) Zip is an old and stable compressed archive format with very wide support.
This seems like another self-contradiction. Zip usually stores files compressed rather than as raw bytestreams, so is every bit as much a compresed file format as SQLAR-with-zlib.
Is there a danger of confusing different kinds of things though? There are files, there are filesystem images, and there are lower level media images. I think each of these needs some different handling. The lowest level disk or tape image is useful for preserving the intricacies of a copy-protected work, but most of the time a higher level description is much more convenient and entirely appropriate.
There's a definite place for low-level disc/tape images, and for filesystem images. I'm not talking about supplanting those.

However, people also store Acorn files elsewhere and want to preserve the metadata. At the moment, putting a .INF file alongside each is the standard for that, but INF feels a little restricted in its representation and it seems I'm not alone in finding a heap of .INF files all over the directory quite clunky. So I'm wondering if it's possible to find an alternative that's better and that people like enough that it might get traction.

Hence the list of potential ways to use SQLAR in the original posting. Supporting all of them would be possible - and there would be significant commonality between them - but it's better to provide less choice and a more packaged solution if some of the options can be discounted.

It is possible to use filesystem images as an archive format. That does have the big advantage that the file can be dropped straight onto an Acorn machine and used as-is, and that's not nothing. But the downside is that it's not especially well suited to being manipulated as an archive, as my table above demonstrates. The most common choice seems to be .ssd files, but those are especially limited in what they can represent (notably, they'd be hopeless for RISC OS stuff). ADFS images are far less limited, but there don't seem to be any portable open-source implementations. So, swings and roundabouts.

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Mon Apr 02, 2018 12:41 am

pau1ie wrote:BigEd has a good point here. What problem are you trying to solve? While .ssd and mmb files have limitations, the great advantage is that they can be read by a real bbc micro.
Yes, although not a real RISC OS box...

I mean, a .ssd file is great as a way of distributing a video game. But am I unusual here? I migrated from DFS to ADFS and/or Econet fileservers as soon as I could, enjoying the larger volumes, longer filenames, larger number of files and hierarchical directory structure.

If ADFS disc images were in common use in the same way as DFS disc images, that might be a sensible option. Though even then dividing discs up into 640k lumps is an artificial restriction borne out of media capacities, not the way one would choose to organise things.
The disadvantage with a more flexible format is that it will need processing into a simpler format that can be read by the micro.
Indeed. But unless you're willing to put up with DFS's limitations, it seems like this is a wheel that needs inventing rather than reinventing?
it would be a lot of work to create the formats, the programs to be able to convert them, and to create and input the data.
Perhaps some motivating examples would help.

BeebAsm focuses on outputting a .ssd rather than a bare executable. That feels very much like two tools have been combined into one: I'd like to be able to assemble some executables, save some BASIC, write a bit of *EXEC, storing them all in some format useful to modern platforms and then finally turn them into a .ssd . (Even if we're fine with .ssd being used as the interchange format for such tools, there's no "libdfsssd" that people can use from their programs.)

Meanwhile, someone's writing an Arduino-based Econet fileserver. There is no good implementation of an Acorn filesystem they can use to underpin it. If you tried to write a "dfs_ssd to fileserver" tool, you'd be sunk because DFS doesn't have enough metadata or hierarchy.

(Unfortunately, for the second time in replying to comments on this thread, I have the uncomfortable sensation that I'm actually gradually talking myself into implementing something more custom and lightweight, because being able to run on something like an Arduino may be more important than ACID and the off-the-shelf support that sqlite provides. Hmm.)

User avatar
BigEd
Posts: 2139
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Acorn files on other platforms

Post by BigEd » Mon Apr 02, 2018 8:02 am

Indeed, I didn't express myself entirely clearly there. I quite like the combination of plain formats collected inside a compressed container format. Of course, from my background, I'd use a plain container format (tar) and then compress it (gzip, probably) but I can see that using zip is much more widely usable.

Likewise, I like the idea of combining the plain file data as elements in the container, with a plain text representation of the metadata next to them. XML is very much not my favourite technology, but seems a very practical one for self-documenting data exchange. JSON would be a close second, but in twenty years time I'd expect XML still to look the same whereas JSON might start to look different. Much like zip staying the same, but gzip possibly giving way to bzip or zx or whatever. It's best to be conservative when designing an archive format. After JSON, maybe Microsoft style INF, and after that, homebrew line-based text like the INF files we use today.

In passing, I rather liked the Interchange File Format (IFF), with tagged and sized chunks, as seen on the Amiga. The only wrinkle there is in the multi-byte size fields changing their spots from big-endian on the Amiga to little-endian as later adopted by IBM and Microsoft. It's easy for a reader to skip chunks which aren't known, and easy to nest IFF chunks within chunks. However, it's not self-documenting.

I've nothing against SQL or SQLite, but would never choose them for an archive or data exchange format. I'm not sure I can easily explain why.

User avatar
myelin
Posts: 460
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Acorn files on other platforms

Post by myelin » Mon Apr 02, 2018 6:39 pm

My guess is that SQLite, despite being fairly lightweight, still feels like a fairly big hammer for the seemingly-small problem of associating metadata with files. It's a great tool, but it's not something everyone is familiar with.

Devil's advocate here: crj, is the current approach of file + file.INF really that bad? It's pretty self-documenting and is compatible with every tool in the Acorn ecosystem. Zip up a folder of files like that and you have something that can be auto-mounted and searched on a bunch of operating systems without any extra tooling, and happens to already be supported by pretty much every tool around.

The only thing I don't like about the file + file.INF approach is that you can't load a file by name without scanning the entire directory (because the .INF can override the name, if you have a name that's not compatible with the external filesystem). On a system capable of running SQLite, though, scanning a directory (especially with a file count compatible with DFS/ADFS) is a tiny amount of work.
SW/EE from New Zealand, now in Mountain View, CA, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

User avatar
jgharston
Posts: 3243
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield
Contact:

Re: Acorn files on other platforms

Post by jgharston » Mon Apr 02, 2018 10:20 pm

crj wrote:By my current understanding:
(table)
Not quite right.
DFS uses 18-bit disk sizes, so multiplied by the sector size, the filesystem limit is 256K, not 200K. 200K is the maximum you can fit on a physical media formatted to 80 tracks. Many disk/drive combinations can format up to 82 tracks, so even on the physical media you can get up to 205K.
DFS uses 18-bit file sizes, so again, the file size limit is 256K, not 200K. (Actually, minus 1 as the maximum size is &3FFFF.)

Attribute meanings for ADFS is LWRe/wre. Other than 'L' for anything that is an owner attribute there is a corresponding public attribute. Many people never encounter the E access bit, and some implementations of ADFS even just ignore it completely.

(gods, that table must have taken ages to enter!)

Edit: and another entry:
ZIP file (compressed or uncompressed)
File metadata stored in the standard &4341 'Acorn' field, see link.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Tue Apr 03, 2018 12:39 am

Hmm. What tools support a 256Kbyte DFS image, though? It won't fit on a disc; can you make a .ssd file that big and have it work?

The extension to ZIP is good to know about; I assumed ZIP files just tended to contain a .INF file alongside the actual file.

For completeness, it looks like ZIP comes in normal and ZIP64 variants. Normal variant is limited to 4GB overall ZIP file size, 4GB contained file, 64Kbyte pathname (it seems hierarchy tends to be delimited using /, de facto; unsure what character sets are widely acceptable), 32-bit load/exec/attrs, undefined attribute meaning. Dynamically extensible, but hard to resize existing files. Compression, portable, open-source, reasonably lightweight, not ACID. ZIP64 extends the 4GB limits to 16EB, without changing anything else.

User avatar
jgharston
Posts: 3243
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield
Contact:

Re: Acorn files on other platforms

Post by jgharston » Tue Apr 03, 2018 3:27 am

crj wrote:Hmm. What tools support a 256Kbyte DFS image, though? It won't fit on a disc; can you make a .ssd file that big and have it work?
Yes*. MkImg will make a disk image up to the maximum possible that the filesystem supports. And you can make any sized blank image manually with
ch%=OPENOUT(filename$): PTR#ch%=&105: BPUT#ch%,size%DIV256: BPUT#ch%,size%AND255
PTR#ch%=size%*256:REM if you want to pad the image
CLOSE#ch%
.
*actually, it will only do up to 255K as it uses integer K sizes, which gives a disk size of %1111111100 sectors, 256K overflows into %0000000000.
crj wrote:The extension to ZIP is good to know about; I assumed ZIP files just tended to contain a .INF file alongside the actual file.
Have you never opened a ZIP file on RISC OS? Never noticed the absense of INF files and the presence of Acorn metadata?
crj wrote:(it seems hierarchy tends to be delimited using /, de facto; unsure what character sets are widely acceptable)
Directory seperators in ZIP files are specified as '/', not tends to be '/'. Character set is any 8-bit character as the pathnames are length specified. There's an extension for 16-bit Unicode pathnames. ZIP pathnames are "unixy" and naturally, translations must be performed going between ZIP files and file systems, for BBC pathnames this is the standard: . <-> / , # <-> ? , $ <-> < , ^ <-> > , @ <-> = (see link). There are functions to do this translation on the WIki, those pages are rather untidy and need rewriting, the FNfn_zip() function in Filename is better written.
crj wrote:undefined attribute meaning
No, the attribute is defined as %-ewrLEWR. It's just the normal Acorn attribute, just as the load and exec addresses are just the normal Acorn load and exec addresses.

See also:
* ZIP file specification
* Extra Field specification

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
jgharston
Posts: 3243
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield
Contact:

Re: Acorn files on other platforms

Post by jgharston » Tue Apr 03, 2018 4:39 am

I couldn't get to sleep, so I've been rewriting the filename/pathname pages on the Wiki.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Tue Apr 03, 2018 4:39 pm

jgharston wrote:Have you never opened a ZIP file on RISC OS?
No, never! Back in the day I used NFS to my RISCiX server, and now my heart is in 8-bit retrocomputing. The reason I'm putting thought into RISC OS as well is that I recognise any solution has to work for both to enjoy widespread acceptance.
Never noticed the absense of INF files and the presence of Acorn metadata?
Besides, if metadata was left as .INF files in the ZIP, the RISC OS client would have to deal with that. A reasonable client would likely hide the .INF files anyway as part of that handling!

Thinking about it, I'd argue that putting .INF files in the Zip file is actually a superior solution, because it involves one fewer standard and less translation. RISC OS has to translate between something and its native FS whatever you do; if you used INF-inside-ZIP then you could use a stock ZIP tool on the foreign OS.
Character set is any 8-bit character as the pathnames are length specified.
Provided at least the more common tools which handle ZIP files support that, yes.
No, the attribute is defined as %-ewrLEWR. It's just the normal Acorn attribute, just as the load and exec addresses are just the normal Acorn load and exec addresses.
...unless the file came from an SJ Research fileserver and uses b2 for P rather than E? And the high-order attributes might be an Econet timestamp. But they are "undefined" under ADFS and presumably there are other filesystems which put useful stuff there? (ADFS really ought to have provided the sequence number!)

User avatar
kieranhj
Posts: 724
Joined: Sat Sep 19, 2015 10:11 pm
Location: Farnham, Surrey, UK
Contact:

Re: Acorn files on other platforms

Post by kieranhj » Tue Apr 03, 2018 7:09 pm

crj wrote: Perhaps some motivating examples would help.

BeebAsm focuses on outputting a .ssd rather than a bare executable. That feels very much like two tools have been combined into one: I'd like to be able to assemble some executables, save some BASIC, write a bit of *EXEC, storing them all in some format useful to modern platforms and then finally turn them into a .ssd . (Even if we're fine with .ssd being used as the interchange format for such tools, there's no "libdfsssd" that people can use from their programs.)
Hey crj, I feel for you, having clearly opened a can of worms here! But I absolutely commend the effort to bring some better and more modern tools to the community, as the ones we have at the moment are limited and/or not being maintained any longer.

<2p>I chose ssd for POP precisely because of the wide spread adoption and portability, even spending a considerable amount of time squeezing it into an ssd rather than leaving it as a dsd (although that was potentially a mistake, but that's another story.)

I agree that the multiple .inf file approach is ugly. I have the opposite annoyance that I hate that the name in the inf can override the filename on the host machine. Previously I used bbcim to automate the creation of disc images for our productions but it doesn't quite do everything I need and then you've got to automate creating .inf files as well, urgh.

For POP I used BeebAsm like a traditional assembler and saved the object files to a folder without creating an ssd in the first pass, this is because everything was then compressed using pucrunch for a second pass. I then used BeebAsm again to create the disc image by using a source .asm file that just contained PUTFILE directives (it warns there's no SAVE command but works fine.) This did lose the file information (like exec address) so just had to hard code this, as it were. Even then, I still had to use a hand-made "template" disc image so I could set things like the title and write iterations etc.

One of my ideal requirements would be the ability to place files at a specific track / sector location within a disc image so I can optimise loading at run time, but I appreciate this is probably quite specialised. Being able to run any tool on the command line for scripting is kind of a must though. I can't help feeling that having human readable / editable metadata is kinda useful as well.

I almost started making my own disc image tool but had enough work to do without that. I imagined something a bit like (or exactly like) the *INFO output as a disc descriptor, so just one file for (disc) metadata rather than a .inf for each file. This is all very DFS focused though and doesn't answer many other people's requirements. I was surprised how many people have asked me for an ADFS version of POP since release.</2p>
Bitshifters Collective | Retro Code & Demos for BBC Micro & Acorn computers | https://bitshifters.github.io/

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Wed Apr 04, 2018 12:00 am

kieranhj wrote:I hate that the name in the inf can override the filename on the host machine
My instinct is that it's fine that it can override the host filename. Given the alternative is being sure that you have a reversible mapping between host filename and Acorn filename - a pretty tall order given how many different filesystems are out there!

The problem, perhaps, with .INF is that it does override the host filename. Having the option, maybe even the default, of using the host's filename, would make sense.

A related problem is that if you rename the host file without renaming the corresponding .INF, that dissociates the two irrecoverably.
One of my ideal requirements would be the ability to place files at a specific track / sector location within a disc image so I can optimise loading at run time, but I appreciate this is probably quite specialised.
Hmm. Specialised, as you say!

I can't see that happening from a command line, but in a programming language it would be very easy to provide an optional callout that takes the address where one file ends and returns the address where the next one should begin.
Being able to run any tool on the command line for scripting is kind of a must though
Does it need to be on the command line, or would a command line do?

My Python is a bit rusty, but I'm thinking that it would be an obvious contender as a language in which to write a modern DFS .ssd manipulator. And Python does provide a command prompt, as well as scripting capability.

What OS do you use for development, out of interest?

User avatar
hoglet
Posts: 7591
Joined: Sat Oct 13, 2012 6:21 pm
Location: Bristol
Contact:

Re: Acorn files on other platforms

Post by hoglet » Wed Apr 04, 2018 6:55 am

kieranhj wrote: For POP I used BeebAsm like a traditional assembler and saved the object files to a folder without creating an ssd in the first pass, this is because everything was then compressed using pucrunch for a second pass. I then used BeebAsm again to create the disc image by using a source .asm file that just contained PUTFILE directives (it warns there's no SAVE command but works fine.) This did lose the file information (like exec address) so just had to hard code this, as it were. Even then, I still had to use a hand-made "template" disc image so I could set things like the title and write iterations etc.
Are you aware of Stephen's mmb_utils?
https://sweh.spuddy.org/Beeb/mmb_utils.html
https://github.com/sweharris/MMB_Utils

They are written in perl, but run happily on Windows.

I use these for the packaging phase in all of my projects.

They are also small enough to be checked in to git, which removes a additional build time dependency.
kieranhj wrote: One of my ideal requirements would be the ability to place files at a specific track / sector location within a disc image so I can optimise loading at run time, but I appreciate this is probably quite specialised. Being able to run any tool on the command line for scripting is kind of a must though. I can't help feeling that having human readable / editable metadata is kinda useful as well.
With mmb_utils I think you get control of the order of the files if you add them one at a time.

Dave

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Wed Apr 04, 2018 6:05 pm

hoglet wrote:Are you aware of Stephen's mmb_utils?
[...]
They are written in perl, but run happily on Windows.
I use these for the packaging phase in all of my projects.
I don't know about Kieran, but I've glanced at them.

While they may solve the specific, narrow problem of shoehorning some specific files into a .ssd, Perl is not an ideal language choice. Partly, it's text-oriented and what's being dealt with is far from textual, so other languages are a better fit which will lead to more literate code. Partly, Python has way overtaken Perl as a language for scripting random stuff, so an implementation in Python is easier to bolt back to back with other things.

My choice for implementing anything would, therefore, be C, C++ or Python. And there aren't many environments in which the extra burden of the C++ runtime (as opposed to the standard library) is prohibitive.

Glancing at mmb_utils, they seem to have a few limitations:
  • .INF files are the only way to get metadata into or out of the .ssd
  • The .INF format it uses seems slightly peculiar ("locked", "CRC=", 24-bit load/exec addresses)
  • Files on host OS disc are the only way to get file data into or out of the .ssd (you can't "echo '*BASIC\nCHAIN \"\" ' | beeb putfile mydisc.ssd -" or whatever in your shell script)
  • Only supports 80t discs
  • No random-access primitives
With mmb_utils I think you get control of the order of the files if you add them one at a time.
It sounds like Kieran is trying to get more clever than that, optimising for the disc spin rate between finishing loading one file and starting to load the next! Though I'm not sure why he cares, given anyone who wants fast loading won't be using a real floppy these days. (-8

User avatar
myelin
Posts: 460
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Acorn files on other platforms

Post by myelin » Wed Apr 04, 2018 7:21 pm

crj wrote:Glancing at mmb_utils, they seem to have a few limitations:
  • .INF files are the only way to get metadata into or out of the .ssd
  • The .INF format it uses seems slightly peculiar ("locked", "CRC=", 24-bit load/exec addresses)
  • Files on host OS disc are the only way to get file data into or out of the .ssd (you can't "echo '*BASIC\nCHAIN \"\" ' | beeb putfile mydisc.ssd -" or whatever in your shell script)
  • Only supports 80t discs
  • No random-access primitives
None of this seems awfully hard to fix; you're right that Perl isn't the easiest language to work with, but adding these features to the existing, working code is still going to be much easier than rewriting the whole thing, and BeebUtils.pm seems readable enough.
SW/EE from New Zealand, now in Mountain View, CA, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Wed Apr 04, 2018 10:13 pm

myelin wrote:None of this seems awfully hard to fix; you're right that Perl isn't the easiest language to work with, but adding these features to the existing, working code is still going to be much easier than rewriting the whole thing
I'm not so sure of that. It's a 1,000-line Perl monolith which combines command-line parsing, upstream and downstream file I/O, DFS, MMB, half a BASIC detokeniser...

And, as I say, the second arm of the problem is that Perl doesn't interoperate especially well with other languages, or embed well in resource-constrained environments.

User avatar
davidb
Posts: 2231
Joined: Sun Nov 11, 2007 10:11 pm
Contact:

Re: Acorn files on other platforms

Post by davidb » Wed Apr 04, 2018 11:19 pm

At some point I'll get back to updating my ADFSlib Python module with the information from the ADFS Format thread.

For DFS, I have a couple of modules that I use for writing disk images, but I tend to copy them between projects rather than develop them independently. Samples can be found in this project, for example. My plan was to extend this to support ADFS images purely for the purpose of generating new images rather than reading old ones. When I get a chance I'll create a new repository with these modules under the stardot organisation on GitHub.

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Thu Apr 05, 2018 12:34 am

Ooh! Stylistically at least, if not in terms of completeness of implementation, that's much more like it. (-8

crj
Posts: 834
Joined: Thu May 02, 2013 4:58 pm
Contact:

Re: Acorn files on other platforms

Post by crj » Thu Apr 05, 2018 1:15 am

A wild idea that's occurred to me: At first glance neither ADFS nor 1770 DFS need very much of the OS. If you emulated:
  • 6502 CPU
  • RAM at &0000-&02FF, &0D00-&1FFF
  • The filing system ROM at &8000-&BFFF (no need for paging!)
  • OS entry points at &FF00-&FFFF
  • Floppy hardware in page &FE80-&FE87
  • Hard disc hardware in page &FC40-&FC47
  • Minimalist WRCH/RDCH/INKEY/Escape/BRK support
  • Tube reported as present, THS entry points in page &400, Tube I/O via &FEE5
...would you not then have an extremely lightweight implementation?
If you wanted to simplify further, you'd use DFS/ADFS variants that had been tricked out to make OSWORD calls instead of accessing the disc hardware.

User avatar
kieranhj
Posts: 724
Joined: Sat Sep 19, 2015 10:11 pm
Location: Farnham, Surrey, UK
Contact:

Re: Acorn files on other platforms

Post by kieranhj » Fri Apr 06, 2018 8:27 pm

hoglet wrote:Are you aware of Stephen's mmb_utils?
https://sweh.spuddy.org/Beeb/mmb_utils.html
https://github.com/sweharris/MMB_Utils

They are written in perl, but run happily on Windows.
I use these for the packaging phase in all of my projects.
They are also small enough to be checked in to git, which removes a additional build time dependency.
I did find mmb_utils on GitHub but my knowledge of perl is non-existent so gave it a pass. My dev environment is just Windows DOS prompt with batch files for scripting, to keep things as broadly portable as possible (across Windows machines at least, I used to be a BASH dude BITD.) Simon is a Python guy so some of our tools are in that (v2.7) which works fine on the command line once the run-time is installed. I would be more than happy with Python tools / libraries as this seems to be emerging as a fairly universal scripting language across Windows & Linux etc.
crj wrote:
hoglet wrote:With mmb_utils I think you get control of the order of the files if you add them one at a time.
It sounds like Kieran is trying to get more clever than that, optimising for the disc spin rate between finishing loading one file and starting to load the next! Though I'm not sure why he cares, given anyone who wants fast loading won't be using a real floppy these days. (-8
Ahahahaha! You'd be surprised. :) I reckon more people use emulators than real hardware with MMC type devices these days and emulators... also emulate floppy disc drives at real speeds. So for everyone playing POP with jsbeeb I spent quite a bit of effort getting the boot time down to ~10 seconds and ~6 seconds to load between game levels and the cutscenes.

Normally I take advantage of the order of files as you add them to the image - I use dummy files of a given sector size to pad as required and then leave or remove them depending on whether the catalog is important.
Bitshifters Collective | Retro Code & Demos for BBC Micro & Acorn computers | https://bitshifters.github.io/

User avatar
simonm
Posts: 209
Joined: Mon May 09, 2016 2:40 pm
Contact:

Re: Acorn files on other platforms

Post by simonm » Sat Apr 07, 2018 6:32 pm

I used to like Perl quite a lot, but then I discovered Python, which is much more structured and easier to interface with underlying systems and binary files. Appreciate it's not everyone's cup of tea though, and it does take a bit of getting used to at first (wot, no braces?! :lol: )
David B's scripts look pretty cool (& far more organised than my typically quick hack Python!). I think the idea of a stardot repo full of handy Acorn Python scripts is a good one too. Handling SSD/DSD files etc. is a doddle in Python and I see no reason why a script equivalent to a BBCIm.exe couldnt be created. Easier to extend too, and cross platform.

User avatar
davidb
Posts: 2231
Joined: Sun Nov 11, 2007 10:11 pm
Contact:

Re: Acorn files on other platforms

Post by davidb » Sat Apr 07, 2018 8:06 pm

I'll upload my DFS-related scripts to the stardot GitHub organisation and we can take things from there. :)

Coeus
Posts: 1077
Joined: Mon Jul 25, 2016 11:05 am
Contact:

Re: Acorn files on other platforms

Post by Coeus » Sat Apr 07, 2018 9:58 pm

simonm wrote:I used to like Perl quite a lot, but then I discovered Python, which is much more structured and easier to interface with underlying systems and binary files. Appreciate it's not everyone's cup of tea though, and it does take a bit of getting used to at first (wot, no braces?! :lol: )
I don't know what you'd call "getting used to". The fact that adding a control construct to some existing code and not re-indenting it correctly causes the behaviour to change doesn't seem to me to be ideal.
simonm wrote:David B's scripts look pretty cool (& far more organised than my typically quick hack Python!). I think the idea of a stardot repo full of handy Acorn Python scripts is a good one too. Handling SSD/DSD files etc. is a doddle in Python and I see no reason why a script equivalent to a BBCIm.exe couldnt be created. Easier to extend too, and cross platform.
If people want to write in Python then it does seem like a repository of useful scripts would be a good idea. I suppose this is the modern approach to scripting - have a scripting language in which you can implement all the building blocks as modules and then tie them together with more code in that same language. BBCim comes from the previous era where one would write a series of small programs in, for example C, and combine them by use of the OSes own scripting language. That tended to be much more common on Unix because the Unix shell language was, well, a language which the DOS command processor was not. Since then the windows command shell seems to have added most of the missing features. So a tool like BBCim should itself be portable, to anything that can do standard file I/O, if you script around it that may end up being OS-specific.

Post Reply