DFS catalogue... is this wrong?

bbc/electron apps, languages, utils, educational progs, demos + more
Post Reply
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

DFS catalogue... is this wrong?

Post by msknight »

I'm reading this... https://beebwiki.mdfs.net/Acorn_DFS_disc_format
...and it seems that my maths aren't coming out right when compared with other DFS reading software.

It struck me that the definition also doesn't allow for attributes such as file lock, so I was wondering whether it was correct.
gfoot
Posts: 134
Joined: Tue Apr 14, 2020 9:05 pm
Contact:

Re: DFS catalogue... is this wrong?

Post by gfoot »

Which maths didn't come out right?

I think the locked state is stored in the top bit of the directory byte.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

Ah. Yes, You're right on the lock byte.

Here's my maths in PHP...

Code: Select all

function read_file($disk, $file, $side) 
{
	$offset=8*$file;
    if ($side == 1)
    {   $offset=$offset+2560;   }
    $f_name = substr($disk,$offset,7);
    $f_dir = intval(ord($disk[$offset+7])) % 128;
    $f_load = intval(ord($disk[$offset+256]))+(intval(ord($disk[$offset+257]))*256);
    $f_exec = intval(ord($disk[$offset+258]))+(intval(ord($disk[$offset+259]))*256);
    $f_leng = intval(ord($disk[$offset+260]))+(intval(ord($disk[$offset+261]))*256);
    $f_start = intval(ord($disk[$offset+262]));
    $e_bit = intval(ord($disk[$offset+263]));
    $f_load = $f_load + (65536*intval((($e_bit / 4) % 4)));
    $f_exec = $f_exec + (65536*intval(($e_bit / 32)));
    $f_leng = $f_leng + (65536*intval((($e_bit / 16) % 4)));
    $f_start = $f_start + (256*intval(($e_bit % 4)));
echo "<p>$offset - $f_name - ".chr($f_dir)." - ".dec2hex($f_load)."h - ".dec2hex($f_exec)."h - ".dec2hex($f_leng)."h - ".dec2hex($f_start)."h - $e_bit</p>";
  return $str;
}
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

...and here's a comparison between DFS Explorer and the raw dump of the PHP script...
Attachments
Screenshot from 2021-09-26 15-57-00.png
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

I think it's me. I had two of the bits reference wrong.
gfoot
Posts: 134
Joined: Tue Apr 14, 2020 9:05 pm
Contact:

Re: DFS catalogue... is this wrong?

Post by gfoot »

It looks pretty good to me. You've written the sectors in decimal instead of hex, and you're not interpreting the top two address bits quite right I think.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

The only thing I don't fully understand is the extra FF at the start of some of the addresses in DFS Explorer.
Attachments
Screenshot from 2021-09-26 16-18-09.png
gfoot
Posts: 134
Joined: Tue Apr 14, 2020 9:05 pm
Contact:

Re: DFS catalogue... is this wrong?

Post by gfoot »

That's better. You had e_bit and start backwards before.

Now I think you are still using the wrong top bits for "exec" - you should divide by 64 not 32.

The top two bits of the addresses should be extended up to form the 32 bit address. So if the top two bits are set, you'll currently get an address starting with 3, but to the system it is treated as starting with FFFF. DFS just doesn't bother storing all those bits
User avatar
geraldholdsworth
Posts: 1002
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: DFS catalogue... is this wrong?

Post by geraldholdsworth »

There was a discussion about those top two bits, over on the Disc Image Manager thread. DFS takes the two bits set as FF, while anything else would be 00, 01, or 02.

I think I left it, with DIM, to multiply these two bits by 0x55, so you would get 00, 55, AA, and FF.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

And now I'm starting to wonder whether it's worth the trouble of unpicking this.

I had a 40 track SSD that I loaded in the Gotek, to check the behaviour on deleting files, and it saved it back to the USB stick as double the file size. So there goes my using the maths of number of sectors against file size, to calculate how many sides the disk image has. Looks like I'd be better off running by the file extension.
Attachments
Screenshot from 2021-09-26 17-24-02.png
User avatar
geraldholdsworth
Posts: 1002
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: DFS catalogue... is this wrong?

Post by geraldholdsworth »

You can't rely on a disc size being correct. DFS image files are very commonly truncated so they just have the data on them. Double sided is a different matter, as these are interleaved.

Have a peek at this.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

An interesting read.

Slightly dissapointed that the "Some photos I thought were quite good" appeared to be red to me... especially as an amateur photographer myself... but hey ho.

I am going to revisit the dropcalc project that I dropped over a decade ago, for the L2J community. That's going to get nuts, pretty quickly. But hey ho.

I don't know about you, but I look back at code I wrote decades ago, scratch my head and wonder how on earth I wrote that stuff.

Oops... pinger has gone. Got to go make dinner.
User avatar
sweh
Posts: 2563
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: DFS catalogue... is this wrong?

Post by sweh »

msknight wrote:
Sun Sep 26, 2021 5:26 pm
I had a 40 track SSD that I loaded in the Gotek, to check the behaviour on deleting files
The entry is just removed from the catalogue (sectors 0/1) and other entries moved up. Data sectors are left unchanged, which leaves a gap in the disk, which is why *COMPACT was used to close the gap.
So there goes my using the maths of number of sectors against file size, to calculate how many sides the disk image has. Looks like I'd be better off running by the file extension.
You can't trust either, unfortunately. A 200Kb SSD could be smaller than that, with the assumption that the software processing it will fill in the rest with zeros. Similarly a DSD disk _could_ be interleaved at the track level (side 0 track 0; side 2 track 0; side 0 track 1; side 2 track 1...) or it _could_ be concatenated (side 0 track0...79; side 1 track 0..79).

There's a level of heuristics involved in autodetection but sometimes tools get it wrong (BeebEm sometimes pops up a warning if it thinks my SSD is a DSD). So in my code I always assume 200Kb SSDs[1] and provide commands to split/merge DSDs with interleaved/concatenated options.

[1] Well, not always. If you select Soldisk mode then it could be 320K "double density" (80*16) and Opus mode could be 720K (80*18). But still single-sided!
Rgds
Stephen
User avatar
geraldholdsworth
Posts: 1002
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: DFS catalogue... is this wrong?

Post by geraldholdsworth »

msknight wrote:
Sun Sep 26, 2021 6:25 pm
Slightly dissapointed that the "Some photos I thought were quite good" appeared to be red to me... especially as an amateur photographer myself... but hey ho.
Yep, I must get that part of the website written. You might like the ESCC site - I'm a member and I also wrote the website. OK, not many photos available to look at from the public side, just the top 4s in each competition.
msknight wrote:
Sun Sep 26, 2021 6:25 pm
I don't know about you, but I look back at code I wrote decades ago, scratch my head and wonder how on earth I wrote that stuff.
That was the reason behind that document - because I'd never remember any of it a couple of years down the line! These days I also comment a lot more in code.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

Yeah... I commented... and I still can't understand it...
http://msknight.com/source1.html

I was a member of the Mid Sussex Camera Club for a while, but the problem is that everyone's got their own ideas... beauty is subjective, as was proven when my entry to the RPS, (which had been guided by an RPS exam board member, who wasn't sitting on the exam panel that I submitted against) came down rather scathingly on my exam submission.
http://msknight.com/gallery.html
...at that point I concluded that the only valid judge of my photography are people who I have previously judged as being people who hold truthful opinion... in other words, if I have a high opinion of someone then I'll gladly accept their feedback, good or bad.
Coeus
Posts: 2357
Joined: Mon Jul 25, 2016 12:05 pm
Contact:

Re: DFS catalogue... is this wrong?

Post by Coeus »

geraldholdsworth wrote:
Sun Sep 26, 2021 5:44 pm
Have a peek at this.
That appears blank when I look at it, except for the title/menu bar at the top.
Coeus
Posts: 2357
Joined: Mon Jul 25, 2016 12:05 pm
Contact:

Re: DFS catalogue... is this wrong?

Post by Coeus »

sweh wrote:
Sun Sep 26, 2021 6:48 pm
You can't trust either, unfortunately. A 200Kb SSD could be smaller than that, with the assumption that the software processing it will fill in the rest with zeros.
I think it is best to accept these short files as input even if you choose to always write the full-sized files.
sweh wrote:
Sun Sep 26, 2021 6:48 pm
Similarly a DSD disk _could_ be interleaved at the track level (side 0 track 0; side 2 track 0; side 0 track 1; side 2 track 1...) or it _could_ be concatenated (side 0 track0...79; side 1 track 0..79).
With the tracks of the two sides interleaved is by far the more common format for .dsd files but there definitely are some out in the wild that have the sides sequential, i.e. all of side 0 and then all of side 2.

An interleaved track DSD need not even be 200K in length if both the sides have only small files on them, only long enough for (2x) the data on the fullest side.

There is a further complication in that a .ssd file can also be double sided. JGH says SSD stands for Sequential Sided Disc rather than the more obvious Single Sided Disc and therefore considers a second side, always non-interleaved and simply following the first side, as legitimate. So if a .ssd file is greater than 200K it may be one of these or it may be an image from a double-density DFS which has more sectors per track!

We have got into a real muddle here by having the most common disc image format a headerless one with no metadata at all, not even about the geometry of the disc that has been imaged, except to a very limited extent in the file extension which may be wrong or differently interpreted.

Then the DFS catalogue does not have sufficient redundancy in its structure to enable reliable detection. There are some checks that can be made, like is the number of sectors reasonable, are the entries for the files in the correctly sorted order (descending by sector number IIRC) but other application data can easily look like a valid DFS catalogue. Every time I tried some kind of auto-detection in B-Em I got reports of it mis-detecting something.
User avatar
geraldholdsworth
Posts: 1002
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: DFS catalogue... is this wrong?

Post by geraldholdsworth »

Coeus wrote:
Mon Sep 27, 2021 2:40 pm
geraldholdsworth wrote:
Sun Sep 26, 2021 5:44 pm
Have a peek at this.
That appears blank when I look at it, except for the title/menu bar at the top.
Yep, not too clever is it? Try a direct link.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

Coeus wrote:
Mon Sep 27, 2021 3:19 pm
..., are the entries for the files in the correctly sorted order (descending by sector number IIRC) ...
Aw Gawd... if you'd never have mentioned that, I'd never have known. Looks obvious now.

I haven't got too far, just threw together a disk storage system for upload and download with basic operations in PHP and MySQL to be platform agnostic, and a file name recall with a quick pictorial analysis of sectors. Still a fair way to go. Don't think I'll even try and cater for the other formats. I'll leave that to the experts :-)
Attachments
Screenshot from 2021-09-27 17-44-18.png
Screenshot from 2021-09-27 17-45-19.png
User avatar
geraldholdsworth
Posts: 1002
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: DFS catalogue... is this wrong?

Post by geraldholdsworth »

I thought of writing Disc Image Manager, or parts of it, in PHP. A better option would be Javascript - with PHP, you'd need to upload the disc image to a server for the code to analyse it, but with Javascript it can be dealt with on the client's machine. Well, as far as I am aware that is the case - I've only pottered around with images (of course) and videos locally using Javascript, but I think it is possible.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

PHP is the only language I'm competent with at the moment. I believe there's two versions of javascript, one that runs on the server and another that runs in the browser/client, but I know very little about it. The one I'm writing will only be very basic, critical functions only, nowhere near what Disc Image Manager can do. And given what's going on around here, it'll probably take a couple of months for me to write that :lol:
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

OK - I think I can now conclude that I can not assume that the directory is always going to be ordered by starting sector. This is going to make the compacting function a little... er... interesting.
Attachments
Screenshot from 2021-09-29 12-43-00.png
User avatar
geraldholdsworth
Posts: 1002
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: DFS catalogue... is this wrong?

Post by geraldholdsworth »

It's in reverse order, if memory serves. The same order as *EX lists them.

What I did for a *COMPACT is to clone the image, delete everything on the clone then import all the files from the original.
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

Ahhh... I never do things the easy way... still tuning it, although I think it works... much testing to do. Already entering this section of code with the drive surface in $d_data and a few other variables that describe a few things.

Code: Select all

if ($compact)
{
    echo "<p>Compact Requested</p>";
    $file=0;
    $next_start=2;
    while ($file<$numoffiles)
    {      
        $row=$d_catalog[$file];
        $sector_offset=0;
        if ($target_drive)   {   $sector_offset=2560;   }  ## If we're working on side 2, add 2560 to jump to the interleaved track.
        $sector_start=$row['Start'];
        $file_length=$row['Leng'];
        $file_offset=$row['Offset'];
        $sectors_used=intval($file_length/256);                         ## Find out how many sectors the file uses.
        if (($file_length % 256) > 0)   {   $sectors_used++;    }       ## ...adding 1 for a partial sector.

        if ($sector_start > $next_start)            ## If the file is beyond the next free state, then we need to bring it forward.
        {
            $sector_work=0;
            $sector_offset=0;

            ## The location of the byte which describes the starting sector is the start of the sector (sector offset)
            ## plus the location of the file in the catalog (file offset) plus the bump for the next track (256) plus 7
            ## bytes forward from that.
            $sector_byte_location=$sector_offset+$file_offset+263;
            $d_data[$sector_byte_location]=chr($next_start % 256);   ## Put the lower 8 bits of the size in the catalogue
            $top_two_start = intval($next_start/256);
            $d_data[$sector_byte_location-1]=chr((intval(ord($d_data[$sector_byte_location-1])/4)*4)+$top_two_start);
            while ($sector_work < $sectors_used)
            {
                $source_sector=$sector_start+$sector_work+$sector_offset;
                $destination_sector=$sector_start+$sector_work+$sector_offset;
                if ($sides==2)
                {
                    $source_sector = $source_sector+(intval($source_sector/10)*2560);
                    $destination_sector = $destination_sector+(intval($destination_sector/10)*2560);
                    $sector_code = substr($d_data,$source_sector,256);
                    $insert_pointer=0;          ## When reading from a string, the starting position is 0.
                    while ($insert_pointer<256)
                    {
                        $d_data[($destination_sector+$insert_pointer)]=$sector_code[$insert_pointer];
                        $insert_pointer++;
                    }
                }
                else
                {
                    $sector_code = substr($d_data,$source_sector,256);
                    $insert_pointer=0;          ## When reading from a string, the starting position is 0.
                    while ($insert_pointer<256)
                    {
                        $d_data[($destination_sector+$insert_pointer)]=$sector_code[$insert_pointer];
                        $insert_pointer++;
                    }
                }
                $sector_work++;
            }
        }
        $next_start = $next_start+$sectors_used;
        $file++;
    }

    ## Re-read the catalogue after all this messing around.
    $file=1;
    $d_catalog=ARRAY();
    while ($file<=$numoffiles)
    {
        $d_catalog[$file-1]=read_file($d_data,$file,$target_drive);
      $file++;
    }
    $array_count=count($d_catalog);
    array_sort_by_column($d_catalog,"Start");
    $file_slashed = addslashes($d_data);
    $sql = 'update bbcfloppies.floppylist set diskdata="'.$file_slashed.'" where filename="'.$target_file.'"';
    $result = mysqli_query($con,$sql);
}
User avatar
msknight
Posts: 838
Joined: Fri Apr 15, 2011 12:07 pm
Location: Sussex, UK
Contact:

Re: DFS catalogue... is this wrong?

Post by msknight »

Don't use the above, by the way. Doesn't work... yet ... :-D
Post Reply

Return to “8-bit acorn software: other”