Automating the creation of SSD's

having trouble with an archived file or want to correct something? report it here!
User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Mon May 18, 2015 6:38 am

I rushed out that last bit of my message - I was more thinking about using the SSDs on real hardware e.g. Data Centre. That's where the SSD names get a bit more limited, because of the 8.3 file name limitation.

But you've made me realise that that issue can be kicked further down the road - if I use descriptive names at the PC end, then "we" can worry about truncating them later.

So I can base the SSD title on the game name for now (some characters will have to be stripped) and then we'll get a good set of well-named SSDs that we can start looking at, tidying up, etc.

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sat May 23, 2015 1:24 am

OK, new version below. (Kids in front of DVD, wife probably fallen asleep in front of it, so I get a quick retro hour!)

This creates a new SSD file, does *OPT 4 3, creates a !BOOT that *EXECs a STARDOT file, and locks all the files on the new SSD. (Wasn't sure what to put in for the disc title, so left that for now.)

I did it this way because I wasn't quite sure what you were all after in the !BOOT and the loaders. Thanks to Stephen for the code snippets (and of course the Perl modules themselves); saved me lots of coding and RTFM. =D>

One new directory must be created first, New_SSD - where you want the new SSD files to go.

There's no error checking; no coding competitions will be won; some of the comments are lies; there are probably 25 better/faster ways to do it; there will be titles/files it does not like; but of the 5 or so games I've tried on Disc 030, all work after being mangled by the script.

USE AT YOUR OWN RISK! :shock: The only "dangerous" thing I'm aware of is the unlink very close to the end of the file - so you can comment that out until you are sure it's all OK.

So, Lee - you got Perl & PHP on a machine? Ready to start trying to convert a few and we'll see what happens?

Code: Select all

$ cat process_mb_dsd.php                                                                                       
<?php

define('VERSION','0.02');

define('PERL','/usr/bin/perl');
define('BEEB','/home/richard.toohey/mmc_beeb/sweh_perl/beeb');

define('DSD_DIR','/home/richard.toohey/mb_dsd_splitter/DSD');
define('SSD_DIR','/home/richard.toohey/mb_dsd_splitter/SSD');
define('NEW_SSD_DIR','/home/richard.toohey/mb_dsd_splitter/New_SSD');

class Game {
        public $Name;
        public $Side=0;
        public $FirstFile;
        public $NextFile;
        public $FilesDirectory;
        public $Files=array();
}

# Grab a DSD and process it
foreach (glob(DSD_DIR.DIRECTORY_SEPARATOR.'*.dsd') as $dsd_path) {
        echo "$dsd_path\n";

        $pp=pathinfo($dsd_path);

        $fn=$pp['filename'];

        # Step 1 - split out the two SSDs
        $ssd0=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_0.ssd";
        $ssd2=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_2.ssd";
        $result=exec(PERL.' '.BEEB." split_dsd $dsd_path $ssd0 $ssd2",$output,$rc);

        # Step 2 - get all files (we need !BOOT first, but we'll need
        # all the other files anyway, to build the separate SSD files)
        $ssd0d=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_0";
        $ssd2d=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_2";
        $result=exec(PERL.' '.BEEB." getfile $ssd0 $ssd0d",$output,$rc);
        $result=exec(PERL.' '.BEEB." getfile $ssd2 $ssd2d",$output,$rc);

        $cat_0=array();
        $cat_2=array();

        # Step 3 - get the catalogues in the order we need them
        $result=exec(PERL.' '.BEEB." info $ssd0",$cat_0,$rc);
        $result=exec(PERL.' '.BEEB." info $ssd2",$cat_2,$rc);

        if ($cat_0) {
                $games=array();
                $game_list=array();
                # Step 4 - read the !BOOT (BASIC) file from side 0 - it will
                # have the menu program.  Find the DATA lines and process them
                $pling_boot=array();
                $result=exec(PERL.' '.BEEB." list $ssd0d/!BOOT",$pling_boot,$rc);
                if ($pling_boot) {
                        #print_r($pling_boot);
                        $side0_data_found=false;
                        $side2_data_found=false;
                        foreach ($pling_boot as $line) {

                                if (preg_match('/.*?0DATA.*/',$line)) {
                                        if ($side0_data_found==false) {
                                                # Must be first title on side 0
                                                #echo "Side 0 data starts with $line\n";
                                                $side0_data_found=true;
                                                $games[]=make_game(0,$line,$ssd0d);
                                        } else {
                                                if ($side2_data_found==false) {
                                                        #echo "side 0 title $line\n";
                                                        $games[]=make_game(0,$line,$ssd0d);
                                                } else {
                                                        #echo "side 2 title $line\n";
                                                        $games[]=make_game(2,$line,$ssd2d);
                                                }
                                        }
                                }

                                if (preg_match('/.*?0REM.*/',$line)) {
                                        if ($side0_data_found) {
                                                #echo "end of side 0? $line\n";
                                                $side2_data_found=true;
                                        }
                                }
                                       
                        }

                        # Step 5 - we have the list of games, so let's work out
                        # the files to go into the SSD
                        if ($games) {
                                for ($i=0;$i<count($games);$i++) {
                                        $j=$i+1;
                                        if ($j<count($games)) {
                                                if ($games[$i]->Side==$games[$j]->Side) {
                                                        $games[$i]->NextFile=$games[$j]->FirstFile;
                                                } else {
                                                        $games[$i]->NextFile="END-OF-DISC";
                                                }
                                        } else {
                                                $games[$i]->NextFile="END-OF-DISC";
                                        }
                                }       

                                foreach ($games as $game) {
                                        echo "Processing $game->Name first file $game->FirstFile\n";
                                        if ($game->Side==0) {
                                                $cat_to_search=array_reverse($cat_0);
                                        } else {
                                                $cat_to_search=array_reverse($cat_2);
                                        }

                                        # And walk this array for files required
                                        $gathering_files=false;
                                        $files_found=array();
                                        foreach ($cat_to_search as $cat_file) {
                                                if (substr($cat_file,0,2)=='$.') {
                                                        $cat_file=trim(substr($cat_file,2,10));
                                                        if (strtoupper($cat_file)==strtoupper($game->FirstFile)) {
                                                                $gathering_files=true;
                                                                #echo "found first, gathering\n";
                                                        }
                                                        # For END-OF-DISC games we'll get them all, anyway
                                                        if (strtoupper($cat_file)==strtoupper($game->NextFile)) {
                                                                $gathering_files=false;
                                                                #echo "encountered next, stop gathering\n";
                                                        }
                                                        if ($gathering_files) {
                                                                #echo "gathering $cat_file ";
                                                                $files_found[]=$cat_file;
                                                        }
                                                }
                                        }
                                        #echo "\n";
                                        #echo "files for game "; print_r($files_found); echo "\n";
                                        $game->Files=$files_found;
                                        echo "\tFiles for game: ";
                                        foreach ($game->Files as $f) {
                                                echo "$f ";
                                        }
                                        echo "\n";

                                        make_game_ssd($game);

                                }
                        }
                } else {
                        echo "Cannot find !BOOT\n";
                }
        } else {
                echo "Cannot find side 0 catalogue\n";
        }
}

exit;

function make_game($side,$line,$local_directory) {
        # Split out line parts <space><line#>DATA<title>,<first_file>
        if (preg_match('/[\s]{1,3}[0-9]{1,3}DATA(.*?),(.*)/',$line,$matches)) {
                $game=new Game();
                $game->Name=$matches[1];
                $game->Side=$side;
                $game->FirstFile=$matches[2];   
                $game->FilesDirectory=$local_directory;
                return $game;
        }
}

function make_game_ssd($game) {
        # So build a .SSD based on the game name, with a !BOOT
        #beeb blank_ssd $DSK
        #beeb opt4 $DSK 3
        #beeb title $DSK "Games Disk"
        #beeb putfile $DSK OBJS/*
        #beeb access $DSK '*.*' L
        $ssd_name=preg_replace('/[^0-9A-Z]/i','',$game->Name).'.ssd';
        $new_ssd_path=NEW_SSD_DIR.DIRECTORY_SEPARATOR.$ssd_name;
        echo "$new_ssd_path\n";
        @unlink($new_ssd_path);
        $result=exec(PERL.' '.BEEB." blank_ssd $new_ssd_path",$output,$rc);
        $result=exec(PERL.' '.BEEB." opt4 $new_ssd_path 3",$output,$rc);
        foreach ($game->Files as $gf) {
                $src=$game->FilesDirectory.DIRECTORY_SEPARATOR.$gf;
                echo "$src\n";
                $result=exec(PERL.' '.BEEB." putfile $new_ssd_path $src",$output,$rc);
        }
        # Build the !BOOT and write it
        file_put_contents('/tmp/!BOOT',"*BASIC\r*EXEC \"STARDOT\"\r");
        # Build the STARDOT loader program and write it
        file_put_contents('/tmp/STARDOT',"CLOSE#0:CHAIN \"$game->FirstFile\"\r");
        $result=exec(PERL.' '.BEEB." putfile $new_ssd_path /tmp/!BOOT",$output,$rc);
        $result=exec(PERL.' '.BEEB." putfile $new_ssd_path /tmp/STARDOT",$output,$rc);
        # Lock all the files on the disc
        $result=exec(PERL.' '.BEEB." access $new_ssd_path '*.*' L",$output,$rc);
}
?>
EDIT: one bug squished and posted new version.

User avatar
leenew
Posts: 3408
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Sat May 23, 2015 7:25 am

Cheers Richard,
I will look at this as soon as possible.
If it can create 1900 perfect SSDs in one hit it will be amazeballs as "the kids" say :D
If anyone else with the appropriate skills can peruse the code and offer any suggestions/improvements/whatever's then speak now before I hit the "go" button... 8)
Cheers
Lee.

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sat May 23, 2015 7:54 am

leenew wrote:If it can create 1900 perfect SSDs in one hit
Doubt it will do that - but hopefully a large proportion of them.

Don't chuck 50 DSDs at it in one go, though - try, say, 5 and see how it goes. If that works, try another 5, then 10, then 20.

I've tried with a couple of .dsd files in the DSD directory and it seemed to work for the SSDs I checked, but anything with unusual file names or files not in $ won't work.

There will come a point where manually tweaking the generated SSDs is quicker than making the code handle every quirk of every title.

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Sat May 23, 2015 6:25 pm

=D> =D> =D>
Great work, Richard!

Just tried it on Disc030.dsd:

Code: Select all

$ php process_mb_dsd.php
/wherever/toohey/DSD/Disc030.dsd
Processing Cavey first file CAVEY
   Files for game: CAVEY CAVEY1 CAVEY2
/wherever/toohey/New_SSD/Cavey.ssd
/wherever/toohey/SSD/Disc030_0/CAVEY
/wherever/toohey/SSD/Disc030_0/CAVEY1
/wherever/toohey/SSD/Disc030_0/CAVEY2
Processing The Maze first file MAZE
   Files for game: MAZE INT MAZCD
/wherever/toohey/New_SSD/TheMaze.ssd
/wherever/toohey/SSD/Disc030_0/MAZE
/wherever/toohey/SSD/Disc030_0/INT
/wherever/toohey/SSD/Disc030_0/MAZCD
Processing Green Beret first file GB-INST
   Files for game: GB-Inst GREEN MOD2 LODCODE BERET
/wherever/toohey/New_SSD/GreenBeret.ssd
/wherever/toohey/SSD/Disc030_0/GB-Inst
/wherever/toohey/SSD/Disc030_0/GREEN
/wherever/toohey/SSD/Disc030_0/MOD2
/wherever/toohey/SSD/Disc030_0/LODCODE
/wherever/toohey/SSD/Disc030_0/BERET
Processing Missile Attack first file MISSILE
   Files for game: MISSILE MisAtt1 MisAtt2
/wherever/toohey/New_SSD/MissileAttack.ssd
/wherever/toohey/SSD/Disc030_0/MISSILE
/wherever/toohey/SSD/Disc030_0/MisAtt1
/wherever/toohey/SSD/Disc030_0/MisAtt2
Processing Powerboat Racing first file POWERBO
   Files for game: POWERBO
/wherever/toohey/New_SSD/PowerboatRacing.ssd
/wherever/toohey/SSD/Disc030_0/POWERBO
Processing Despatch Rider first file BBCLOAD
   Files for game: BBCload track sounds
/wherever/toohey/New_SSD/DespatchRider.ssd
/wherever/toohey/SSD/Disc030_0/BBCload
/wherever/toohey/SSD/Disc030_0/track
/wherever/toohey/SSD/Disc030_0/sounds
Processing Future Shock first file LOADER
   Files for game: LOADER SCREEN SHOCK1 SHOCK15 FUTURE
/wherever/toohey/New_SSD/FutureShock.ssd
/wherever/toohey/SSD/Disc030_0/LOADER
/wherever/toohey/SSD/Disc030_0/SCREEN
/wherever/toohey/SSD/Disc030_0/SHOCK1
/wherever/toohey/SSD/Disc030_0/SHOCK15
/wherever/toohey/SSD/Disc030_0/FUTURE
Processing Hunkidory first file HUNKY
   Files for game: HUNKY HUNKY2 Hcode1 Hcode2 SCRDATA
/wherever/toohey/New_SSD/Hunkidory.ssd
/wherever/toohey/SSD/Disc030_0/HUNKY
/wherever/toohey/SSD/Disc030_0/HUNKY2
/wherever/toohey/SSD/Disc030_0/Hcode1
/wherever/toohey/SSD/Disc030_0/Hcode2
/wherever/toohey/SSD/Disc030_0/SCRDATA
Processing Mango first file MANGO
   Files for game: MANGO MANGO2
/wherever/toohey/New_SSD/Mango.ssd
/wherever/toohey/SSD/Disc030_0/MANGO
/wherever/toohey/SSD/Disc030_0/MANGO2
Processing Paperboy first file PAPER
   Files for game: PAPER LOAD PROG DATA
/wherever/toohey/New_SSD/Paperboy.ssd
/wherever/toohey/SSD/Disc030_2/PAPER
/wherever/toohey/SSD/Disc030_2/LOAD
/wherever/toohey/SSD/Disc030_2/PROG
/wherever/toohey/SSD/Disc030_2/DATA
Processing Gatecrasher first file GATE1
   Files for game: GATE1 GATE2 GATE3
/wherever/toohey/New_SSD/Gatecrasher.ssd
/wherever/toohey/SSD/Disc030_2/GATE1
/wherever/toohey/SSD/Disc030_2/GATE2
/wherever/toohey/SSD/Disc030_2/GATE3
Processing Stellar Rescue first file HEADER
   Files for game: HEADER STELLAR
/wherever/toohey/New_SSD/StellarRescue.ssd
/wherever/toohey/SSD/Disc030_2/HEADER
/wherever/toohey/SSD/Disc030_2/STELLAR
Processing Raid Over Moscow first file MOSCOW
   Files for game: MOSCOW MOSCOW1 RAID RAID1 RAID2
/wherever/toohey/New_SSD/RaidOverMoscow.ssd
/wherever/toohey/SSD/Disc030_2/MOSCOW
/wherever/toohey/SSD/Disc030_2/MOSCOW1
/wherever/toohey/SSD/Disc030_2/RAID
/wherever/toohey/SSD/Disc030_2/RAID1
/wherever/toohey/SSD/Disc030_2/RAID2
Processing "Super Barricade?(2P)" first file BARCADE
   Files for game: BARCADE BARCAD2
/wherever/toohey/New_SSD/SuperBarricade2P.ssd
/wherever/toohey/SSD/Disc030_2/BARCADE
/wherever/toohey/SSD/Disc030_2/BARCAD2
Processing One Last Game first file ONELOAD
   Files for game: ONELOAD ONELAST
/wherever/toohey/New_SSD/OneLastGame.ssd
/wherever/toohey/SSD/Disc030_2/ONELOAD
/wherever/toohey/SSD/Disc030_2/ONELAST
Processing Crazee Rider first file CRAZEE
   Files for game: Crazee CrazeeR CrazScr CrazPan CrazMod CRider Game??
/wherever/toohey/New_SSD/CrazeeRider.ssd
/wherever/toohey/SSD/Disc030_2/Crazee
/wherever/toohey/SSD/Disc030_2/CrazeeR
/wherever/toohey/SSD/Disc030_2/CrazScr
/wherever/toohey/SSD/Disc030_2/CrazPan
/wherever/toohey/SSD/Disc030_2/CrazMod
/wherever/toohey/SSD/Disc030_2/CRider
/wherever/toohey/SSD/Disc030_2/Game??
Processing Cosmonaut first file COSMO
   Files for game: COSMO SCORE OBJECT
/wherever/toohey/New_SSD/Cosmonaut.ssd
/wherever/toohey/SSD/Disc030_2/COSMO
/wherever/toohey/SSD/Disc030_2/SCORE
/wherever/toohey/SSD/Disc030_2/OBJECT
Processing Trench first file TRENCH
   Files for game: TRENCH trench1 trench2
/wherever/toohey/New_SSD/Trench.ssd
/wherever/toohey/SSD/Disc030_2/TRENCH
/wherever/toohey/SSD/Disc030_2/trench1
/wherever/toohey/SSD/Disc030_2/trench2
$ ls
DSD         SSD         process_mb_dsd.php
New_SSD         mmb_utils
$ cd New_SSD/
$ ls
Cavey.ssd      Gatecrasher.ssd      OneLastGame.ssd      SuperBarricade2P.ssd
Cosmonaut.ssd      GreenBeret.ssd      Paperboy.ssd      TheMaze.ssd
CrazeeRider.ssd      Hunkidory.ssd      PowerboatRacing.ssd   Trench.ssd
DespatchRider.ssd   Mango.ssd      RaidOverMoscow.ssd
FutureShock.ssd      MissileAttack.ssd   StellarRescue.ssd


Lee, processing Disc030 took about 6 seconds! I think your life just got a lot easier!

I haven't tested all the generated SSDs, but the ones I tried have all worked so far.

Thanks, Richard!

Btw, what's the issue with directory letters other than "$"?

Just updated the script to write the !BOOT first, to keep Mick happy! Also eliminated the STARDOT loader script and moved its contents to the !BOOT.
Last edited by lurkio on Sun May 24, 2015 7:25 pm, edited 1 time in total.

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sat May 23, 2015 9:13 pm

lurkio wrote:Btw, what's the issue with directory letters other than "$"?
It does a *INFO equivalent, and that has some non-file lines, so I got it to look for $. at the start of the line to say "this line is a file name". So if the line starts D.FILE, it would be skipped.

Around line 111 (specifically line 115):

Code: Select all

111                     # And walk this array for files required
112                     $gathering_files=false;
113                     $files_found=array();
114                     foreach ($cat_to_search as $cat_file) {
115                         if (substr($cat_file,0,2)=='$.') {
116                             $cat_file=trim(substr($cat_file,2,10));
So wouldn't be that hard to change - just need to dream up the right regular expression to match line starts <anything>.<7 chars><spaces> - something like that. Or just skip the first n lines and blank lines.

I know one of the games on Disc 030 has a file "Game??" and that gets converted by the Perl tools to "Game__" so that would need to be addressed as well.

Sounds like a good start, though. :D

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Automating the creation of SSD's

Postby jgharston » Sat May 23, 2015 10:00 pm

richardtoohey wrote:
lurkio wrote:Btw, what's the issue with directory letters other than "$"?
It does a *INFO equivalent, and that has some non-file lines, so I got it to look for $. at the start of the line to say "this line is a file name". So if the line starts D.FILE, it would be skipped.
DFS's *INFO output is a rigidly fixed format:
(1-character directory).(filename padded to 9 spaces)(L or space)(2 spaces)(6-digit load address)(space)(6-digit exec address)(space)(6-digit length)(space)(3-digit start sector)
What's the output of your tool's *INFO equivalent?

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sat May 23, 2015 10:16 pm

jgharston wrote:What's the output of your tool's *INFO equivalent?
OK, not exactly "equivalent", then. How about "mildly similar to"?

Code: Select all

Disk title: DISC 030     (80)  Disk size: &320 - 200K
Boot Option: 2 (RUN)   File count: 31

Filename: Lck Lo.add Ex.add Length Sct
$.MANGO2    L  001900 002B9B 002200 299
$.MANGO     L  001900 008023 000D5D 28B
...
$.CAVEY     L  001900 008023 000C3B 008
$.!BOOT     L  001900 001E00 000600 002
So the code checking for $. could be changed to skip the first five lines of the catalogue, and anything after that can be taken as file names.

I think there will still be a bit of work matching the names here from the names that have been extracted to local storage.

I wasn't going to get too bothered about this unless there were a lot of titles using other directories - it might be easier to manually build/adjust those titles. Let the script do 99% of the work, and manually build the remaining SSDs. But I'm not sure of the percentages - if there are only 40% that work without further tweaking, then tweaking is the better option.

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Sat May 23, 2015 11:49 pm

richardtoohey wrote:I know one of the games on Disc 030 has a file "Game??" and that gets converted by the Perl tools to "Game__" so that would need to be addressed as well.

Yes, but fortunately the Beeb filename "Game??" seems to be preserved inside the Game__.inf file and on the generated SSD, which is exactly the result we want. (But I can see how there could be circumstances in which certain chars in Beeb filenames might still cause conflicts. Hopefully those circumstances are unlikely [-o< )

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sat May 23, 2015 11:58 pm

lurkio wrote:fortunately the Beeb filename "Game??" seems to be preserved inside the Game__.inf file and on the generated SSD, which is exactly the result we want.
Errr, yes, of course I coded it knowing the utilities worked like that ... :^o :^o :^o

One less thing to worry about, then! :D

Where is the definitive source of the DSDs? I can test some more conversions. "We" (whoever we are?) can each take a slice of 10 DSDs to convert/test/report issues with or is there a more cunning plan already under way?

User avatar
rcook
Posts: 65
Joined: Fri May 22, 2015 4:43 pm
Location: Seattle, WA
Contact:

Re: Automating the creation of SSD's

Postby rcook » Sun May 24, 2015 6:54 am

poink wrote:Firstly, I've not found anything good for extracting files from DFS disc images.


I've started working on this: https://github.com/rcook/ssdtools. Currently I've only implemented "ssdexport" for extracting the contents of .ssd image files - but that should do what you're asking. I'm going to start working on the complementary "ssdimport" script when I get the chance. This script will take the output from "ssdexport" and create an equivalent .ssd image. The scripts will all be written in portable Python.

-Richard
Proud parent to Acorn Electrons 06-ALA01-0003584 and 07-ALA01-0017948

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sun May 24, 2015 7:10 am

I thought about starting from scratch, but (a) following the "programmer's are lazy" maxim and (b) seeing how much work would be involved for Watford/Opus/Acorn/etc. formats - I used Stephen's Perl-based utilities from here: http://sweh.spuddy.org/Beeb/mmb_utils.html

It was very good at extracting the files from SSDs. And adding files. And making a DSD into two SSDs. And making a blank SSD. And everything else I threw at it. :D

User avatar
Elminster
Posts: 1632
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK

Re: Automating the creation of SSD's

Postby Elminster » Sun May 24, 2015 7:18 am

richardtoohey wrote:I thought about starting from scratch, but (a) following the "programmer's are lazy" maxim and (b) seeing how much work would be involved for Watford/Opus/Acorn/etc. formats - I used Stephen's Perl-based utilities from here: http://sweh.spuddy.org/Beeb/mmb_utils.html


Sounds a bit too lazy to me. You should immediately rewrite it to run using ARM assembler on the Raspberry Pi

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Automating the creation of SSD's

Postby jgharston » Sun May 24, 2015 11:26 am

And for RISC OS export tools already exist in BeebArc:
*/<BeebArc$Dir>.DFSSplit <source file> <dest dir> [-ssd|-dsd :0|:2|:0:2] [-65Host] [-v] [-dfs62]
*/<BeebArc$Dir>.ADFSSplit <source file> <dest dir> [-dsd] [-type adfs|hadfs] [-v]
and creation tools in MkImage:
*MkImage outfile inpath [-fs dfs|adfs|hadfs] [-i@file] [-o opt] [-r] [-s size] [-t title]

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 24, 2015 2:21 pm

richardtoohey wrote:OK, new version below. (Kids in front of DVD, wife probably fallen asleep in front of it, so I get a quick retro hour!)

I don't see you setting the disk title ("beeb title"). This isn't too important for standalone SSDs, but with MMBs the default catalogue title (shown by *DCAT) comes from the SSD title. It's worth setting it, if possible.
Rgds
Stephen

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 24, 2015 2:23 pm

richardtoohey wrote:I thought about starting from scratch, but (a) following the "programmer's are lazy" maxim and (b) seeing how much work would be involved for Watford/Opus/Acorn/etc. formats - I used Stephen's Perl-based utilities from here: http://sweh.spuddy.org/Beeb/mmb_utils.html

It was very good at extracting the files from SSDs. And adding files. And making a DSD into two SSDs. And making a blank SSD. And everything else I threw at it. :D

Yay :-)

If you're on OS-X then you might also be interested in https://github.com/chrisa/mmb-fuse - this person took my libraries and built a FUSE file system on top of it so you can mount SSDs directly :-) It might even work on Linux (which is my platform of choice), but I haven't tried.
Rgds
Stephen

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 24, 2015 2:40 pm

richardtoohey wrote:OK, not exactly "equivalent", then. How about "mildly similar to"?

Code: Select all

Disk title: DISC 030     (80)  Disk size: &320 - 200K
Boot Option: 2 (RUN)   File count: 31

Filename: Lck Lo.add Ex.add Length Sct
$.MANGO2    L  001900 002B9B 002200 299
$.MANGO     L  001900 008023 000D5D 28B
...
$.CAVEY     L  001900 008023 000C3B 008
$.!BOOT     L  001900 001E00 000600 002
So the code checking for $. could be changed to skip the first five lines of the catalogue, and anything after that can be taken as file names.

My recommendation is that you throw away all lines up to and including /^Filename:/

If you have a disk with multiple catalogues on it (eg Solidisk) then an extra line can show in the header. You'll also notice a "catalogue sector" column in this case as well

Code: Select all

% beeb info stl:games3a.ssd | head -20
Disk title: Games 3 (58)  Disk size: &500 - 320K
Boot Option: 3 (EXEC)   File count: 66
(Multi-catalogues; last at &488)

Filename:  Lck Lo.add Ex.add Length Sct  Cat
$.!Boot        014556 010000 000046 4FC  488
M.Menu         FF1900 FF8023 00096D 4F2  488
G.Fbird2    L  002000 004700 00274C 4CA  488
G.Jcb       L  FF1900 FF8023 000140 4C8  488
G.Jcb2      L  003000 0063CC 003438 493  488
G.Horse     L  001900 001900 000900 48A  488
G.Race      L  001900 001900 000E00 47A  257
G.GGs       L  004B50 004B50 0024BB 455  257
G.Dallas    L  FF1900 FF8023 0034CF 420  257
G.Bumble    L  002000 004100 00212A 3FE  257
G.Guy1      L  001300 00801F 000A23 3F3  257
G.Guy2      L  001900 00801F 00162D 3DC  257
G.Monster   L  000E00 000E00 000700 3D5  257
G.Mon3      L  000E00 000E00 002600 3AF  257
G.Mon2      L  FF1900 FF8023 0002CF 3AC  257
Rgds
Stephen

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 24, 2015 2:49 pm

lurkio wrote:Yes, but fortunately the Beeb filename "Game??" seems to be preserved inside the Game__.inf file and on the generated SSD, which is exactly the result we want. (But I can see how there could be circumstances in which certain chars in Beeb filenames might still cause conflicts. Hopefully those circumstances are unlikely [-o< )

If you find such a scenario then please let me know... There _shouldn't_ be any cases :-) The INF files are essential for keeping state that can't be kept on a Unix filesystem (characters such as / can't be in a Unix filename; load/exec/LOCK aren't represented). I didn't create the INF concept; I found it in 'bbcim' from W.H.Scholten and decided to copy it as an existing standard :-)
Rgds
Stephen

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 24, 2015 2:55 pm

jgharston wrote:DFS's *INFO output is a rigidly fixed format:
(1-character directory).(filename padded to 9 spaces)(L or space)(2 spaces)(6-digit load address)(space)(6-digit exec address)(space)(6-digit length)(space)(3-digit start sector)
What's the output of your tool's *INFO equivalent?

The code segment that does this:

Code: Select all

  foreach (sort { $files{$b}{start} <=> $files{$a}{start} } keys %files)
  {
    next unless $files{$_}{name};
    my $n=$files{$_}{name} . (" "x10); $n=substr($n,0,10);
    printf "%10s  %s  %06X %06X %06X %03X",
                          $n,
                          ($files{$_}{locked}?"L":" "),
                          $files{$_}{load},
                          $files{$_}{exec},
                          $files{$_}{size},
                          $files{$_}{start};
    printf("  %03X",$files{$_}{cat_sector}) if $chain;
    print "\n";
  }

So skip and discard up to /^Filename:/
Then for each line
the first 10 characters are a space-padded filename (including dir and ".")
then two spaces,
then the lock-flag (L or space)
then two spaces,
then a 6-digit hex load
then one space
then a 6-digit hex exec
then one space
then a 6-digit hex size
then one space
then a 3-digit hex start sector
THEN OPTIONALLY two spaces followed by a 3-digit hex catalogue sector
Rgds
Stephen

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Sun May 24, 2015 4:10 pm

sweh wrote:
lurkio wrote:But I can see how there could be circumstances in which certain chars in Beeb filenames might still cause conflicts.
If you find such a scenario then please let me know... There _shouldn't_ be any cases :-)

I was thinking of this sort of thing:
1.png

But it looks like you thought of that!

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Sun May 24, 2015 7:21 pm

I've hacked the code to add titles to all the SSDs (currently just the name of the first file; not a great title, but it's at least semi-meaningful and guaranteed to fit!).

I've also added support for all directories. I think. This is the first time I've done any PHP, so there are probably bugs, although I've processed Disc030 and haven't found any problems yet. I'll do some more tests and see.

[EDIT: I've now updated the following listing several times, in place. For details, see the following posts in the thread.]

Code: Select all

<?php

define('VERSION','0.12');

define('PERL','/usr/bin/perl');
define('BEEB','/*REPLACE_ME*/mmb_utils/beeb');

define('DSD_DIR','/*REPLACE_ME*/DSD');
define('SSD_DIR','/*REPLACE_ME*/SSD');
define('NEW_SSD_DIR','/*REPLACE_ME*/New_SSD');

class Game {
    public $Name;
    public $Side=0;
    public $FirstFile;
    public $FilesDirectory;
    public $Files=array();
}

# Create a blank template for the new SSDs
$result=exec(PERL.' '.BEEB.' blank_ssd SSDtemplate',$output,$rc);
$result=exec(PERL.' '.BEEB.' opt4 SSDtemplate 3',$output,$rc);

# Grab a DSD and process it
foreach (glob(DSD_DIR.DIRECTORY_SEPARATOR.'*.dsd') as $dsd_path) {
    echo "$dsd_path\n";

    $pp=pathinfo($dsd_path);

    $fn=$pp['filename'];

    # Step 1 - split out the two SSDs
    $ssd0=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_0.ssd";
    $ssd2=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_2.ssd";
    $result=exec(PERL.' '.BEEB." split_dsd $dsd_path $ssd0 $ssd2",$output,$rc);

    # Step 2 - get all files (we need !BOOT first, but we'll need
    # all the other files anyway, to build the separate SSD files)
    $ssd0d=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_0";
    $ssd2d=SSD_DIR.DIRECTORY_SEPARATOR."{$fn}_2";
    $output=array();
    $result=exec(PERL.' '.BEEB." getfile $ssd0 $ssd0d",$output,$rc);

    # Map Beeb filenames to Unix filenames
    $fmap=array();
    foreach($output as $o) {
        if (preg_match('/^Saving (.+) as (.+)$/',$o,$matches)) {
            $fmap['0:'.$matches[1]]=$matches[2];
        }
    }
    $output=array();
    $result=exec(PERL.' '.BEEB." getfile $ssd2 $ssd2d",$output,$rc);
    foreach($output as $o) {
        if (preg_match('/^Saving (.+) as (.+)$/',$o,$matches)) {
            $fmap['2:'.$matches[1]]=$matches[2];
        }
    }
    #print_r($fmap);

    $cat_0=array();
    $cat_2=array();

    # Step 3 - get the catalogues in the order we need them
    $result=exec(PERL.' '.BEEB." info $ssd0",$cat_0,$rc);
    $result=exec(PERL.' '.BEEB." info $ssd2",$cat_2,$rc);

    if ($cat_0) {
        $games=array();
        # Step 4 - read the !BOOT (BASIC) file from side 0 - it will
        # have the menu program. Find the DATA lines and process them
        $pling_boot=array();
        $result=exec(PERL.' '.BEEB." list $ssd0d/!BOOT",$pling_boot,$rc);
        if ($pling_boot) {
            #print_r($pling_boot);
            $side0_data_found=false;
            $side2_data_found=false;
            foreach ($pling_boot as $line) {

                if (preg_match('/^.*?[0-9]DATA.*/',$line)) {
                    if ($side0_data_found==false) {
                        # Must be first title on side 0
                        #echo "Side 0 data starts with $line\n";
                        $side0_data_found=true;
                        $games[]=make_game(0,$line,$ssd0d);
                    } else {
                        if ($side2_data_found==false) {
                            #echo "side 0 title $line\n";
                            $games[]=make_game(0,$line,$ssd0d);
                        } else {
                            #echo "side 2 title $line\n";
                            $games[]=make_game(2,$line,$ssd2d);
                        }
                    }
                }

                if (preg_match('/.*?0REM.*/',$line)) {
                    if ($side0_data_found) {
                        #echo "end of side 0? $line\n";
                        $side2_data_found=true;
                    }
                }
            }
           
            # Make a list of the first files of the games on each disc side
            $firsts=array();
            $firsts[0]=array();
            $firsts[2]=array();

            # Step 5 - we have the list of games, so let's work out
            # the files to go into the SSD
            if ($games) {
                $gamescount=count($games);
                #echo "Count $fn - $gamescount\n";
                for ($i=0;$i<$gamescount;$i++) {
                    array_push($firsts[$games[$i]->Side],
                               strtoupper($games[$i]->FirstFile));
                }
                #print_r($firsts);

                foreach ($games as $game) {
                    echo "Processing $game->Name first file $game->FirstFile\n";
                    if ($game->Side==0) {
                        $cat_to_search=array_reverse($cat_0);
                    } else {
                        $cat_to_search=array_reverse($cat_2);
                    }
                    # Last few rows are metadata, not files
                    $cat_to_search = array_slice($cat_to_search,0,-4);

                    # And walk this array for files required
                    $gathering_files=false;
                    $files_found=array();
                    foreach ($cat_to_search as $cat_file) {
                        $cat_file=trim(substr($cat_file,0,12));
                        #echo "cat_file: $cat_file\n";

                        if (strtoupper($cat_file)=='$.'.strtoupper($game->FirstFile)
                            || strtoupper($cat_file)==strtoupper($game->FirstFile))
                        {
                            $gathering_files=true;
                            #echo "found first ($cat_file), gathering\n";
                        }
                        # For END-OF-DISC games we'll get them all, anyway
                        elseif (in_array(strtoupper(preg_replace('/^\$\./','',$cat_file)),
                                         $firsts[$game->Side]))
                        {
                            $gathering_files=false;
                            #echo "encountered next ($cat_file), stop gathering\n";
                        }
                       
                        if ($gathering_files) {
                            #echo "gathering $cat_file ";
                            $files_found[]=$cat_file;
                        }
                    }
                   
                    $game->Files=$files_found;
                    echo "\tFiles for game: ";
                    foreach ($game->Files as $f) {
                        echo "$f ";
                    }
                    echo "\n";

                    make_game_ssd($game);
                }
            }
        } else {
            echo "Cannot find !BOOT\n";
        }
    } else {
        echo "Cannot find side 0 catalogue\n";
    }
}

@unlink('SSDtemplate');
exit;

function make_game($side,$line,$local_directory) {
    # Split out line parts <space><line#>DATA<title>,<first_file>
    if (preg_match('/^[\s]{1,3}[0-9]{1,3}DATA(.*?),([^,]*)$/',$line,$matches)) {
        $game=new Game();
        $game->Name=$matches[1];
        $game->Side=$side;
        $game->FirstFile=$matches[2]; 
        $game->FilesDirectory=$local_directory;
        return $game;
    }
}

function make_game_ssd($game) {
    # So build a .SSD based on the game name, with a !BOOT
    #beeb blank_ssd $DSK
    #beeb opt4 $DSK 3
    #beeb title $DSK "Games Disk"
    #beeb putfile $DSK OBJS/*
    #beeb access $DSK '*.*' L

    global $fmap, $fn;

    # Create and title the new SSD
    $ssd_name=$fn.'-'.preg_replace('/[^0-9A-Z]/i','',$game->Name).'.ssd';
    $new_ssd_path=NEW_SSD_DIR.DIRECTORY_SEPARATOR.$ssd_name;
    echo "$new_ssd_path\n";
    @unlink($new_ssd_path);
    copy('SSDtemplate',$new_ssd_path);
    $result=exec(PERL.' '.BEEB." title $new_ssd_path '$game->FirstFile'",$output,$rc);

    # Build the !BOOT
    file_put_contents('/tmp/!BOOT',"*BASIC\r*FX21\rCLOSE#0:CHAIN \"$game->FirstFile\"\r");

    # Write the game files to the new SSD
    $src='';
    foreach ($game->Files as $gf) {
        $s=$game->FilesDirectory.DIRECTORY_SEPARATOR.$fmap[$game->Side.':'.$gf];
        echo "$s\n";
        $src=$src.$s.' ';
    }
    $result=exec(PERL.' '.BEEB." putfile $new_ssd_path /tmp/!BOOT $src",$output,$rc);

    # Lock all the files on the disc
    $result=exec(PERL.' '.BEEB." access $new_ssd_path '*.*' L",$output,$rc);
}
?>
Last edited by lurkio on Fri May 29, 2015 8:11 pm, edited 14 times in total.

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sun May 24, 2015 7:38 pm

lurkio wrote:This is the first time I've done any PHP
=D> Welcome to the dark side, young Jedi. :lol:

Maybe bump up the version number on line 3? Just so that we can keep track of versions. And if it gets to be really useful, I'm sure someone will tell us to put it under version control e.g. github. But let's get it finished first, eh?

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Sun May 24, 2015 7:41 pm

richardtoohey wrote:
lurkio wrote:This is the first time I've done any PHP
=D> Welcome to the dark side, young Jedi. :lol: Maybe bump up the version number on line 3?

Your wish is my command, master Darth Toohey. :evil:

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sun May 24, 2015 7:43 pm

Use the source, LukeLurkio. :lol:

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Thu May 28, 2015 12:44 am

I've updated the script, above, with some optimisations that Stephen suggested.

On my machine, it now takes less than 2 mins to run through all thirty of the DSDs that Mick's remastered so far. Previously, it was taking over 10 mins.

(I've also fixed a regexp bug that was causing Run Silent, Run Deep on disc 22 to fail.)

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Thu May 28, 2015 1:06 am

The old version took about 7 minutes on my machine, and from dipping into a few SSDs, all good so far.

I'll try the new version now.

EDIT:

Code: Select all

PHP Notice:  Undefined variable: src in process_mb_dsd.php on line 208
You can declare the variable to fix this notice - so before you enter the loop, something like:

Code: Select all

$src='';
Doesn't affect the running of the script.

Also you've put your paths in the 0.07 version (in the defines at the top) - you removed those in previous versions.

New version runs in 3 minutes; produces 540 SSD files from 33 DSDs. :D
Last edited by richardtoohey on Thu May 28, 2015 1:14 am, edited 2 times in total.

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Thu May 28, 2015 1:08 am

I can think of one potential bug...

Code: Select all

    $output=array();
    $result=exec(PERL.' '.BEEB." getfile $ssd0 $ssd0d",$output,$rc);
    $result=exec(PERL.' '.BEEB." getfile $ssd2 $ssd2d",$output,$rc);

    # Map Beeb filenames to Unix filenames
    $fmap=array();
    foreach($output as $o) {
        if (preg_match('/^Saving (.+) as (.+)$/',$o,$matches)) {
            $fmap[$matches[1]]=$matches[2];
        }
    }

What happens if there's a file "$.HELLO" on both sides of the disk? Won't the second entry over-ride the first?
Rgds
Stephen

User avatar
lurkio
Posts: 1292
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Thu May 28, 2015 1:57 am

richardtoohey wrote:

Code: Select all

PHP Notice:  Undefined variable: src in process_mb_dsd.php on line 208
You can declare the variable to fix this notice - so before you enter the loop, something like:

Code: Select all

$src='';
sweh wrote:I can think of one potential bug... What happens if there's a file "$.HELLO" on both sides of the disk? Won't the second entry over-ride the first?

Fixed both bugs. See version 0.08 above.

User avatar
tricky
Posts: 1923
Joined: Tue Jun 21, 2011 8:25 am
Contact:

Re: Automating the creation of SSD's

Postby tricky » Thu May 28, 2015 6:10 am

Am I looking in the wrong thread, or are there 100 DSDs?
Is anyone likely to make these SSDs available, or as a last resort, can anyone suggest a simple way to get this running on windows? (I once got one python app to run, but have never got any perl running, I usually end up rewriting it in C/C++!

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Thu May 28, 2015 6:18 am

Yes, the plan is to make the SSDs available. From the first post:
leenew wrote:This will form a new section of the archive, much like the STH disc SSD archive.
Lee was faced with converting 100+ DSDs manually and found the prospect errr "uninviting".

So we've been working towards an automated solution, and I think we're pretty much there.

So the first 30 DSDs take about 2 or 3 minutes to convert into 540 SSDs.

Not sure what the end-plan was of exactly where to upload & host them, mind you? :-k


Return to “archive issues”

Who is online

Users browsing this forum: No registered users and 1 guest