Automating the creation of SSD's

having trouble with an archived file or want to correct something? report it here!
User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Thu May 28, 2015 10:25 am

This is looking really good now. It is going to save me a shed load of work :D
One thing for Mick (If you are reading this)
Disc 013 side 0 has Wizadore and Neutron files on the disc in the wrong order. (Each game has files in the right order but Neutron comes before Wizadore and it should be after) This will obviously cock up the SSD creation.
Do you want to re-post a corrected version Mick?
Lee.

User avatar
lurkio
Posts: 1151
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Thu May 28, 2015 11:22 am

tricky wrote:Am I looking in the wrong thread, or are there 100 DSDs?

I was going by this thread, where Mick is posting the final remastered versions of his "30th Anniversary" games DSDs. He's up to Disc 30 to date. (In that same thread, Lee has posted the first eight DSDs converted to individual game SSDs.)

leenew wrote:This is looking really good now. It is going to save me a shed load of work :D

Yay!

leenew wrote:Disc 013 side 0 has Wizadore and Neutron files on the disc in the wrong order ... This will obviously cock up the SSD creation.

Actually, it doesn't, for some reason! (I think it's because Richard's script will produce the right output so long as the files for each game are contiguous on the disc and each game's "first file" appears somewhere in the DATA section of the !BOOT.)

richardtoohey wrote:Not sure what the end-plan was of exactly where to upload & host them, mind you? :-k

I'm probably not the best person to communicate this, because I'm not sure of the details (but that's never stopped me before).

As I understand it, a Grand Unified Acorn Preservation Project (GUAPP) is in the early stages of planning and preparation. (Although I'm not sure of the project's exact scope, so I probably shouldn't have begun that name I just dreamt up with the word "Grand". Or "Unified". (See? I told you I shouldn't be communicating this.))

Getting Mick Brown's DSDs split into per-game SSDs was actually meant to be the second stage of the prep work, I think. But Richard had already started working on the script, so I just kept on updating it (i.e. slowly ruining it) to try to get all the features we needed working.

The actual first stage involves tidying up all the disc and tape images in the StairwayToHell archive. Mainly getting filenames consistent, but probably also some other tasks that I've forgotten because I stupidly failed to take any notes. This is what I'm supposed to be working on -- initially to come up with filenaming conventions and directory structure. I think.

Wiser heads will jump in to correct/chide me, I'm sure.

User avatar
richardtoohey
Posts: 3354
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Thu May 28, 2015 7:43 pm

lurkio wrote:I'm supposed to be working on
Tsk, tsk, and look at how easily you've been distracted. :wink: Never happens to me! :^o :^o :^o Now what was I meant to be doing? :lol:

User avatar
lurkio
Posts: 1151
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Thu May 28, 2015 7:53 pm

richardtoohey wrote:Tsk, tsk, and look at how easily you've been distracted. :wink: Never happens to me!

I believe you. Btw, I've got this game I need converting for the Master... :wink: :lol:

I'm still finding weird anomalies in the PHP script -- at least when it runs on my system (PHP 5.5.14 on OS X 10.10). The funniest one is this error:

Code: Select all

Processing Up Up And Away first file UP&AWAY
   Files for game: $.UP&AWAY
/Users/desaic/toohey/New_SSD/UpUpAndAway.ssd
sh: AWAY: command not found

It doesn't seem to affect the generation of the SSD, but it's baffling because it seems that the shell is for some reason interpreting the ampersand as a run-in-background operator (I think??) and then assuming that "AWAY" is the next command! But my question is why is the shell interpreting the output of the echo statement as a command at all??!

Code: Select all

echo "Processing $game->Name first file $game->FirstFile\n";

When, just for testing, I hardcoded the first-file name UP&AWAY into that line of code above, it still threw the same error!

And FWIW, when I knock up a very simple PHP script that does nothing but literally echo the same text, it runs without error!

I think I might just have to strip out all shell chars from the printed string if I can't figure out what the actual problem is.

EDIT: I don't think it's the echo statement that's causing the error after all. If I comment out all relevant echos, the error still occurs! So I think it's something to do with the mmb_utils.

I stumbled over this bug while I was trying to test for a different bug: I think that if there are two games with the same name on different DSDs, the second might overwrite the first.

User avatar
sweh
Posts: 1833
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Thu May 28, 2015 8:21 pm

lurkio wrote:

Code: Select all

Processing Up Up And Away first file UP&AWAY
   Files for game: $.UP&AWAY
/Users/desaic/toohey/New_SSD/UpUpAndAway.ssd
sh: AWAY: command not found

It doesn't seem to affect the generation of the SSD, but it's baffling because it seems that the shell is for some reason interpreting the ampersand as a run-in-background operator (I think??) and then assuming that "AWAY" is the next command! But my question is why is the shell interpreting the output of the echo statement as a command at all??!

It's possibly the "beeb title" command, perhaps, because that's being passed to the shell. Does the SSD just have the title "UP" ?

If so then your PHP code is doing the equivalent of

Code: Select all

sh -c "beeb title foo.sdd UP&AWAY"

and so that's where the bug is.

You should quote the title before passing it to the shell.

(It's clearly not the mmb_utils that have the bug because it's "sh" that's throwing the error message, which indicates it's how you're calling the shell that's at fault)
Rgds
Stephen

User avatar
lurkio
Posts: 1151
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Thu May 28, 2015 8:32 pm

sweh wrote:It's possibly the "beeb title" command, perhaps, because that's being passed to the shell. Does the SSD just have the title "UP" ? If so then your PHP code is doing the equivalent of

Code: Select all

sh -c "beeb title foo.sdd UP&AWAY"
and so that's where the bug is. You should quote the title before passing it to the shell. (It's clearly not the mmb_utils that have the bug because it's "sh" that's throwing the error message, which indicates it's how you're calling the shell that's at fault)

You just beat me to it! Yes, that was it. I was missing the single quotes I've now added around $game->FirstFile here:

Code: Select all

    $result=exec(PERL.' '.BEEB." title $new_ssd_path '$game->FirstFile'",$output,$rc);

#-o

v0.09 is here.

EDIT: Updated to v0.10 because this regexp was missing DATA line numbers that didn't end in zero:

Code: Select all

if (preg_match('/.*?0DATA.*/',$line)) {


Still looking into the bug where if there are two games with the same name on different DSDs, the second overwrites the first.

User avatar
sweh
Posts: 1833
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Thu May 28, 2015 10:30 pm

lurkio wrote:Still looking into the bug where if there are two games with the same name on different DSDs, the second overwrites the first.

Suggestion: prefix the per-game SSD name with the DSD name; so instead of UP&AWAY.SSD create Games030-UP&AWAY.SSD (or whatever).
Rgds
Stephen

User avatar
lurkio
Posts: 1151
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Thu May 28, 2015 10:59 pm

sweh wrote:Suggestion: prefix the per-game SSD name with the DSD name; so instead of UP&AWAY.SSD create Games030-UP&AWAY.SSD (or whatever).

Yes, that seems to be the simplest solution. Done.

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Fri May 29, 2015 9:02 am

Hmmm.... One week's hard work with my eyes burning out of my head has turned into one button press and a 2 and a half minute wait :-k
Personally, I prefer the new way :lol:

Sweh, Richard, Lurkio, I cannot thank you all enough, so this will have to do... =D>

Lee.

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Fri May 29, 2015 11:16 am

Now I have seen the power of modern languages and utilities, I wonder if we can take it to the next step?....
This is just a thought at the moment, but from what I have seen, maybe it will be do-able [-o< .
When Mick finally uploads all of his discs, and the new script processes them, I will have thousands of instantly created SSD's which is brilliant.

The next job I was going to have to do was create subfolders of Publishers, e.g. Acornsoft, Superior Software, Micro Power etc. etc. etc. and place all of the SSD's in their respective publisher subfolder.
BUT! wait a minute! I already have that information in the spreadsheet I created a while back.
Here it is, (just the portion relating to Mick's discs):
MB Discs.zip
(76.83 KiB) Downloaded 33 times


So, the logical next step for this wonder-script, is instead of churning out thousands of games into one folder, look at the spreadsheet for the disc in question and chuck the resulting SSD in the correct subfolder :D

BANG! Instant archive :D

Lee.

User avatar
richardtoohey
Posts: 3354
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Fri May 29, 2015 11:24 am

leenew wrote:BANG!
That's you getting shot for bringing out more requirements down the track, after the release. :lol:

You can export the sheet as a CSV, gobble the required columns and do this sort of thing, yes. Obviously relies on disc numbers and game titles being exactly the same in both spreadsheet and files.

Or with the recent changes to the file names, it might be possible do as post-processing - i.e. generate all the SSDs and then move them into appropriate folders. Not as efficient but keeps the two jobs separate.

chrisn
Posts: 295
Joined: Sat Apr 19, 2014 11:31 am
Location: UK

Re: Automating the creation of SSD's

Postby chrisn » Fri May 29, 2015 11:36 am

Is there an updated copy of the spreadsheet available? I know that Mick occasionally moves games between disks from time to time, and I think the copy of the spreadsheet I have will be out of date.

I convert all Mick's dsds to ssds and import them into a BEEB.MMB file, and I've been thinking of using the spreadsheet to automate creating a menu program for it.

Chris

User avatar
Elminster
Posts: 1632
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK

Re: Automating the creation of SSD's

Postby Elminster » Fri May 29, 2015 11:42 am

leenew wrote:Now I have seen the power of modern languages and utilities, I wonder if we can take it to the next step?....


And that will get you into another kettle of snails. Best language or scripting language to use hours of debate there. (I know PHP get a huge amount of flak)

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Fri May 29, 2015 11:57 am

chrisn wrote:Is there an updated copy of the spreadsheet available? I know that Mick occasionally moves games between disks from time to time, and I think the copy of the spreadsheet I have will be out of date.

I convert all Mick's dsds to ssds and import them into a BEEB.MMB file, and I've been thinking of using the spreadsheet to automate creating a menu program for it.

Chris


Chris, look 4 posts up, I have posted Micks portion of the spreadsheet. Or do you need the full one?
If so, here it is:
BBC Disc And Tapes.zip
(196.48 KiB) Downloaded 36 times

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Fri May 29, 2015 11:59 am

richardtoohey wrote:
leenew wrote:BANG!
That's you getting shot for bringing out more requirements down the track, after the release. :lol:


He He, If only I had known this was possible beforehand Richard! I would have spelled out my requirements much more clearly :lol:

In a nutshell... I want...the world! :wink:

Lee.

chrisn
Posts: 295
Joined: Sat Apr 19, 2014 11:31 am
Location: UK

Re: Automating the creation of SSD's

Postby chrisn » Fri May 29, 2015 12:10 pm

leenew wrote:Chris, look 4 posts up, I have posted Micks portion of the spreadsheet. Or do you need the full one?

Hah, I missed that... thanks for this!

I was actually thinking of a spreadsheet that I think Mick had created himself, it had lots of tabs with detail about the status of each game, and changes he'd made. I can't seem to find the forum post where it came from now...

Chris

User avatar
lurkio
Posts: 1151
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Fri May 29, 2015 12:16 pm

It occurred to me that people might be interested to see the output from the latest version (0.11) of the PHP script, so I've uploaded it here:

https://s3.amazonaws.com/beeb/MickBrownSSDs.zip

576 SSDs derived from 32 of Mick's remastered "30th Anniversary" DSDs.
Last edited by lurkio on Fri May 29, 2015 8:23 pm, edited 2 times in total.

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Fri May 29, 2015 12:17 pm

There's this thread Chris: viewtopic.php?f=6&t=5782&hilit=sth143+excel

I did see Mick's spreadsheet a while ago, but I can't put my hands on it at the moment.
I am also not sure that it covered all 106 discs, but he may have completed it since I last saw it?..

Lee.

Michael Brown
Posts: 1860
Joined: Sat Apr 03, 2010 12:54 pm
Location: Nottingham

Re: Automating the creation of SSD's

Postby Michael Brown » Fri May 29, 2015 12:29 pm

Hi all,

If it helps you,
here is my excel spreadsheet (as of today).
It contains almost everything you could possibly want to know.

regards,
Mick.

Software.xls
(844.5 KiB) Downloaded 62 times

chrisn
Posts: 295
Joined: Sat Apr 19, 2014 11:31 am
Location: UK

Re: Automating the creation of SSD's

Postby chrisn » Fri May 29, 2015 12:35 pm

Thank you Mick and Lee ! =D> :D

Chris

User avatar
lurkio
Posts: 1151
Joined: Tue Apr 09, 2013 11:30 pm
Location: Doomawangara
Contact:

Re: Automating the creation of SSD's

Postby lurkio » Fri May 29, 2015 8:22 pm

This isn't what Lee was after, but I've updated the script to v0.12.

For each game, the script now starts with the game's first file (listed in the !BOOT) and works its way down the catalogue until it either hits the end of the disc-side or encounters the first file of any other game on that side of the disc.

In this way, even if the order in which the games are listed in the !BOOT doesn't match the order in which they appear on the disc, the script will still generate correct SSDs (so long as all the files for any particular game are contiguous (and don't interleave with files from other games), which is always the case on all of Mick's DSDs, I believe).

I had thought that the script was already doing this, but it turns out it wasn't. (The SSDs for Wizadore and Neutron from Disc013.dsd are now derived correctly.)

Here's a link to a zipped archive of all the SSDs derived by version 0.12 of the script from the first 32 DSDs in Mick's "30th Anniversary" games collection:
https://s3.amazonaws.com/beeb/MickBrownSSDsV012.zip
Last edited by lurkio on Sat Jul 08, 2017 11:17 am, edited 1 time in total.

User avatar
sweh
Posts: 1833
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 31, 2015 5:44 pm

So I said I wasn't going to do this, but I was bored...

(Needs a newer version of mmb_utils from 2015/05/30 to work properly; there's a copy in the attached zip file, or you can get it from the original website)

Code: Select all

% time ./split_mb_dsd.pl
Processing SRC/Disc001.dsd
Processing SRC/Disc002.dsd
Processing SRC/Disc003.dsd
Processing SRC/Disc004.dsd
Processing SRC/Disc005.dsd
Processing SRC/Disc006.dsd
Processing SRC/Disc007.dsd
Processing SRC/Disc008.dsd
Processing SRC/Disc009.dsd
Processing SRC/Disc010.dsd
Processing SRC/Disc011.dsd
Processing SRC/Disc012.dsd
Processing SRC/Disc013.dsd
Processing SRC/Disc014.dsd
Processing SRC/Disc015.dsd
Processing SRC/Disc016.dsd
Processing SRC/Disc017.dsd
Processing SRC/Disc018.dsd
Processing SRC/Disc019.dsd
Processing SRC/Disc020.dsd
Processing SRC/Disc021.dsd
Processing SRC/Disc022.dsd
Processing SRC/Disc023.dsd
Processing SRC/Disc024.dsd
Processing SRC/Disc025.dsd
Processing SRC/Disc025BeebEm.dsd
Processing SRC/Disc026.dsd
Processing SRC/Disc027.dsd
Processing SRC/Disc027BeebEm.dsd
Processing SRC/Disc028.dsd
Processing SRC/Disc029.dsd
Processing SRC/Disc030.dsd

real    0m3.19s
user    0m1.60s
sys     0m0.84s

% diff -r RESULTS Lurkio
%

So this code processes the 32 DSD in 3 seconds, and the resulting SSDs are identical to Lurkio's results that he got using the php script.

It assumes the source DSDs are in a SRC/ directory and puts the resulting SSDs into the RESULTS/ directory.

It basically uses the backend routines of the mmb_utils BeebUtils.pm file, and keeps all the data in memory. Se we don't need to create two SSDs from the DSD; we don't need to extract all the files from both SSDs, we don't need to LIST the !BOOT program and look for DATA statements, and so on. We load the DSD into memory, split it in memory, read the catalogue in memory, parse !BOOT in memory, create the new SSDs in memory... the only thing we need to write out are the finished game SSDs. This cuts down on disk I/O and other processing massively.

Enjoy :-)
Attachments
split_mb_dsd.zip
(31.1 KiB) Downloaded 39 times
Rgds
Stephen

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Sun May 31, 2015 6:28 pm

Wow!
My workload has gone from over one hour's processing per disc (me doing it manually), to 10 minutes to process 32 discs (Richard's original script), to 3 minutes to process 32 discs (Lurkio's improvements) to 3 SECONDS to process 32 discs with Stephen's in memory script!!! :lol:
Mick's full DSD collection of around 115 discs can now be split into 2070 individual SSDs in around 12 seconds!
That's progress 8)
Lee.

User avatar
1024MAK
Posts: 6676
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...

Re: Automating the creation of SSD's

Postby 1024MAK » Sun May 31, 2015 6:42 pm

Lee, it looks like you have almost made yourself redundant! :lol:

Mark
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk NOW!
BeebWiki‬ - for answers to many questions...

User avatar
leenew
Posts: 3330
Joined: Wed Jul 04, 2012 3:27 pm
Location: Doncaster, Yorkshire

Re: Automating the creation of SSD's

Postby leenew » Sun May 31, 2015 7:17 pm

1024MAK wrote:Lee, it looks like you have almost made yourself redundant! :lol:

Mark


:mrgreen: :shock: :? :wink: #-o

User avatar
richardtoohey
Posts: 3354
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sun May 31, 2015 10:05 pm

sweh wrote:So I said I wasn't going to do this, but I was bored...
I knew if we waited long enough, you'd have a quick look, just to see what was possible, no chance you'd actually do anything, oh no! ... :lol:

=D> =D> =D>

User avatar
sweh
Posts: 1833
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 31, 2015 10:10 pm

leenew wrote:Mick's full DSD collection of around 115 discs can now be split into 2070 individual SSDs in around 12 seconds!

Well, let's give that a try. We can copy the existing DSDs a few times to get a similar number.

OK, I have 132 DSDs...

It took 13 seconds and generated 2376 SSDs. So 12 seconds seems reasonable.
Rgds
Stephen

User avatar
sweh
Posts: 1833
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Automating the creation of SSD's

Postby sweh » Sun May 31, 2015 10:11 pm

richardtoohey wrote:
sweh wrote:So I said I wasn't going to do this, but I was bored...
I knew if we waited long enough, you'd have a quick look, just to see what was possible, no chance you'd actually do anything, oh no! ... :lol:

Really, I wasn't going to do this. I saw that people were using my utils and thought "huh, I wonder if we can use the library..." and started to write code. Then stopped 'cos, hey, the PHP code did the job and I didn't want to knock other people's work. And, besides, "not my problem" :-) But then Lurkio PM'd me and we had some back'n'forths (which is how the code got optimised down to 2 minutes). And that got me thinking about it again. A solution was running through the back of my mind.

And then, after my girlfriend went home, I thought... "eh, let's see what happens". And I spent an hour or so. I found that I'd not properly layered one function (add_file_to_ssd needed to be split into 'file handling' and 'ssd handling') and another routine (*opt4) should have been made a function. But other than that the BeebUtils.pm did pretty much all we could expect. The hardest part was parsing Mick's !BOOT program 'cos some lines have extraneous spaces at the end :-)

Now a real software architect will look at this and scream. The code requires far too much knowledge of the data structures the library creates. There's no proper layering. But, hey, it seems to work.

So you can blame Lurkio. I figured the PHP code was good enough (really, once Mick has finished his disks how often will the split routine be run? You can wait 2 hours!) but he got me thinking about the problem.

So now you have an over-optimised solution :-)
Rgds
Stephen

User avatar
richardtoohey
Posts: 3354
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Automating the creation of SSD's

Postby richardtoohey » Sun May 31, 2015 10:23 pm

sweh wrote:I thought... "eh, let's see what happens"
Fatal! :lol:

Anyone want to re-write in C to see if we can get it down to one second or less? ](*,)

Probably getting close to optimum already - a lot of the time now (wild assumptions ahead) is going to be I/O - reading the DSD and writing out SSD files. I'm sure you could optimize the code some more, but you'd be shaving microseconds off. Diminishing returns (and, as you say, pointless for this particular task!)

User avatar
rcook
Posts: 65
Joined: Fri May 22, 2015 4:43 pm
Location: Seattle, WA
Contact:

Re: Automating the creation of SSD's

Postby rcook » Mon Jun 01, 2015 6:28 pm

lurkio wrote:The actual first stage involves tidying up all the disc and tape images in the StairwayToHell archive. Mainly getting filenames consistent, but probably also some other tasks that I've forgotten because I stupidly failed to take any notes. This is what I'm supposed to be working on -- initially to come up with filenaming conventions and directory structure. I think.


Are there any plans to introduce any kind of geographic redundancy for key properties such as Stairway to Hell in the form of mirror sites so that we never have to worry about these valuable resources "disappearing" on us in the future? If someone had such a plan, then members of this forum might be willing to contribute server resources.
Proud parent to Acorn Electrons 06-ALA01-0003584 and 07-ALA01-0017948


Return to “archive issues”

Who is online

Users browsing this forum: No registered users and 1 guest