BeebSCSI 7_7

for bbc micro/electron hardware, peripherals & programming issues (NOT emulators!)
User avatar
danielj
Posts: 7075
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester
Contact:

Re: BeebSCSI 7_7

Post by danielj » Mon Mar 04, 2019 12:52 pm

Make a pull request to the github repo :)

d.

User avatar
KenLowe
Posts: 527
Joined: Mon Oct 18, 2004 4:35 pm
Location: Scotland
Contact:

Re: BeebSCSI 7_7

Post by KenLowe » Mon Mar 04, 2019 1:41 pm

Unfortunately, I'm no expert on Github!

I notice that the latest commit on Feb 21 was to 'Fix File transfer as they would have extra bytes inserted', which seems to be different to the fix mentioned by dp11. The last update to filesystem.c is more than a year ago.

Or am I missing something???

User avatar
BeebMaster
Posts: 2686
Joined: Sun Aug 02, 2009 4:59 pm
Location: Lost in the BeebVault!
Contact:

Re: BeebSCSI 7_7

Post by BeebMaster » Mon Mar 04, 2019 2:27 pm

This is a great news, as it has been a real inhibiting factor when trying to use BeebSCSI with Level 3 Econet file server. I'm looking forward to being able to try it out when the patch is available.

I wonder - should the data file not be opened for read/write access on a *MOUNT or *DIR :0 etc. and stay open until there is a *DISMOUNT or *BYE ?
Image

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Mon Mar 04, 2019 2:56 pm

I haven't made a pull request yet. As I've been playing to understand and fix the issue, so the code is not the neatest.

The previous commit I made was to fix the transfer program which transfers files from FAT to ADFS.

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Mon Mar 04, 2019 10:22 pm

Pull request done. Please test and report back.

User avatar
KenLowe
Posts: 527
Joined: Mon Oct 18, 2004 4:35 pm
Location: Scotland
Contact:

Re: BeebSCSI 7_7

Post by KenLowe » Mon Mar 04, 2019 10:46 pm

dp11 wrote:
Mon Mar 04, 2019 10:22 pm
Pull request done. Please test and report back.
Wow. That's quite a lot of changes! I now need to figure out how to compile it. I've found some guidance at:

https://www.domesday86.com/?page_id=439

but I need to find the time to figure it all out.
Last edited by KenLowe on Mon Mar 04, 2019 10:47 pm, edited 1 time in total.

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Tue Mar 05, 2019 6:53 am

dp11 - I appreciate the hard work and I'll look into the pull request shortly.

I'm a bit swamped with work from the ld-decode project at the moment, which is why I'm a bit slow on the BeebSCSI side of things as it's using up 110% of my hobby time just to keep up :)

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Tue Mar 05, 2019 7:17 am

It is quite a lot of changes, which is why I'm a little worried about missing something or making a mistake. I'm certain some of the flushes can be removed, but I want it to be safe. Also it might be worth keeping track of multiple open luns then copying between luns would also be fast. As I say please test.

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Sat Mar 23, 2019 3:43 pm

I've given the changes some testing and it doesn't seem to work correctly. The firmware is unable to MOUNT any other LUNs; so it works fine until you attempt a *MOUNT to switch to another LUN - then only power-cycling will bring it back to functioning.

Looking at the debug traces I can see the following:

File system: filesystemCheckLunDirectory(): ERROR: f_opendir returned FR_TOO_MANY_OPEN_FILES

So, clearly, the changes aren't correct yet. I'll try to have a look into what's happening to see if it's possible to make the push request work correctly.

dp11> Can you try the same test on your board and confirm that you get the same issue?

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Sat Mar 23, 2019 7:02 pm

I'm away from hardware at the moment.

Can you try adding filesystemFlush to the beginning of filesystemCheckLunDirectory .

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Sat Mar 23, 2019 9:36 pm

I've added the additional flush (seems to work now) and moved the other flush commands down a little (so it only flushes if the initial error check is successful) - I've also bumped the firmware version reporting to 2.05

I tested, formatting (using SuperForm), mounting, SCSI jukebox and also ran it through a read/write speed test. All seems to pass.

Clone the dev-0311 branch and give it a whirl; if it looks good then I'll merge it and produce a .BIN for SD card bootloader upgrades.

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Sun Mar 24, 2019 8:08 am

Ok, I've now finished testing and fixed a few things here and there... My testing on both ADFS and VFS now works flawlessly :)

I've merged the dev branch and prepared a new release. You can get the 2.5 version from here:

https://github.com/simoninns/BeebSCSI/releases/tag/V2.5

The release files include the .BIN file necessary for SD Card flash upgrades (so there's no need to build the firmware yourself). There's also a new .adl floppy image containing the updated BSTrans (which I've also updated in the quick-start LUN image).

Upgrade at will (and let me know if there are any issues) :)

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Mon Apr 01, 2019 3:50 am

Anyone verified that the filestore problem is solved with the new build? It would be good to get a little feedback beyond my own testing :)

User avatar
BeebMaster
Posts: 2686
Joined: Sun Aug 02, 2009 4:59 pm
Location: Lost in the BeebVault!
Contact:

Re: BeebSCSI 7_7

Post by BeebMaster » Mon Apr 15, 2019 10:52 am

I did some testing over the weekend with firmware 2.5, and it's mixed results.

One part of the problem has definitely gone, which is that there is no longer any slow-down when sequentially stepping through a disc "image" from start to finish:
BeebSCSI2-5Test4.png
BeebSCSI2-5Test6.png
This enables a mount time of under 30 seconds for a 512MB drive in the Level 3 file server, compared to about a minute using a CF/IDE drive or many minutes in the previous BeebSCSI firmware.

However, accessing a location "far out" in the disc image is still as slow. This is a repeat read from a far sector, and it takes about a second to complete each read (compared to multiple reads per second in the stepping example):
BeebSCSI2-5Test7.png
BeebSCSI2-5Test9.png
If anything I think that may be slower than I was experiencing before.

Also I have noticed a recurrence of the problem I had a while back where BeebSCSI seems to suddenly stop being able to access the disc at all. This manifests itself as a Disc error 2C, and can only be solved by powering off the BeebSCSI and back on again.

For some reason I can only get this to happen sporadically when trying to read from the file server using a 32-bit machine:

BeebSCSI2-5Test25.png
BeebSCSI2-5Test27.png
It doesn't do it using the 8-bit machine in "user mode" from ADFS, and it doesn't do it when accessing the file server from another 8-bit machine.

I don't know what to make of that, until I can get a clear picture of when and why it happens I can't really comment further. I think what I'd like to do is borrow someobdy else's BeebSCSI from a different production batch just to try to eliminate some hardware fault in my 2 mini-boards.
Image

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Mon Apr 15, 2019 11:52 am

The first issue is probably due to the file server making multiple requests or sub-dividing the SCSI requests in some way.

The second could be anything, from a BeebSCSI issue to a bug in the file server (I would guess that the file server is sending a STOP command to the LUN - which would cause BeebSCSI to dismount the drive... which is what it should do - an IDE drive wouldn't do this since it isn't SCSI and ignores just about everything other than read and write commands).

In both cases it's impossible to know what's actually happening without the debug output from the BeebSCSI. Would it be possible to perform the tests again, but this time capture the debug log so it can be debugged?

User avatar
BeebMaster
Posts: 2686
Joined: Sun Aug 02, 2009 4:59 pm
Location: Lost in the BeebVault!
Contact:

Re: BeebSCSI 7_7

Post by BeebMaster » Mon Apr 15, 2019 3:20 pm

simoni wrote:
Mon Apr 15, 2019 11:52 am
The first issue is probably due to the file server making multiple requests or sub-dividing the SCSI requests in some way.
Just to be clear: this is BeebSCSI being accessed by ADFS using OSWORD &72; the file server isn't running. It's just a loop of OSWORD &72 reads from the same start sector each time.

I'd guess that problem is still related to the way FAT opens/closes/seeks the "LUN" files on the SD card.
simoni wrote:
Mon Apr 15, 2019 11:52 am
The second could be anything, from a BeebSCSI issue to a bug in the file server (I would guess that the file server is sending a STOP command to the LUN - which would cause BeebSCSI to dismount the drive... which is what it should do - an IDE drive wouldn't do this since it isn't SCSI and ignores just about everything other than read and write commands).
I need to do more tests to see if I can narrow down the hardware/software causing this problem, since it only appears to be happening on RISC PC with Econet module podule when trying to load a large-ish file. I'll have to try different 32-bit machine, different Econet hardware etc. before I can start to draw any conclusions. Initially on what I've seen, all I can think is that something on the Archimedes Econet side is accidentally sending too much data back to the file server on occasion which is having the effect of either overwriting the server machine's ADFS workspace, confusing BeebSCSI into thinking "nothing is there", or spuriously getting a stop-unit command through to the 1MHz bus, which as you say, wouldn't have any effect on an IDE drive setup.

I can try all these tests again and log the debug output, but I can't do that immediately as my BeebSCSI boards don't have the output port header, and then I will need some sort of cable. Is it serial or USB? I don't have a working PC setup with a serial interface handy just now (but could organise one given a bit of time).
Image

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Mon Apr 15, 2019 3:35 pm

If your code is repeatedly requesting a single 256 byte sector, it's going to be slow. It's also quite an unrealistic scenario since each read can pull back a couple of hundred sectors in a single call. You are basically driving the FAT stack in a manner which it isn't designed to be driven. It could be optimised by caching the single sector response but, since it would only help when a piece of code is requesting the same sector again and again, it would be like VW's emission test software - only useful when someone runs that test :)

As for the serial output, you need a pin header connected to a TTL to USB serial converter like the Arduino USB2Serial board; there are a ton of clones on ebay, so it's a very cheap board to source.
Last edited by simoni on Mon Apr 15, 2019 3:36 pm, edited 1 time in total.

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Mon Apr 15, 2019 5:54 pm

Just as another data point. My quick test of Pi1MHz appears not to have a slow down ( I might have made a mistake)

My SDCARD might be quicker than yours as I see 11 reads per second or the difference is down to CPU speed difference.

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Mon Apr 15, 2019 6:02 pm

NB just for reference using TIME while ADFS access occurs is not accurate as IRQs are missed. TIME$ is accurate as that comes from the RTC.

The speed test on Domesday86 for BeebSCSI is likely to be incorrect.
Last edited by dp11 on Mon Apr 15, 2019 6:07 pm, edited 1 time in total.

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Tue Apr 16, 2019 3:53 am

The speed test code on domesday86.com is very simple, it also measures the total operation time (as opposed to just the data transfer speed). The data transfer speed is much higher than quoted, but it's not really a practical measurement. Since BeebSCSI is open-source, I didn't feel the need to 'bump' the results to make it look good - under normal use it is much faster than IDE/CF solutions anyway (around >3x better).

It's interesting that the Pi1MHz results are not the same, since it's the same code really; Could be the SD card; performance can vary wildly with different brands and types. Once the schematics and code for the Pi1MHz are published then I'll build one too for comparison.

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Tue Apr 16, 2019 6:41 am

simoni wrote:
Tue Apr 16, 2019 3:53 am
The speed test code on domesday86.com is very simple, it also measures the total operation time (as opposed to just the data transfer speed). The data transfer speed is much higher than quoted, but it's not really a practical measurement. Since BeebSCSI is open-source, I didn't feel the need to 'bump' the results to make it look good - under normal use it is much faster than IDE/CF solutions anyway (around >3x better).
Unortunately as it uses TIME to time adfs access it will be artificial fast as while adfs access occur TIME looses time. My rough measurements show around 30% of interrupts are missed. So for example if a program takes 1minute to run (timed using a stop watch) Time will report 40seconds ish.

User avatar
simoni
Posts: 463
Joined: Wed May 25, 2016 6:18 pm
Contact:

Re: BeebSCSI 7_7

Post by simoni » Tue Apr 16, 2019 7:13 am

Doesn't surprise me that TIME is inaccurate over long measurements (although, in this case, it's a very short measurement, so the inaccuracy should be less on average).

My point was really that there is a difference between the overall operation speed and the actual data transfer speed (the 'speed test' code tests the operation speed - it's not the transfer speed (which is in the region of 122Kbytes/sec with BeebSCSI if you want to use a 'GoSDC' style measurement)).

In reality, BeebSCSI (or GoSDC for that matter) can't realistically provide that sort of speed under any practical usage - which is what I was talking about in my previous post (about 'bumping' results).

I'm not sure that it's really that important but, if anyone wants to write a more accurate tester, I'd happily include the code in the documentation.

Actually, since ADFS is the limiting DMA transfer factor, it would be interesting to see how the same code performs against the Pi1MHz port. The Pi's ARM processor is much (much) faster than the 8-bit AVR, so there should be an improvement in overhead (but not for the actual SCSI sector data transfer). Any difference would be just due to overhead communication processing time.

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Tue Apr 16, 2019 8:00 am

simoni wrote:
Tue Apr 16, 2019 7:13 am
Doesn't surprise me that TIME is inaccurate over long measurements (although, in this case, it's a very short measurement, so the inaccuracy should be less on average).
My guess is the inaccuracy is the same per unit of transfer as the same number of IRQs will be missed for each transfer unit give or take. I just scaled to a minute as that is easy to spot with a stopwatch in my test and the human error in the measurement is about 1%

dp11
Posts: 918
Joined: Sun Aug 12, 2012 8:47 pm
Contact:

Re: BeebSCSI 7_7

Post by dp11 » Wed Apr 17, 2019 7:56 pm

The another difference in the code between beebSCSI and Pi1MHz is that I use FatFS R0.13c which is the latest.

Post Reply