Slow OSBGET file access

discussion of beeb/electron applications, languages, utils and educational s/w
User avatar
hjalfi
Posts: 70
Joined: Sat May 13, 2017 10:17 pm
Location: Zürich, Switzelrand
Contact:

Slow OSBGET file access

Postby hjalfi » Sun Sep 10, 2017 12:39 pm

I've been adding file I/O support to Cowgol (see the Projects forum). It all works fine, but file streams seem to be really slow. This is surprising, as one of the awesome things about the BBC MOS is the really, really fast disk system.

The core of my test program (full thing: https://github.com/davidgiven/cowgol/bl ... dspeed.cow) is:

Code: Select all

var fd: uint8 := file_openin(&arg[0]);
print("File handle: 0x");
print_hex_i8(fd);
print_newline();

var bytes: uint32 := 0;
var before: uint32 := gettime();

while file_eof(fd) == 0 loop
    var c: int8 := file_getchar(fd);
    bytes := bytes + 1;
end loop;

var time: uint32 := gettime() - before;
print("Total bytes: 0x"); print_hex_i32(bytes); print_newline();
print("Total time: 0x"); print_hex_i32(time); print_newline();


(I can't do 32-bit division yet so that's left as an exercise for the reader.)

On ADFS and DFS I'm getting about 700 bytes per second. On VDFS in b-em it's about 1000 bytes per second. Bear in mind that Cowgol compiles into real (if not great) machine code, so there should be minimal language overhead.

So why is this so slow?

It occurs to me that I know nothing about how big the disk buffers are --- this is on a BBC Master. Does anyone know? Are they big enough to store an entire sector at a time? Would I get any benefit to doing user-space buffering and OSGBPB rather than OSBGET/OSBPUT (which is effectively the CP/M model)? Or is this just how fast a 2MHz 6502 is?

OSFILE whole-file-at-a-time gets several kilobytes a second. Of course, that's not a different problem because it doesn't have to indirect via a buffer and can just slap the data straight into memory from the NMI handler, but still...

philb
Posts: 97
Joined: Sat Aug 05, 2017 6:05 pm

Re: Slow OSBGET file access

Postby philb » Sun Sep 10, 2017 1:16 pm

The overhead of OSBGET certainly can be significant, though how significant depends on the filesystem. On NFS (not ANFS or ENFS) the throughput for unbuffered byte-at-a-time access can easily be down in single digits.

There are a variety of utilities around which will intercept OSBGET/OSBPUT and convert them into OSGBPB block transfers. Even a relatively small buffer size can give you a noticeable speedup. But yes, if you want good performance then it is best to use multiple byte transfers in your application whenever possible.

User avatar
hoglet
Posts: 6393
Joined: Sat Oct 13, 2012 6:21 pm
Location: Bristol

Re: Slow OSBGET file access

Postby hoglet » Sun Sep 10, 2017 1:16 pm

When I was working on 6502 Life last year, I discovered last year that OSGBPB is also very slow with DFS-like file systems:
http://www.stardot.org.uk/forums/viewto ... 35#p154135

Dave

SteveF
Posts: 429
Joined: Fri Aug 28, 2015 8:34 pm

Re: Slow OSBGET file access

Postby SteveF » Sun Sep 10, 2017 2:12 pm

Just to add some semi-concrete timing information (sadly in PLASMA not BASIC so I won't quote any code), writing 1000 copies of the string "1234567890qwertyuiop\n" to a file on b-em in Master 128 mode (DFS 2.24) takes 27.8 seconds using OSBPUT and 20.2 seconds using OSGBPB (one call per string, not one call per 256 bytes).

In terms of reading from files, I have a comment in my source code suggesting that on DFS 2.24 OSFILE can load 6.25K in 1.74s while doing the same with OSGBPB takes 4.47s - this is disappointing and I think is the same issue hoglet refers to. You can only use OSFILE if you have enough memory to buffer the entire file though.

ETA: While I doubt language overhead is a major issue, what performance do you get if you implement your file read function to just return (say) 5000 'X' bytes then indicate EOF, without actually making any OS calls? The reason I mention it is that IIRC traditionally C uses macros for getc()/putc() which access a buffer hidden inside the FILE structure and only make a function call every 256 or so bytes to refill the buffer, suggesting that function call overhead can be significant even on machines bigger than a BBC Micro.

User avatar
hjalfi
Posts: 70
Joined: Sat May 13, 2017 10:17 pm
Location: Zürich, Switzelrand
Contact:

Re: Slow OSBGET file access

Postby hjalfi » Sun Sep 10, 2017 6:15 pm

SteveF: excellent point. I replaced file_eof() and file_getchar() with fakes, and it measures about 9kB/sec. Which... isn't so hot, actually.

Also, I would expect that using OSGBPB for amounts smaller than a sector to be about the same speed as OSBGET/OSBPUT --- all it'd be doing is copying from user memory to the internal buffer and back again (because the hardware's incapable of reading or writing smaller quantities than a sector anyway). My assumption was that a disk-based file system would have a 256 byte buffer somewhere, but I don't have any evidence to support that.

Again, I'd assume that OSGBPB for amounts bigger than a sector would copy via the buffer until a sector boundary and then do bulk reads or writes to user memory. Could be wrong...

(I'm aware that early NFS did every OSBGET and OSBPUT via a network RPC. I prefer to treat that kind of pathalogical behaviour with the disdain it deserves...)

I'll try knocking up a version with buffering.

philb
Posts: 97
Joined: Sat Aug 05, 2017 6:05 pm

Re: Slow OSBGET file access

Postby philb » Sun Sep 10, 2017 6:24 pm

I guess DFS must be able to buffer a whole sector somewhere otherwise OSBPUT (or any write of less than a whole sector) couldn't work. And clearly if OSBGET was going to the disk every time then the throughput would be terrible, one byte per rotation of the disk which would be something like 5 bytes per second. So you'd think it probably reads 256 bytes at a time and serves OSBGET out of that sector buffer, but who knews.

http://mdfs.net/Docs/Comp/BBC/Disk/DFSMem seems to say that there is a 256-byte buffer per open channel.

User avatar
lazarusr
Posts: 600
Joined: Thu Sep 10, 2015 8:56 pm
Location: London

Re: Slow OSBGET file access

Postby lazarusr » Sun Sep 10, 2017 6:56 pm

As explained in the ADUG, on DFS 1.20 a call to OSGBPB to read n bytes simply makes n calls to OSBGET and is therefore essentially pointless.

User avatar
jgharston
Posts: 2663
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Slow OSBGET file access

Postby jgharston » Sun Sep 10, 2017 7:25 pm

hjalfi wrote:My assumption was that a disk-based file system would have a 256 byte buffer somewhere, but I don't have any evidence to support that.

It's a valid assumption and supported by the fact that PAGE is at &1900 in a DFS machine, and explained in all and any documentation as because there are buffers for each opened channel.

hjalfi wrote:Again, I'd assume that OSGBPB for amounts bigger than a sector would copy via the buffer until a sector boundary and then do bulk reads or writes to user memory. Could be wrong...

Supported by a comment in the Advanced Disk User Manual, that was my assumption, and was the way I implemented OSGBPB in HADFS without ever considering thinking of implementing it any other way, and - I'm fairly certain - is how ADFS implements it. However, it turns out that all DFSs implement OSGBPB as FOR n=1 TO total:CALL OSBGET/OSBPUT:NEXT n. (As does TAPE/ROM on the Master.)

hjalfi wrote:(I'm aware that early NFS did every OSBGET and OSBPUT via a network RPC.

No, it does it via a NetFS transaction.
BBCFile is a BASIC library that replaces PRINT# and INPUT# with buffered versions. How effective it is depends on how large the lumps of data you are transfering are.

lazarusr wrote:As explained in the ADUG, on DFS 1.20 a call to OSGBPB to read n bytes simply makes n calls to OSBGET and is therefore essentially pointless.

Actually, the ADUG says DFS 0.90 does it this way, which is why I assumed that DFS 1+ didn't do it that way - otherwise, why specifically single out DFS 0.90?

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
hjalfi
Posts: 70
Joined: Sat May 13, 2017 10:17 pm
Location: Zürich, Switzelrand
Contact:

Re: Slow OSBGET file access

Postby hjalfi » Mon Sep 11, 2017 10:21 pm

After much work and many Cowgol bugfixes (turns out that trying to do certain arithmetic operation where the right-hand parameter is placed in zero page generates corrupt code, who knew!), I have a buffered version of my benchmark.

Unbuffered, 700 bytes per second: https://github.com/davidgiven/cowgol/bl ... dspeed.cow

Buffered, 2000 bytes per second: https://github.com/davidgiven/cowgol/bl ... ffered.cow

That's with a single 256 byte buffer. If I take out the call to OSGBPB I get 3300 bytes per second, which indicates the limit of my benchmark.

I haven't tried with bigger buffers. This is actually the second version of the benchmark. The first version managed 1000 bytes/sec, with the file I/O dummied out. Turns out I was using 32-bit shifts to scale the seek position, and my shift routine is grotesquely slow. This version avoids shifts completely, which is why the byte counts are a bit off --- it's discarding any unaligned data at the end of the file. (CP/M doesn't have to worry about this.)

The downside is that the program is now twice as big. The buffered version is 1700 bytes, up from 900 bytes. Pointer arithmetic is killing me.

(Files attached, for your comfort and convenience. Copy onto a handy disk, insert into a BBC Master or a BBC with Tube and run with:

Code: Select all

*go f800   [because we're about to trash Basic's zero page]
*save bigfile 0+8000
*benchbuf bigfile
*benchunbuf bigfile


.)
Attachments
benchmarks.zip
(2.32 KiB) Downloaded 4 times


Return to “software: other”

Who is online

Users browsing this forum: No registered users and 2 guests