ROMless Serial Filing System

[gfoot]

Hi guys

As I mentioned in another thread, I've been working on a ROMless serial filing system. The idea is you just connect the BBC to a PC with a serial cable, and the PC can serve files without having to install any hardware or firmware on the BBC itself.

I've finished a proof of concept and uploaded it to github, with some documentation that explains what it does and how it works: https://github.com/gfoot/serialfs

Making it work with PAGE=&E00 and being strict about the no ROM thing was an interesting challenge, and overall works pretty well. I thought there would be more significant problems than there are! The communication is also quite fast, at 76800 baud, faster than discs for sure, but not as fast as Turbo SPI at loading data.

I don't know how useful this is as an everyday system - something like UPURS is surely better once you get set up with it - but I do like how easy it is to plug in to a new BBC, without needing to worry about ROMs or other hardware.

[SteveF]

This looks like a really neat piece of work - I enjoyed reading the write-up, and there's obviously a lot of attention to detail with things like the retries during the initial setup. 76800 baud is a lot better than I would have expected as well!

I don't actually have real hardware to use with it so I haven't tried it out. That said, while reading the write-up, I did wonder if the initial "*FX2,1" startup should automatically select your filing system as well, rather than requiring an extra *S command before it becomes active. Just a thought, I could see arguments either way.

Taking this a bit further: would it save code if instead of intercepting the CLI vector to recognise "*S", you intercepted the OSBYTE vector and made "*FX2,1" the way to re-select your filing system as well? So "*FX2,1" would be your equivalent of "*DISC" or "*TAPE", almost.

[BigEd]

This is excellent! A minimal bootstrap with a tiny footprint. I do have an UPURS setup but this is surely complimentary.

[gfoot]

This looks like a really neat piece of work - I enjoyed reading the write-up, and there's obviously a lot of attention to detail with things like the retries during the initial setup. 76800 baud is a lot better than I would have expected as well!

At some point, failed connections were really frustrating during development, so I kind of had to do something about it! The awkward thing with the baud rates is that when you press Break some, but not all, of the settings are reset, and the beeb is left in a state where its serial port is set to a read speed that the OS can't reliably support. So as soon as I made it crank up the speed after connecting, I also had to make it deal with some of those settings also being set on startup.

76800 seems to be the fastest speed the BBC's hardware supports for reading. To read data any faster, the 6850 requires an external clock synchronization circuit that I don't believe the BBC's serial ULA provides - at least, it doesn't work well at 153600 baud, in my test program I got sequences of maybe around 64 bytes of valid data alternating with 64 bytes of errors from the 6850, probably framing errors but I didn't check.

I don't actually have real hardware to use with it so I haven't tried it out. That said, while reading the write-up, I did wonder if the initial "*FX2,1" startup should automatically select your filing system as well, rather than requiring an extra *S command before it becomes active. Just a thought, I could see arguments either way.

Yes, that would make sense - there's not much point having it loaded but inactive. I think the *S command still needs to exist so that you can switch to other filing systems and back.

Taking this a bit further: would it save code if instead of intercepting the CLI vector to recognise "*S", you intercepted the OSBYTE vector and made "*FX2,1" the way to re-select your filing system as well? So "*FX2,1" would be your equivalent of "*DISC" or "*TAPE", almost.

Maybe. *FX2 and similar are quite destructive for this system - it can't really tolerate changes to the settings while its running. So trapping them and preventing them, at least, would probably be a good idea. It's yet another hook that's impossible to unhook later on though!

[lovebug]

nice work gfoot

[davidb]

Nice work!

I wondered about ways to bootstrap filing systems using existing OS features or minimal amounts of code, so this is very useful and quite inspirational.

[Coeus]

This is an interesting and novel approach.

Taking this a bit further: would it save code if instead of intercepting the CLI vector to recognise "*S", you intercepted the OSBYTE vector and made "*FX2,1" the way to re-select your filing system as well? So "*FX2,1" would be your equivalent of "*DISC" or "*TAPE", almost.

Maybe. *FX2 and similar are quite destructive for this system - it can't really tolerate changes to the settings while its running. So trapping them and preventing them, at least, would probably be a good idea. It's yet another hook that's impossible to unhook later on though!

I think leaving this serial filing system as the default after the initial *FX2,1 makes sense but I am not sure if re-selecting on a further *FX2,1 would be ideal. One of the things someone might want to do is use a program like JGH's TreeCopy which copies files between filing systems and assumes each can be selected with a * command.

But hooking OSBYTE A=2 would probably be a good idea and also OSBYTE A=3 and maybe even OSBYTE A=5 to prevent things from selecting other features that use the serial hardware. Of course, that doesn't stop programs poking the hardware.

[Rich Talbot-Watkins]

Clever idea! Really like how it bootstraps itself (and hides itself while doing so!).

How long does something like Repton 3 take to load? Have you contemplated a simple compression scheme like LZ4 to help loading time?

Edit: never mind, I see you already said on the first post that it runs faster than disk! That's great!

[gfoot]

Clever idea! Really like how it bootstraps itself (and hides itself while doing so!).

How long does something like Repton 3 take to load? Have you contemplated a simple compression scheme like LZ4 to help loading time?

Edit: never mind, I see you already said on the first post that it runs faster than disk! That's great!

Yes, I was a bit disappointed when it was running at 9600 baud, but 76800 is great. The caveat is, it's quite a few months since I loaded anything from a disc, I was judging from memory! The seek time here is pretty much zero, which makes it feel very snappy for day-to-day use.

Regarding overall throughput, Repton 3 loads in 4 seconds (5 separate files in my version), and more concretely it takes about 2.75 seconds to load a 20K MODE 0 screen dump, which is on par with what you'd expect given the baud rate - i.e. 76800 bits per second with 10 bits per byte including one start bit and one stop bit. For comparison that's 7.5K per second.

I guess in theory discs spin at 300RPM and at single density store 2.5K per track, so at maximum they could load 12.5K per second - but stepping from track to track slows that down quite a bit for larger transfers. Testing in beebjit, with default disc settings it seems to take about 4 seconds to load a screen dump, including a seek time of about one second in this case. With *FX255,0 active, that disc load takes about 3.5 seconds. Overall I think the transfer rate is comparable.

Another caveat is that at the moment I'm entirely disabling interrupts while receiving blocks of data, as any data sent when the client is not ready gets lost. So the client disables interrupts, sends a sentinel byte to the server, then blocks waiting for however many bytes are expected, before enabling interrupts again. Any interrupts that were pending in this period then get serviced, but things like the clock will get delayed because entire cycles are missed, and keyboard type-ahead won't work.

I have some plans to mitigate this, most likely by receiving the data in smaller blocks and enabling interrupts briefly in between to let the system process the backlog. A block size of 64 bytes for example would mean we "only" disable interrupts for about 8ms at a time, so things like the system clock updates and keyboard polling would not entirely miss any interrupt cycles, they'd just be delayed by less than a centisecond. Smaller block sizes would create shorter delays, and even down to about 16 bytes (2ms) it would probably not significantly slow down the transfer.

[cmorley]

My "in house" appropriately named bin2beeb runs at 76k8. There is plenty of time to do a CRC16 in between bytes arriving so everything sent & received over the serial is CRCed and retried if necessary. Super reliable even on a 5m cable of questionable origin. The model B really is quite splendid when it comes to serial! I was thoroughly disappointed in comparison with the A3000 which only does 19200 IIRC!

I use Tom Seddon's beeblink with the Tube interface but I suspect that could be reworked to 76k8 built in serial easily enough.

[gfoot]

My "in house" appropriately named bin2beeb runs at 76k8. There is plenty of time to do a CRC16 in between bytes arriving so everything sent & received over the serial is CRCed and retried if necessary. Super reliable even on a 5m cable of questionable origin. The model B really is quite splendid when it comes to serial! I was thoroughly disappointed in comparison with the A3000 which only does 19200 IIRC!

I considered adding CRCs but didn't bother as so far I haven't seen any corruption, and my code to detect framing errors or late reads from the ACIA also never triggers at this baud rate. Have you ever had a CRC error with your system?

If I do add CRCs, I think I'd go with CRC-8 to save on code space which is quite precious here. I adapted one of jgh's already well-optimised CRC-16 routines about a month ago I think. I already send a dummy byte to the server to say we're ready for data, so it would be easy to send the CRC of the previous block instead and have the server verify it before continuing. Because the server is so in control of the process it could then carry on with the bulk transfer, then send a second transfer command to the client afterwards to patch up the block(s) that had errors.

It is hard to verify these sorts of systems though unless the link is actually generating the errors you're guarding against!

[cmorley]

Have you ever had a CRC error with your system?

On occasion but they are very infrequent. A testament to the Beeb's serial port! bin2beeb sits in page 1 (stack) and the CRC16 fits in there with everything else just fine.... only other memory I use is a few bytes of zero page.

[gfoot]

Nice. Have you published bin2beeb? It'd be interesting to see the similarities.

[cmorley]

No. It only ever got half finished! Goteks and Tom's excellent beeblink filesystem lowered the dev priority somewhat. Version 2 has been in development for 4 years or something now I did get the bootstrap down to 19 bytes IIRC

Code: Select all

.fastrx_bootstrap_start
.fastrx_bootstrap_entry
	sei
	tsx:stx acia_d ; send sync to PC
	ldx #LO(fastrx_loader_mid-1)
	txs
	.bootstrap_loop
	_rx_byte
	pha
.fastrx_bootstrap_continue
	bcs bootstrap_loop ; rxbyte leaves with carry set
.fastrx_bootstrap_end
SAVE "fastrx_bootstrap.bin",fastrx_bootstrap_start,fastrx_bootstrap_end

[gfoot]

Nice hack using pha! No CRC there though. Do you relocate the existing stack down and put your code at the top of the page so that it's out of the way?

I found BeebLink after getting my serial cable, but after reading things for a while I realised it, and HostFS, didn't actually support normal serial ports. It's the main reason I started making my own - allergy to ROMs being the other!

I'm considering making a version which loads at E00 instead, so that it's more compatible with software that's designed to run from disc. E00-1100 is also a lot more space than I'm using now. However, after *DISK the code would be lost and it would need bootstrapping again with *FX2,1 to go back to SerialFS, and that only works at an interactive BASIC prompt so can't be done in the middle of a program. It's a trade-off of limitations. Actually storing the bootstrap code on disc could also be interesting for supporting that use case.

[cmorley]

Nice hack using pha! No CRC there though. Do you relocate the existing stack down and put your code at the top of the page so that it's out of the way?

The bootstrap loads the loader. Loader copies the original stack contents to the PC and overwrites it with the serial code. (I can restore the stack on exit) Then does a CRC on the full page 1. Then switches to 76k8 and all subsequent comms are fully CRCed.
I initially send the bootstrap by crafting !&100=&... statements. So all I need is a *FX2,1

Code: Select all

\\\ fast loader
org stack_base
clear stack_base,stack_end

.fastrx_bootstrap_start
.fastrx_bootstrap_entry
	sei
	tsx:stx acia_d ; send sync to PC
	ldx #LO(fastrx_loader_mid-1)
	txs
	.bootstrap_loop
	_rx_byte
	pha
.fastrx_bootstrap_continue
	bcs bootstrap_loop ; rxbyte leaves with carry set
.fastrx_bootstrap_end
SAVE "fastrx_bootstrap.bin",fastrx_bootstrap_start,fastrx_bootstrap_end

\\\ continue fast loader
org stack_base
clear stack_base,stack_end
guard stack_end

;.init_crc EQUW 0

skipto fastrx_bootstrap_continue
\	bcc bootstrap_loop ; break loop
	bcs * ; break loop
	
	; get code
	ldy #LO(fastrx_loader_mid) ; exchange to end of stack
	{
	.l
		ldx &100,y
		_txrx_wait
		lda acia_d:stx acia_d
		sta &100,y
		iny:bne l
	}

.fastrx_loader_mid
	; CRC stack region y=0 on entry
	sty crc:sty crc+1 ; clear CRC
	{
	.l
		dey
		lda &110,y
		jsr fn_crc16
		cpy #LO(stack_base):bne l ; reverse CRC from &1ff to &110
	}
	; send crc
	lda crc:ora crc+1:sta acia_d ; send CRC back
	beq idle_wait ; use idle_wait to sync before changing baud
	
.loader_fail
	;lda #7:JSR &FFEE
	brk
	EQUB 255
	EQUS 7,"Loader CRC fail",0
.fastrx_loader_end


msg_len = &17f;

\\\ fast rx main code
skipto &180

.fastrx_start
	
; main idle wait loop
.idle
	sei ; repeat in case last cmd enabled interrupts
	lda #1 ; overwritten
copy_serialula=*-1
	sta serialula
	lda #1 ; overwritten
copy_acia=*-1
	sta acia
.idle_wait
	ldx #LO(msg_len):txs ; set stack pointer
	jsr fn_rxbyte:jsr fn_txbyte ; RX & echo
	bne idle_wait ; zero go, else echo idle

.go ; A=0 on entry
	sta crc:sta crc+1 ; clear CRC-16
	{
	.l
		jsr fn_rxbyte ; stack command(s), first byte is length
		dec msg_len:beq done ; test end of message
		tsx:bpl l ; loop if not run out of stack
	.done
	}
	pla:pla ; dump CRC
.checkcrc
	lda crc:ora crc+1 ; check CRC
	jsr fn_txbyteA ; send to PC
	bne idle ; bad command
; good command
	rts ; RTS jumps to command

	
.fn_rxbyte ; returns value on top of stack
	_rx_byte
.fn_stackandcrc16 ; stack & CRC
	pha
.fn_crc16 ; trashes A & X
	_crc_fast(crc)
	rts

.fn_txbytecrc ; send value in A
	jsr fn_stackandcrc16 \do crc
	jmp fn_txbyte
	
.fn_txcrc ; send crc
	lda crc:jsr fn_txbyteA
	lda crc+1
.fn_txbyteA ; send value in A
	pha
.fn_txbyte  ; sends value on top of stack
	_tx_wait:pla \ wait for transmit empty
	_tx_byte
	rts
	
.fastrx_end

SAVE "fastrx.bin", stack_base, stack_end

[sweh]

...and HostFS, didn't actually support normal serial ports.

My version or JGH's version? They both should, but you may need to change some conditional flags on mine (it default compiles to UPURS). It's likely not as fast as yours, though.

[gfoot]

I have some plans to mitigate this, most likely by receiving the data in smaller blocks and enabling interrupts briefly in between to let the system process the backlog. A block size of 64 bytes for example would mean we "only" disable interrupts for about 8ms at a time, so things like the system clock updates and keyboard polling would not entirely miss any interrupt cycles, they'd just be delayed by less than a centisecond. Smaller block sizes would create shorter delays, and even down to about 16 bytes (2ms) it would probably not significantly slow down the transfer.

Unfortunately this doesn't work well. It massively slows down the transfer - doing the maths, it looks like the penalty for a round trip between client and server is about 16ms, which is huge. In addition this round trip happens with interrupts disabled, so the BBC still suffers missed interrupts.

It is possible that using RTS to control the flow, rather than an explicit byte sent by the client, would remove the round trip time, I will try that at some point. I already use this for detecting the *FX2,1, so it is wired through in my cable (to DSR though, not CTS).

[gfoot]

...and HostFS, didn't actually support normal serial ports.

My version or JGH's version? They both should, but you may need to change some conditional flags on mine (it default compiles to UPURS). It's likely not as fast as yours, though.

I'm not sure, and perhaps I got mixed up. I find it hard to determine the differences between these things - it's great that there are so many options available, as different users definitely have very different needs, but I think I said somewhere else recently, I feel we're kind of in need of a web page briefly summarizing the pros and cons of different solutions (discs, different types of MMFS, Pi1MHz, UPURS, and all the rest).

[Coeus]

Unfortunately this doesn't work well. It massively slows down the transfer - doing the maths, it looks like the penalty for a round trip between client and server is about 16ms, which is huge. In addition this round trip happens with interrupts disabled, so the BBC still suffers missed interrupts.

This is why sliding window protocols took off, and before that protocols with large blocks. But that would be where there is a modem involved or a trip across a network with possibly many routers in it. You would hope the round trip would be pretty quick here.

I wonder if this is context switching on the PC end, i.e. as soon as you stop sending and start waiting for the char from the BBC, The PC OS switches away from your server code to run something else and doesn't necessarily switch back to it again as soon as the character does arrive. If so, hardware handshaking may be a solution, i.e. if you can hand off a larger transfer to the driver and that honours the handshaking without needing your server code to be back on the CPU.

This would also be a case where padding would work, just like the days of the old teletype. If there was a value that could be sent that did not occur in the data then the sending end could send a block of real data, i.e. the 64 bytes, then pad for a long enough for a typical interrupt cycle on the BBC end, then resume sending real data. Because the timing is done by padding rather than a timer the whole transmit payload can be handed off to the driver.

Of course, that was when sending ASCII so NUL could be used for padding. For binary you would need some scheme to avoid the pad character appearing in the data. One way would be to also choose an escape character such that the pad character is translated into a two character sequence starting with the escape character (for which the second character is not the escape character) and if the escape character appears in the data it is doubled. NUL would probably be a bad choice for either pad or escape because it is quite common in binary data.

[Coeus]

I guess in theory discs spin at 300RPM and at single density store 2.5K per track, so at maximum they could load 12.5K per second - but stepping from track to track slows that down quite a bit for larger transfers.

So the peak transfer rate of floppies is higher. That means using an NMI to service the 6850 would be fast enough and would solve the interrupt problem because you would then be interrupting the normal IRQ handler rather than the other way round. I can see how to do that without a hardware mod, though.

[gfoot]

I wonder if this is context switching on the PC end, i.e. as soon as you stop sending and start waiting for the char from the BBC, The PC OS switches away from your server code to run something else and doesn't necessarily switch back to it again as soon as the character does arrive. If so, hardware handshaking may be a solution, i.e. if you can hand off a larger transfer to the driver and that honours the handshaking without needing your server code to be back on the CPU.

Yes, it's only written in Python and it's possible that either the interpreter is too slow for this, or the serial interface implementation is doing a short sleep when I ask to read data while none is waiting, and not getting woken up quickly when data does arrive.

Unfortunately either due to issues with Python's serial library or my USB serial adaptor, hardware handshaking seems to have no effect - data is sent even if I set RTS low, and I get "overrun" errors because I'm not reading it.

I've had better results from taking RTS low and then high, and detecting the rising edge in my own software on the server, but I need a hefty 2ms delay otherwise it misses the edge completely. I never really expected the server code to be the slow part, but I might have to rewrite it in C one day!

This would also be a case where padding would work, just like the days of the old teletype. If there was a value that could be sent that did not occur in the data then the sending end could send a block of real data, i.e. the 64 bytes, then pad for a long enough for a typical interrupt cycle on the BBC end, then resume sending real data. Because the timing is done by padding rather than a timer the whole transmit payload can be handed off to the driver.

That's interesting. I use a similar technique on startup to get the Beeb out of the high-baud configuration. The OS isn't capable of servicing the interrupts quickly enough and characters are lost, but I can just delay between characters on the server and send them more spread out, and it handles it fine.

Inserting a pad character would allow the delay to happen in hardware, but it would cause the ACIA to flag overrun errors. But I can make my code ignore those when they're expected.

Of course, that was when sending ASCII so NUL could be used for padding. For binary you would need some scheme to avoid the pad character appearing in the data. One way would be to also choose an escape character such that the pad character is translated into a two character sequence starting with the escape character (for which the second character is not the escape character) and if the escape character appears in the data it is doubled. NUL would probably be a bad choice for either pad or escape because it is quite common in binary data.

I don't think the specific character matters too much. I can precede each block with a specific start character that's different to the pad character - like the way the cassette interface sends a long high tone between blocks, then the block header always starts with a low bit.

The remaining problem is that the amount of delay needed is hard to predict. Interrupt handling on the Beeb takes varying amounts of time. I would need to pad every packet to some "worst case" value, just in case an interrupt took that long to execute. It is well worth trying though. as every attempt based on using software control in Python has been rather slow so far.

Also NMIs - yes it did strike me that life would be easier if that was an option! And of course that must be how Econet works.

[BigEd]

As I recall, in the UPURS adventure, it turned out that some USB serial adapters were really too slow to react to RTS, or didn't react at all.

[dominicbeesley]

Yes rts and cts can be very slow especially on some usb serial adapters.

Reading and writing small numbers of bytes seem also to be quite slow. Even with explicit attempts at flushing it seems that the USB to serial implementations try to fill/coalesc a buffer before transferring. An attempt to send a single byte will wait until some timeout before actually sending it and I couldn't find any way of improving it. This seems to be also true of things like the ft245 which provides a usb to parallel interface which I've tried going up to the 1 mhz bus and user port but never managed to get very good round trips.

Real COM ports should be better but they are less common.

Excellent idea this!

D

[tom_seddon]

What a great project! I love the dynamic loading.

Unfortunately either due to issues with Python's serial library or my USB serial adaptor, hardware handshaking seems to have no effect - data is sent even if I set RTS low, and I get "overrun" errors because I'm not reading it.

It'll be no help, but it might at least be consolation: I've had various problems with this sort of thing when trying to support UPURS in BeebLink, even when using a genuine FTDI adapter. From memory, the default macOS Catalina serial drivers just seemed to ignore hardware flow control entirely! - and even when things do work, it's very common for the Beeb to receive an extra byte after indicating it's no longer ready.

This all seemed to be somewhat driver-dependent, something MartinB complains about a bit in the UPURS thread, so after a while I just gave up trying to figure it out. The code is still there, and there are combinations of server OS + driver that do work, and I don't really promise more than that.

But I will be revisiting this issue at some point.

Unfortunately this doesn't work well. It massively slows down the transfer - doing the maths, it looks like the penalty for a round trip between client and server is about 16ms, which is huge. In addition this round trip happens with interrupts disabled, so the BBC still suffers missed interrupts.

If you're using an FTDI adapter, are you falling foul of its latency timer? This specifies how long the device waits before sending the buffered data over USB when its internal buffer is non-empty, but not full. This seems to default to 16 ms, but you can set it to something lower: https://github.com/tom-seddon/beeblink/ ... n.ts#L1196

--Tom

[gfoot]

As I recall, in the UPURS adventure, it turned out that some USB serial adapters were really too slow to react to RTS, or didn't react at all.

Ah ok, it could be in the adapter. It does pass the signal through - my server code can tell whether RTS is high or low (though it's wired to DSR it seems) - but the adaptor doesn't seem to actively do anything with the signal itself.

Reading and writing small numbers of bytes seem also to be quite slow. Even with explicit attempts at flushing it seems that the USB to serial implementations try to fill/coalesc a buffer before transferring. An attempt to send a single byte will wait until some timeout before actually sending it and I couldn't find any way of improving it. This seems to be also true of things like the ft245 which provides a usb to parallel interface which I've tried going up to the 1 mhz bus and user port but never managed to get very good round trips.

Coeus's earlier comments really got me thinking, and I'll try some padding next. I wanted to first see what the best performance I could achieve using software polling of RTS on the server. This turns out to be 6 seconds to load a 20K screen dump - up from about 2.75 when interrupts were disabled.

The final settings for that turned out to be 32-byte packets, toggling RTS to tell the server to start sending the next packet, and overlapping that toggle with the last few bytes of the previous packet. The timeline looks sort of like this:

Code: Select all

ACIA bytes incoming  --29---30---31-- - - - -------------------------

6502 read operations ----R----R------ - - - -------R-----------------

6502 IRQ enable      __________/""""" " " " "\_______________________

RTS                  ________________ _ _ _ ___""""""""""""""""""""""
                                               \
PC server code                                  \..send next packet...

So after reading the penultimate byte, there's only one more remaining in the packet, so no risk of it getting overwritten. So we can enable interrupts briefly now - just turn them on then off again. In the diagram I've shown an interrupt occurring and delaying everything at that point by an arbitrary amount, but the server is just waiting so the last byte is still safe in the ACIA's receive register, and can be read after the pending interrupts are handled.

Once interrupts are turned off again, we can toggle RTS to signal to the server that it should start sending the next packet. In practice this seems to take about as long to start sending as the entire packet takes to receive at 76800 baud.

Anyway I think that's my baseline. I overlapped it as much as I could do safely. It's possible that I could signal the server before turning on interrupts, but there's a danger the BBC's interrupt handling could take longer than the PC does to start the next packet.

So going back to Coeus's suggestion, I'm going to try just padding every 32-byte packed with maybe a millisecond's worth of dummy bytes, and see if that's enough to generally deal with interrupt costs. Anything less than 5ms of padding would be a win compared to the flow control system I just implemented.

Real COM ports should be better but they are less common.

There'd be a certain irony to it if another Beeb was a better server than a modern PC!

[gfoot]

Unfortunately this doesn't work well. It massively slows down the transfer - doing the maths, it looks like the penalty for a round trip between client and server is about 16ms, which is huge. In addition this round trip happens with interrupts disabled, so the BBC still suffers missed interrupts.

If you're using an FTDI adapter, are you falling foul of its latency timer? This specifies how long the device waits before sending the buffered data over USB when its internal buffer is non-empty, but not full. This seems to default to 16 ms, but you can set it to something lower: https://github.com/tom-seddon/beeblink/ ... n.ts#L1196

Ah that's interesting too. I tried using pySerial's "set_low_latency_mode" function, but it doesn't improve my latest version. 20k in 6 seconds is about 9.5ms per 32-byte packet, so it looks like the latency is already below 16ms now.

[gfoot]

So after reading the penultimate byte, there's only one more remaining in the packet, so no risk of it getting overwritten. So we can enable interrupts briefly now - just turn them on then off again. In the diagram I've shown an interrupt occurring and delaying everything at that point by an arbitrary amount, but the server is just waiting so the last byte is still safe in the ACIA's receive register, and can be read after the pending interrupts are handled.

Once interrupts are turned off again, we can toggle RTS to signal to the server that it should start sending the next packet. In practice this seems to take about as long to start sending as the entire packet takes to receive at 76800 baud.

I took one more pass at this last night, more successfully - here's an updated diagram. (The numbers in the top row now show how many bytes are remaining in the packet, rather than the byte number within the packet.)

Code: Select all

ACIA bytes incoming  --31---30---29---28-- - - -02---01---00-- - - - ---------31---30---29---28-- - -

6502 read operations ----R----R----R----R- - - ---R----R------ - - - ---R-------R----R----R----R- - -

6502 IRQ enable      _____________________ _ _ _________/""""" " " " "\__________________________ _ _

RTS                  _______________"""""" " " """"""""""""""" " " " """"""""""""""""""""""______ _ _
                                    \                                                      \
PC server code                       \..send next packet...... . . . ...sent                \..send..

The change here is that I signal the server to start sending much earlier - when the previous packet has 29 bytes remaining to be read. I still don't enable interrupts until there's only one more byte remaining. The danger here is that if IRQ processing takes an unusually long time, the 6502 won't be ready to read that final byte before the first byte of the next packet arrives and overwrites it. 29 is the highest value I could use without seeming to cause this problem but there's still a risk especially if user interrupt handlers are installed.

With this configuration the overall speed is about 4.7k per second, about 60% of the original speed when interrupts were disabled.

Finally I think there are a couple of interesting metrics here. The packet duration for 32 bytes of payload is a fixed 6.64ms. Each byte is 0.13ms, so there's about 2.5ms of space between the packets over the wire. This space is what seems to be required to make room for interrupt servicing - so I guess in a typical-worst-case, where everything needs attention at once, the Beeb spends about 2.5ms in the interrupt handler.

Also, the latency between RTS changing state and the server actually sending the first byte of the next packet is 6.64ms-3*0.13ms = 6.25ms. That includes everything from the hardware up to the Python interpretter actively polling the RTS state, through calling the send function, and back down to the hardware again. Although this is annoying, it is at least no longer the limiting factor - worst-case IRQ response is now the bottleneck.

[BigEd]

Just got around to giving this a spin (sorry, I'm very slow on the uptake.)

It's not quite working... I'm using a Master, is that new territory?

I'm running from a MacBook too, with a particular USB adapter which possibly has trouble with the DSR output - the reason I say that is that I have to run the server twice, the first time it stalls waiting for DSR.

The serial fs appears to connect, bootstrap, activate, and return control to the keyboard, but then:

Code: Select all

>*.

SerialFS: Invalid command code

On the server side, with loglevel 3, I see

Code: Select all

$ python server/serialfs-server.py 
Assembling data/init.x
himem-*=himem-*=29
*-top=*-top=0
*-org-256=*-org-256=0
156
Assembling data/main.x
himem-*=himem-*=63
Assembling data/error.x
himem-*=himem-*=90
Assembling data/sendblock.x
himem-*=himem-*=80
Assembling data/debug.x
Assembling data/activate.x
himem-*=himem-*=93
Assembling data/recvblockwithirq.x
himem-*=himem-*=7
Assembling data/message.x
himem-*=himem-*=142
Assembling data/osfile.x
himem-*=himem-*=42
Assembling data/recvblocksmall.x
himem-*=himem-*=81
Assembling data/recvblock.x
himem-*=himem-*=88
Assembling data/catalogue.x
himem-*=himem-*=51

Listening
Line open
Initializing link
Initializing link
    Trying to reset client from 19200 baud
Initializing link
Initializing link
    Trying to reset client from 78600 baud
Initializing link
Initializing link
    Trying to reset client from 38400 baud
Initializing link
Initializing link
    Trying to reset client from 4800 baud
Initializing link
Initializing link
    Trying to reset client from 2400 baud
Initializing link
Initializing link
    Trying to reset client from 1200 baud
Initializing link
b'>'
Link active
 exec: *FX3,0
Sending init code
 exec: A$=""
 exec: FOR I%=0 TO 253:A$=A$+GET$:N.
 exec: OSCLI("FX3,0"):OSCLI("FX2,2"):OSCLI("FX204,1"):$&A00=A$:CALL&A00
Waiting...
unexpected: b'*'
unexpected: b'F'
unexpected: b'X'
unexpected: b'3'
unexpected: b','
unexpected: b'0'
unexpected: b'\n'
unexpected: b'\r'
Upgrading speed to 76800 baud
Sending code: message
    send_code: waiting for client
    send_code: sending
   0000: a2 12 bd 0d 0a 20 ee ff  ..... ..
   0008: ca d0 f7 4c ac 0a 0d 0a  ...L....
   0010: 65 76 69 74 63 61 20 53  evitca S
   0018: 46 6c 61 69 72 65 53 0a  FlaireS.
Sending code: data/activate.x
    send_code: waiting for client
    send_code: sending
   0000: 20 06 0a 4c ac 0a ad 1e   ..L....
   0008: 02 c9 fb d0 08 ad 1f 02  ........
   0010: c9 0a d0 01 60 a9 06 20  ....`.. 
   0018: 2e 0a a2 0d bd 31 0a 9d  .....1..
   0020: 12 02 ca 10 f7 a9 8f a2  ........
   0028: 0f a0 00 4c f4 ff 6c 1e  ...L..l.
   0030: 02 e9 0a ec 0a ef 0a f2  ........
   0038: 0a f5 0a f8 0a fb 0a     ....... 
Sending code: data/main.x
    send_code: waiting for client
    send_code: sending
   0000: 4c 09 0a 00 00 00 00 00  L.......
   0008: 00 20 22 0a ad 03 0a ae  . ".....
   0010: 04 0a ac 05 0a 2c 06 0a  .....,..
   0018: 18 50 01 38 30 01 60 6c  .P.80.`l
   0020: 07 0a 48 a9 2f 8d 08 02  ..H./...
   0028: a9 0a 8d 09 02 68 60 86  .....h`.
   0030: f8 84 f9 a0 ff c8 b1 f8  ........
   0038: c9 20 f0 f9 c9 2a f0 f5  . ...*..
   0040: b1 f8 29 df c9 53 d0 10  ..)..S..
   0048: c8 b1 f8 c9 21 b0 09 a9  ....!...
   0050: 2a 48 48 a4 f9 4c 9c 0a  *HH..L..
   0058: a4 f9 6c fe 0a           ..l..   
Ready
Received command 0a, regs 0f 00 ee
Invalid command code
Sending code: error
    send_code: waiting for client

    send_code: sending
   0000: 20 06 0a 4c 41 0a 48 a9   ..LA.H.
   0008: 13 8d 08 02 a9 0a 8d 09  ........
   0010: 02 68 60 86 f8 84 f9 a0  .h`.....
   0018: ff c8 b1 f8 c9 20 f0 f9  ..... ..
   0020: c9 2a f0 f5 b1 f8 29 df  .*....).
   0028: c9 53 d0 10 c8 b1 f8 c9  .S......
   0030: 21 b0 09 a9 2a 48 48 a4  !...*HH.
   0038: f9 4c 9c 0a a4 f9 6c fe  .L....l.
   0040: 0a 00 e0 53 65 72 69 61  ...Seria
   0048: 6c 46 53 3a 20 49 6e 76  lFS: Inv
   0050: 61 6c 69 64 20 63 6f 6d  alid com
   0058: 6d 61 6e 64 20 63 6f 64  mand cod
   0060: 65 00                    e.      
Received command 0a, regs 09 04 fd
Invalid command code
Sending code: error
    send_code: waiting for client

    send_code: sending
   0000: 20 06 0a 4c 41 0a 48 a9   ..LA.H.
   0008: 13 8d 08 02 a9 0a 8d 09  ........
   0010: 02 68 60 86 f8 84 f9 a0  .h`.....
   0018: ff c8 b1 f8 c9 20 f0 f9  ..... ..
   0020: c9 2a f0 f5 b1 f8 29 df  .*....).
   0028: c9 53 d0 10 c8 b1 f8 c9  .S......
   0030: 21 b0 09 a9 2a 48 48 a4  !...*HH.
   0038: f9 4c 9c 0a a4 f9 6c fe  .L....l.
   0040: 0a 00 e0 53 65 72 69 61  ...Seria
   0048: 6c 46 53 3a 20 49 6e 76  lFS: Inv
   0050: 61 6c 69 64 20 63 6f 6d  alid com
   0058: 6d 61 6e 64 20 63 6f 64  mand cod
   0060: 65 00                    e.      
Received command 0a, regs 09 04 fd
Invalid command code
Sending code: error
    send_code: waiting for client

(Installation and use notes:
needed to install 'xa' with

Code: Select all

sudo port install xa

start server with

Code: Select all

python server/serialfs-server.py

edit the USB device name in a server/session.py and server/connection.py from
/dev/ttyUSB0
to
/dev/tty.usbserial-FT93T7PV
edit the loglevel in server/serialfs-server.py from 0 to 3
)

[sweh]

/dev/tty.usbserial-FT93T7PV

Maybe try /dev/cu.usbserial-FT93T7PV to see if that makes it work first time?