Quadbike: Experimental, DSP tape capture -- WAVs needed!

discuss pc<>acorn file transfer issues and the use of other utils
Post Reply
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

Might be time I created a thread for this.

If you've been paying attention to this thread you will have seen a little discussion about the possibility of using DSP-type algorithms to capture tapes, rather than the traditional "waveform walking" time-domain based approach.

I've spent the past few weeks experimenting with this idea, pursuing a hunch that a combination of a software phase-locked loop (PLL) for synchronisation and something Fourier-esque to identify frequencies might produce reasonable results. I had intended just to use a full-blooded discrete Fourier transform for the frequency domain portion, but BigEd suggested deploying the Goetzel algorithm instead, so that is what I am using.

I don't have any code to share just yet, but I finally (ten minutes ago) added CSW export to Quadbike, so some results are now starting to emerge:
Screenshot_20210620_074514.png
There are plenty of errors in the capture, but there is still much room for improvement here. In particular, one advantage of this approach is that every bit captured comes with an associated confidence level, which provides various options for automatic (and, indeed manual) correction of bit errors. I have not attempted any error correction yet.

Anyway, I seem to have a total of about six Acorn tapes remaining in my collection. I used to have more, but I don't know what happened to them. Is there a repository of WAV recordings of tapes available online anywhere that I can use to test this? If not, I would be grateful if anyone digitising tapes could keep their WAV or FLAC files, and permit me access to them for testing this.

I will release the code for QB, but I would like to improve its fidelity a little bit first.

Thanks.

D
User avatar
BigEd
Posts: 4234
Joined: Sun Jan 24, 2010 10:24 am
Location: West Country
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by BigEd »

Ooh, I'll follow this closely!
User avatar
flaxcottage
Posts: 4692
Joined: Thu Dec 13, 2012 8:46 pm
Location: Derbyshire
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by flaxcottage »

This looks to be very interesting.

It would be extremely interesting if I actually understood the maths behind it. :lol:
- John

Check out the Educational Software Archive at www.flaxcottage.com
User avatar
vanekp
Posts: 1060
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by vanekp »

Diminished wrote:
Sun Jun 20, 2021 7:52 am
Might be time I created a thread for this.

If you've been paying attention to this thread you will have seen a little discussion about the possibility of using DSP-type algorithms to capture tapes, rather than the traditional "waveform walking" time-domain based approach.

I've spent the past few weeks experimenting with this idea, pursuing a hunch that a combination of a software phase-locked loop (PLL) for synchronisation and something Fourier-esque to identify frequencies might produce reasonable results. I had intended just to use a full-blooded discrete Fourier transform for the frequency domain portion, but BigEd suggested deploying the Goetzel algorithm instead, so that is what I am using.

I don't have any code to share just yet, but I finally (ten minutes ago) added CSW export to Quadbike, so some results are now starting to emerge:

Screenshot_20210620_074514.png

There are plenty of errors in the capture, but there is still much room for improvement here. In particular, one advantage of this approach is that every bit captured comes with an associated confidence level, which provides various options for automatic (and, indeed manual) correction of bit errors. I have not attempted any error correction yet.

Anyway, I seem to have a total of about six Acorn tapes remaining in my collection. I used to have more, but I don't know what happened to them. Is there a repository of WAV recordings of tapes available online anywhere that I can use to test this? If not, I would be grateful if anyone digitising tapes could keep their WAV or FLAC files, and permit me access to them for testing this.

I will release the code for QB, but I would like to improve its fidelity a little bit first.

Thanks.

D
no real repository of tapes especially not in a wav format, I have put my collection on Google drive not all are original tape wav files (made with makeuef) some come from other sources but the csw and final uef files are also there plus any documentation I have for the programs.
https://drive.google.com/drive/folders/ ... sp=sharing
Regards Peter.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

flaxcottage wrote:
Sun Jun 20, 2021 8:50 am
It would be extremely interesting if I actually understood the maths behind it. :lol:
I am having a similar problem. :lol: Fortunately the Internet provides code samples, and also things like this filter designer, which lets you draw the filter response you want and then just spits out the C code needed to implement it. :shock:
vanekp wrote:
Sun Jun 20, 2021 11:13 am
I have put my collection on Google drive not all are original tape wav files (made with makeuef) some come from other sources but the csw and final uef files are also there plus any documentation I have for the programs.
Ah, excellent. This looks like exactly what I wanted, thanks!
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

Does anyone know under what circumstances single cycles of 2400 Hz tone occur (or, to look at it another way, odd numbers of cycles of 2400 Hz tone)? I'm guessing leader sections can have odd numbers of cycles -- do they happen anywhere else? Could a custom protection mechanism create them at will?
Coeus
Posts: 2249
Joined: Mon Jul 25, 2016 12:05 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Coeus »

Diminished wrote:
Wed Jun 23, 2021 4:04 pm
Does anyone know under what circumstances single cycles of 2400 Hz tone occur (or, to look at it another way, odd numbers of cycles of 2400 Hz tone)? I'm guessing leader sections can have odd numbers of cycles -- do they happen anywhere else? Could a custom protection mechanism create them at will?
Could this be behind it (from the Advanced User Guide):
The problem occurred because it was possible for the serial
ULA to get out of sync for a few bits when the 6850 divide bits
were changed. This tended to corrupt the first character of the
first block in a SAVE, or the first character of any block during
sequential access (since the 6850 is reset for each block during
putbytes). The cure is to write a dummy byte to tape at the
start of a SAVE and the start of every block during putbytes.
If the leader of a prerecorded program is played back, a run in
tone followed by a blip (the dummy byte) followed by more
run in tone will be heard. It is necessary to have a run in
period of high tone after the dummy byte. Preferably this
should be done by polling the 6850 to check if the TDR is
empty, since it is difficult to accomplish if the 6850 is
continually interrupting. The 6850 can then be turned on to
interrupt just before starting the block write operation.
See page 393.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

Coeus wrote:
Wed Jun 23, 2021 7:02 pm
Could this be behind it (from the Advanced User Guide):
I'm not really sure. I think that explanation just talks about writing a dummy byte, whereas I'm talking about a single 2400 Hz cycle (which would be half of a bit). All I know about this is that I have read in some document that it can happen (although I can't remember in which piece of documentation I read it). It may only affect the leader tone sections.

I am pretty sure at this point that the PLL idea is toast. It seemed like the "correct" way to lock on to an incoming carrier signal, but I can't get it to work reliably enough for some reason. I suspect the problem is that the signal from the tape isn't actually a true carrier, because it chops and changes between 1200 Hz and 2400 Hz tone quite a lot.

I had a plan to deal with this, which was to reconstruct a coherent carrier by frequency doubling the signal (to produce 2400 Hz carrier from the 1200 Hz zero-bits). This can be done essentially by squaring the signal, via the identity:

cos(2x) = 2.cos^2(x) - 1

This will obviously double the 2400 Hz carrier up to 4800 Hz, so you then have to mix this with the original signal (possibly with some phase shift) to fill in the 2400 Hz tone for the 1-bit sections. The resulting signal will contain a lot of other harmonic garbage, but it should be out of the frequency range we care about. You can then just use an aggressive bandpass filter to filter off the 2400 Hz carrier, and feed that into the PLL.

This is good enough to keep the PLL locked during blocks, but unfortunately it isn't good enough for accurate synchronous sampling of Goetzel power levels. The PLL has to fight to remain locked to this fake carrier signal, with its output frequency constantly jittering this way and that. This jitter will sometimes delay the sampling of a bit beyond the time at which the bit has already ended, leading to errors. (I can get some traces of this phenomenon if anyone really wants to see them.)

I did try to add an "adjustment mechanism" whereby the algorithm can examine the PLL's state and try to make a decision about whether to expedite or delay each sample based on the phase error exhibited by the PLL sometime shortly afterwards. I've implemented this, and it sometimes helps, but can also make things worse.

I'll probably keep the PLL code in quadbike as an option for the curious, and so I can continue to tinker with it. It has actually proved quite useful as a deliberately unreliable source of data for developing higher-level error correction methods.

Anyway, the summary is that I don't think the PLL is going to provide sufficiently reliable sync timings for accurate power sampling unless I have some sort of breakthrough, so I'll probably have to come up with something else.

Right now I'm trying to work on "multi-source mode", which is based on the idea that you frequently have multiple recordings of the same tape. It should be possible to amalgamate these recordings for maximum accuracy. Most straightforwardly this would be the left and right channels of a single stereo file, but another common scenario would be having a tape where the same software is recorded on both sides. I also have a cycle-level error correction algorithm in its early stages.

I recently integrated libsndfile, which is a relief, as I can finally load WAVs and FLACs instead of 8-bit raw data ...
User avatar
BigEd
Posts: 4234
Joined: Sun Jan 24, 2010 10:24 am
Location: West Country
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by BigEd »

How about running a delay locked loop on the output of the Goertzel detectors? (I'm not absolutely sure if DLL is the term I need here: I mean something which resynchronises on each change. Of course, the UART does something a little like this, except it detects the framing and resyncs on each start bit transition - that's what start bits are for! It then has to count up to 8.)

(Edit: spello!)
Last edited by BigEd on Mon Jul 12, 2021 7:46 am, edited 1 time in total.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

BigEd wrote:
Tue Jun 29, 2021 9:38 am
How about running a delay locked loop on the output of the Goetzel detectors? (I'm not absolutely sure if DLL is the term I need here: I mean something which resynchronises on each change. Of course, the UART does something a little like this, except it detects the framing and resyncs on each start bit transition - that's what start bits are for! It then has to count up to 8.)
Yes -- this will likely be one of three synchronisation modes offered.

I'll keep the PLL and slap it with a big OUT OF ORDER sign, as the first option.

As you suggest, cycle-level sync derived loosely from the Goetzel output will be the second option. The problem with this is that the sync source evaporates during leader tone runs, and I want to be able to count exact cycles of leader tone with precision, since this is intended for preservation purposes. I'll therefore probably have to label this option with a warning sign, too, although I suspect this "ad-hoc mode" is likely to achieve lower bit error rates than anything else when dealing with severely degraded media. (Incidentally, one of the nice things about the PLL would have been that you can measure gap lengths on the tape without having to do any extra work -- when the PLL loses lock it just continues to oscillate at its centre frequency, so you can continue to count its cycles for time measurement just as you would if it were locked onto leader tone).

The third and default option will probably just be deriving sync by walking the waveform in the time domain, but still enjoying the benefits of frequency measurements in the frequency domain. You still get to judge the "confidence level" of each bit, which will be vital for further error correction work.

I'll try to put together some sound files of outputs taken from various bits of the PLL, since I found those pretty interesting, particularly the unfiltered phase error.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

It's strange how the PLL + jitter correction seems to work very well on some tapes, but miserably on others.

Beebug magazine cassette vol. 5 number 5 works without losing a bit! Most other tapes generate complete rubbish. I'm not sure what's so special about this one. The jitter correction did have to fix over 600 timing errors, though, so it was hardly a flawless performance.
Screenshot_20210630_153223.png
Before I give up on the PLL, here are some audio treats based on a couple of blocks of tape (bunch of FLAC files):
audible-adventures-in-pll-land.zip
(1.6 MiB) Downloaded 6 times
File 1 is the original signal from my Walkman. File 2 is a 2400 Hz (ish!) carrier rebuilt as described above. File 3 is the output from the PLL's oscillator. File 4 is derived directly from 3 -- it's a digital pulse train which dictates exactly when power sampling should be done. You can hear frequency jitter in all of these 2400 Hz signals. File 5 is probably the most interesting -- it's the phase error from the PLL, which doesn't sound like I expected it to. If it weren't for that shortwave radio whistle, it might make a useful sound effect.
User avatar
vanekp
Posts: 1060
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by vanekp »

a note here all my wav files I have used an audio filter applied in Goldwave as per this article https://acorn.huininga.nl/pub/unsorted/ ... %20CSW.pdf before using csw and the nmakeuef to make the final uef file.
Regards Peter.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

"Mode II sync" is tentatively implemented now -- using a traditional wave walking algorithm to recover a clock signal, and then using that clock signal to trigger sampling of power levels in the frequency domain. This clock recovery algorithm collects a bit more data than some others do: It recognises not just zero-crossings or peaks, but both (and then performs a bad heuristic on the gaps between them).

What I find particularly remarkable about this is that it almost works.
Screenshot_20210706_203659.png
I suspect the errors here are caused by sporadically absent sync pulses causing the frequency domain sampling not to happen. The plan is ultimately to correct for missing sync pulses by interpolating between good ones, but I suspect in this case that a bug (rather than a bad tape signal) may be responsible.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

Scrubbed together the code this morning to replace missing sync pulses through interpolation. Elk Boxer and Snapper now load.
Screenshot_20210707_075948.png
Most other tapes are still erratic, but I'm getting there.
User avatar
Diminished
Posts: 689
Joined: Fri Dec 08, 2017 9:47 pm
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by Diminished »

July appears to have become Cassette Month on Stardot.

I have fixed so many awful bugs over the past week that at this point I am flabbergasted that I managed to capture anything at all prior to this point. I even have some hope that it might be possible to resurrect PLL-sync mode and get some useful results out of it, given some of the mistakes I found on the frequency domain side.

Anyway, I have been continuing to work on walk mode sync (mode II), where peaks and zero-crossings generate sync pulses that are then used to take frequency domain samples. Here is a lewd picture of Elk Boxer:
boxer_capture2.png
The traces, top to bottom, are as follows:

1. Original signal from tape
2. Signal from (1), normalised, bandpass-filtered, and then shifted back in time to eliminate the filter delay
3. Sync signal generated from a time-domain walk of (2)
4. Oversampled Goertzel 1200-Hz power
5. Oversampled Goertzel 2400-Hz power

(Yes, I have previously been spelling Goertzel wrongly. Some search-and-replace on the source code will be required).

Encouragingly, this now loads scarybeasts' Fortress recording. He described it as needing "a few bits patched by hand", but Quadbike's mode-II decodes it without any fuss, and this is before any higher-level error correction has been done.

The big things that need doing now are:

- walk mode currently expects a specific polarity, so I need to find a way of detecting the input polarity, flipping the wave if necessary
- fix multi-source mode, for automatically aligning and amalgamating multiple recordings of the same tape; this is implemented but it doesn't work properly yet
- sync mode III, the free-running mode, which I expect to give the highest possible chance of producing a loadable image (at the cost of pedantic accuracy)
- possible sync mode IV, which would be a hybrid between modes II and III, and probably the best option if it could be made to work

There are a lot of other things that need sorting out too; the code is very much an experimental mess.
User avatar
vanekp
Posts: 1060
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: Quadbike: Experimental, DSP tape capture -- WAVs needed!

Post by vanekp »

Following your progress with interest, it looks like its coming along nicely.
Regards Peter.
Post Reply

Return to “software & utilities for the pc, mac or unix”