hq.uef Specification

want to talk about MESS/model b/beebem/b-em/electrem/elkulator? do it here!
User avatar
Pernod
Posts: 1001
Joined: Fri Jun 08, 2012 10:01 pm
Location: Croydon, UK

hq.uef Specification

Postby Pernod » Mon Nov 02, 2015 6:12 pm

Is there an official specification document anywhere for hq.uef? All I can find online are for the non hq variety.
- Nigel

BBC Model B, ATPL Sidewise, Acorn Speech, 2xWatford Floppy Drives, AMX Mouse, Viglen case, etc.

User avatar
CMcDougall
Posts: 5622
Joined: Wed Feb 02, 2005 3:13 pm
Location: Shadow in a Valley of Scotland

Re: hq.uef Specification

Postby CMcDougall » Mon Nov 02, 2015 10:05 pm

from the MakeUEF manual at end (included in CSW Viewer39), could try giving Fraser a shout.....

Designer and programmer:
Fraser Ross, mail to:fraserross@f2s.com?subject=MakeUEF V2.3
ImageImageImage

User avatar
Pernod
Posts: 1001
Joined: Fri Jun 08, 2012 10:01 pm
Location: Croydon, UK

Re: hq.uef Specification

Postby Pernod » Mon Nov 02, 2015 10:24 pm

CMcDougall wrote:could try giving Fraser a shout.....

Tried and failed, email bounced.
- Nigel

BBC Model B, ATPL Sidewise, Acorn Speech, 2xWatford Floppy Drives, AMX Mouse, Viglen case, etc.

User avatar
b_b_c_m_i_c_r_o_2
Posts: 218
Joined: Sun Jun 25, 2006 10:15 pm

Re: hq.uef Specification

Postby b_b_c_m_i_c_r_o_2 » Mon Nov 30, 2015 11:05 am

I had the same response with that email address when I sent him a query yesterday:


SMTP error from remote mailer after RCPT TO:
<fraserross@f2s.com>:
host a.mx.nildram.net [85.119.248.30]:
553 sorry, unknown email address (#5.7.1)


I am not sure if he is currently active on this stardot forum.

User avatar
Pernod
Posts: 1001
Joined: Fri Jun 08, 2012 10:01 pm
Location: Croydon, UK

Re: hq.uef Specification

Postby Pernod » Mon Nov 30, 2015 11:40 am

- Nigel

BBC Model B, ATPL Sidewise, Acorn Speech, 2xWatford Floppy Drives, AMX Mouse, Viglen case, etc.

User avatar
CMcDougall
Posts: 5622
Joined: Wed Feb 02, 2005 3:13 pm
Location: Shadow in a Valley of Scotland

Re: hq.uef Specification

Postby CMcDougall » Mon Nov 30, 2015 7:59 pm

b_b_c_m_i_c_r_o_2 wrote:not sure if he is currently active on this stardot forum.

fraser:
Joined:Tue May 20, 2003 8:21 pm
Last active: Sat Mar 17, 2012 11:12 am :shock:
ImageImageImage

ThomasHarte
Posts: 366
Joined: Sat Dec 23, 2000 5:56 pm

Re: hq.uef Specification

Postby ThomasHarte » Tue Jan 19, 2016 10:47 pm

Very late to the party, of course.

I'm Thomas Harte, I knocked up the original UEF specification before Fraser Ross got rigorous on it. An accurate version might be that I'm responsible for the deficiencies, he's responsible for the successes.

That being said, I therefore at least can provide the information you want: the draft 28 spec that is linked to is the full .hq.uef specification.

An approximate version of the history is that the first draft was more or less just chunks 0100, 0110, 0111 and 0112. In fact I think that's all my original MakeUEF recognised and almost certainly covers 99% of Electron UEFs out there. I added 0101 to improve BBC compatibility but that's limited in scope.

Aside: MakeUEF was itself perhaps a little askew in its design; it effectively reproduces the ROM logic for loading files and then, having loaded them, saves them out again in UEF form. They're therefore mostly going to be very uniform and on-spec in terms of carrier tone length, gaps, etc, regardless of the original masters. The best excuse I can make is that I was working on a lot of things at the time, that the file format at least didn't bake in those assumptions, that I needed MakeUEF to be only good enough to produce transfers of the tapes I actually owned in the then-absence of any other Electron tape captures online, etc, etc, etc. Excuses aplenty, in summary.

Fraser was correctly concerned both that the UEF format wasn't sufficient for accurate preservation of tapes and specifically that MakeUEF wasn't attempting an accurate preservation of tapes (versus preservation of the software that once was on tapes in a manner that is compatible with tape storage).

The HQ additions then are primarily 0104, to hit a bunch of other BBC possibilities that it might be nice to store in shorthand but, importantly, to allow Atom format through the not-strictly-Kansas-City extra short wave, 0116 to increase the precision of gaps, and 0114, 0115 and 0116 to record patterns with a less-coupled correlated meaning as data, to allow a switch of default encoding and to record phase changes.

Motivation being simply that anything any of the Acorn machines could discern as a difference from a tape should be recordable in a tape preservation format.

If you support all of the 01xxs then you support HQ.UEF.

Fraser also supplied CSW reading code for ElectrEm as a more targeted way of preserving tapes; as per the previous edit of this post my recollection was that he pioneered the format but I see the Ramsoft definition based on a quick Google search and could not say for certain that the same name is just a coincidence.

User avatar
vanekp
Posts: 342
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands

Re: hq.uef Specification BUG

Postby vanekp » Fri Nov 10, 2017 11:21 pm

I think there is a bug is makeuef it switches the parity 8E1 and 8O1 around.....
how do i know this I created a file with 4 blocks in it 1st block 8N2, 2nd 8N1, 3rd 8E1, 4th 8O1 (writing &11, &15, &19, &1D to &FE08) on my BBC I can read them back with a program confirming they are correct.
but in order to successful create a uef file from this wav I have to switch the parity for blocks 3 and 4 around, see makeuef log below

Code: Select all

MakeUEF V2.3.

Command line:
-i paritytest.csw -w 0 180 -z 1 8n2 -z 2 8n1 -z 3 8o1 -z 4 8e1

Date (D/M/Y) and time:
11/11/2017   00:01

Expected baud rate is 1201.

Gap written, length 0.08730159 seconds.


Carrier tone written, length 18406 waves.
Baud rate of the preceding data was 1201.091

The following block size is 34
Data block written with a non-standard data format.
1 data blocks written.
Carrier tone written, length 1968 waves.
Baud rate of the preceding data was 1201.095

The following block size is 34
Data block written with a non-standard data format.
2 data blocks written.
Carrier tone written, length 1964 waves.
Baud rate of the preceding data was 1201.1

The following block size is 34
Data block written with a non-standard data format.
3 data blocks written.
Carrier tone written, length 1970 waves.
Baud rate of the preceding data was 1201.088

The following block size is 34
Data block written with a non-standard data format.
4 data blocks written.
Carrier tone written, length 13900 waves.
Security waves written, length 1 waves, as follows:
S,
The security waves end with a pulse.
Baud rate of the preceding data was 1201.103


Gap written, length 0.2037415 seconds.



I got to wondering because 8bitkick was having problems with parity in his PLAYUEF program and made a comment that the parity is inverted.
Peter.

User avatar
8bitkick
Posts: 68
Joined: Thu Aug 11, 2016 4:45 pm
Location: California
Contact:

Re: hq.uef Specification BUG

Postby 8bitkick » Sat Nov 11, 2017 2:01 am

vanekp wrote:I think there is a bug is makeuef it switches the parity 8E1 and 8O1 around.....
how do i know this I created a file with 4 blocks in it 1st block 8N2, 2nd 8N1, 3rd 8E1, 4th 8O1 (writing &11, &15, &19, &1D to &FE08) on my BBC I can read them back with a program confirming they are correct.
but in order to successful create a uef file from this wav I have to switch the parity for blocks 3 and 4 around, see makeuef log below
<snip>
I got to wondering because 8bitkick was having problems with parity in his PLAYUEF program and made a comment that the parity is inverted.
Peter.


So... trying to implement UEF chunk 0x0104 in PlayUEF I was testing Video Classics-Firebird-Silverbird-hq.uef (which uses a custom loader) generated by MakeUEF V1.9 via CSW.

It only loads when I invert the parity bit in the audio stream wrt to the 0x0104 UEF chunk encoding. See debug log below of first 0x0104 chunk of that file... it's is the inverse of what I would expect at least...

I am more than willing to accept I have make a mistake along the way! But something seems off with the way the UEF was generated, consistent with what Peter found.

This doesn't load (but looks right to me)

Code: Select all

UEF chunk - 0x0104 0000011f bytes of 8O1
=====================================
0f 8O1 = START:0 11110000 Parity: 1 STOP:1
f0 8O1 = START:0 00001111 Parity: 1 STOP:1
0f 8O1 = START:0 11110000 Parity: 1 STOP:1
2f 8O1 = START:0 11110100 Parity: 0 STOP:1


Game loads OK on BBC issue 7 (But this is 8E1 not 8O1... so UEF chunk was encoded wrong...?)

Code: Select all

UEF chunk - 0x0104 0000011f bytes of 8O1
=====================================
0f 8O1 = START:0 11110000 Parity: 0 STOP:1
f0 8O1 = START:0 00001111 Parity: 0 STOP:1
0f 8O1 = START:0 11110000 Parity: 0 STOP:1
2f 8O1 = START:0 11110100 Parity: 1 STOP:1

ThomasHarte
Posts: 366
Joined: Sat Dec 23, 2000 5:56 pm

Re: hq.uef Specification BUG

Postby ThomasHarte » Sat Nov 11, 2017 5:06 am

8bitkick wrote:
vanekp wrote:I think there is a bug is makeuef it switches the parity 8E1 and 8O1 around.....
....


This doesn't load (but looks right to me)
...

That looks right to me too, and matches my current implementation unless I'm suffering a failure of comprehension:

Code: Select all

      uint8_t parity_value = byte;
      parity_value ^= (parity_value >> 4);
      parity_value ^= (parity_value >> 2);
      parity_value ^= (parity_value >> 1);

      switch(parity_type) {
         default: break;
         case 'E': queue_bit(parity_value&1);         break;
         case 'O': queue_bit((parity_value&1) ^ 1);   break;
      }


I'm therefore inclined to agree with vanekp's assessment.

User avatar
vanekp
Posts: 342
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands

Re: hq.uef Specification

Postby vanekp » Sat Nov 11, 2017 9:44 am

So does this mean all UEF files that make use of 8E1 8O1 parity are not 100% replicas of the original?
And guess nothing will be done to fix the bug in makeuef as the last person (fraser) to have been working on it does not seem to be around anymore.
Peter.

ThomasHarte
Posts: 366
Joined: Sat Dec 23, 2000 5:56 pm

Re: hq.uef Specification

Postby ThomasHarte » Sat Nov 11, 2017 8:31 pm

I'll write a new MakeUEF if needs be; although I was very naive about things like signal processing when Fraser took the reins, I fancy I'm somewhat better now. I definitely prefer that plus correction and republication of all files found to be problematic as a solution to altering the specification. Is that unrealistic?

User avatar
vanekp
Posts: 342
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands

Re: hq.uef Specification

Postby vanekp » Sat Nov 11, 2017 9:34 pm

Makes more scene as it is very confusing for others that are trying to do something with a UEF file (e.g. 8bitkick) to find out after hours of hair pulling that the specifications are not what one expects in this case that 8E1 and 8O1 are swapped around.
Peter.

User avatar
8bitkick
Posts: 68
Joined: Thu Aug 11, 2016 4:45 pm
Location: California
Contact:

Re: hq.uef Specification

Postby 8bitkick » Sun Nov 12, 2017 5:45 am

ThomasHarte wrote: That looks right to me too, and matches my current implementation


Hi - now we have solved the mystery, I've updated PlayUEF to check for the version of MakeUEF reported in 0x0000 info chunk, and handle 0x0104 parity accordingly.

Wish I had known about TapeUEF.cpp before I'd embarked on PlayUEF! Nice to have a definitive implementation to refer to now 8)

User avatar
vanekp
Posts: 342
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands

Re: hq.uef Specification

Postby vanekp » Mon Nov 27, 2017 5:05 pm

Grrr ran into the problem again of 8E1 8O1 being swapped around in a UEF file *Sigh*

User avatar
8bitkick
Posts: 68
Joined: Thu Aug 11, 2016 4:45 pm
Location: California
Contact:

Re: hq.uef Specification

Postby 8bitkick » Mon Nov 27, 2017 6:06 pm

vanekp wrote:Grrr ran into the problem again of 8E1 8O1 being swapped around in a UEF file *Sigh*


PlayUEF now looks if current/earlier MakeUEF version was used to generated given UEF and inverts parity if so... if PlayUEF is not handling it right, let me know what UEF file

ThomasHarte
Posts: 366
Joined: Sat Dec 23, 2000 5:56 pm

Re: hq.uef Specification

Postby ThomasHarte » Mon Nov 27, 2017 6:21 pm

Stupid question, probably: where is the current MakeUEF available from? Every link I can find points back to the sadly-departed http://www.acornpreservation.org/. It'd be interesting to see how easy a straight binary hack of the executable would be, since all we really want to do is change three string constants — the version number, and switch around an 'O' and an 'E' — and it's a fairly safe bet that it's an unsigned, unencrypted product.

User avatar
vanekp
Posts: 342
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands

Re: hq.uef Specification

Postby vanekp » Mon Nov 27, 2017 8:08 pm

only place I can find it for download is http://www.stairwaytohell.com/essentials/MakeUEF-v2.3.zip but as you say a lot of the links point to the long debunked Acorn Preservation site. I don't know the code so have no idea where to look in it. O it is compressed with UPX (UPX -d makeuef.exe to uncompressed it)

And 8bitkick No its my own little project where I was getting nowhere then I remembered the mix up in the parity and wanted to kick myself for not remembering and wondering why I could not get a copy of the game to work on my BBC.

need to look in the makeuefdll.dll file (it's also compressed with UPX)
makeuef.png

But if I swap these two it complaint about an invalid switch.

Fraser
Posts: 543
Joined: Tue May 20, 2003 7:21 pm

Re: hq.uef Specification

Postby Fraser » Fri Dec 01, 2017 11:13 am

I have just discovered this thread and I'll have a look at it and some other things that have changed in the last few years. My email address is now fraser.ross8@btinternet.com. Has the site acornpreservation.org not been working for a while now? I don't think the Electrem site has been working either. You can see a backup here: https://web.archive.org/web/20120505062 ... ron.co.uk/ The csw.exe program does not work on Windows 10 so I am thinking about writing a replacement.

Fraser
Posts: 543
Joined: Tue May 20, 2003 7:21 pm

Re: hq.uef Specification

Postby Fraser » Fri Dec 01, 2017 11:42 am

If there is a bug with the parity bit, and I think there probably is, then I'll compile a new MakeUEF with it fixed. Unfortunately some emulators that read hq.uef presumably have the bug too.

User avatar
vanekp
Posts: 342
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands

Re: hq.uef Specification

Postby vanekp » Fri Dec 01, 2017 2:25 pm

I run CSW in a dosbox or on a old XP computer but it would be great if there was a windows compatable version of it, would make life a lot easer.
And thanks Fraser for your reply on this thread glad your still around we all though we had lost you here :D
Yes I was wondering about that as well that it could cause a ripple effect with other emulators, BeebEm does not care about parity anyway so it will not be effected, not sure about others though.
Ahhh that explains why some games will not load in mame as I think mame is correct because if I make a wav file on my bbc i.e. with the correct parity switching say like fortress then it works fine in mame but not one that comes from an exsisting uef file where the parity is inverted.
Peter.

ThomasHarte
Posts: 366
Joined: Sat Dec 23, 2000 5:56 pm

Re: hq.uef Specification

Postby ThomasHarte » Fri Dec 01, 2017 3:01 pm

Fraser wrote:If there is a bug with the parity bit, and I think there probably is, then I'll compile a new MakeUEF with it fixed.

Hooray! I'd got no further than reading up on Goertzel's algorithm versus, say, one of the FM-decoding algorithms for discriminating phase, and determining that of the other micros I've looked up, only the MSX also uses the Kansas City standard, but those guys have the worst possible tape images — file contents only — so there's little transferable from there in terms of available code.

Fraser wrote:Unfortunately some emulators that read hq.uef presumably have the bug too.

From what I can make out, B-Em doesn't even retain the bits per packet or parity type in memory, and BeebEm at least has the flags in memory, but just skips over them. In both cases it looks like the authors have decided to hard-code a 1:1 mapping between UEF data bytes and BBC perceived bytes.

jsBeeb treats the UEF properly as a wave source, but luckily is also presumably pretty easy to deploy updates to.

Fraser
Posts: 543
Joined: Tue May 20, 2003 7:21 pm

Re: hq.uef Specification

Postby Fraser » Mon Dec 04, 2017 1:20 pm

I was wondering how you wrote the CRC routine for cassette blocks? The BBC advanced user guide has an assembly routine that you might have derived it from.

ThomasHarte
Posts: 366
Joined: Sat Dec 23, 2000 5:56 pm

Re: hq.uef Specification

Postby ThomasHarte » Mon Dec 04, 2017 5:28 pm

Fraser wrote:I was wondering how you wrote the CRC routine for cassette blocks? The BBC advanced user guide has an assembly routine that you might have derived it from.

It turns out to be the same polynomial as the standard CCITT CRC16, as also sighted in X.25, V.41 the Bluetooth spec, and as part of IBM-format floppies, though with a reload value of zero. So I think in the olden days I probably copied it from somewhere, though my current little class started as the most obvious bit shifter direct from the definition of a CRC, then switched to table based in the most obvious manner. So it's probably identical to a million other table-based implementations.

User avatar
Arcadian
Posts: 2804
Joined: Fri Nov 24, 2000 12:16 pm
Contact:

Re: hq.uef Specification

Postby Arcadian » Tue Dec 05, 2017 11:54 am

(Wow, it's Fraser! Welcome back fella! :) ).
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk

Image
ABug NORTH (Manchester) (19-21 January 2018)
ABug SOUTH (Hampshire) (1-3 June 2018)


Return to “emulators”

Who is online

Users browsing this forum: No registered users and 6 guests