Wires? Pah.....

discuss PC<>Acorn file transfer issues & the use of FDC, XFER, Omniflop/disk etc.
User avatar
mga1103
Posts: 184
Joined: Mon Jan 24, 2011 4:00 pm
Location: Galway, Ireland
Contact:

Re: Wires? Pah.....

Post by mga1103 » Sat Aug 06, 2011 6:12 pm

(Back again! :wink: )

While certain parties have been off enjoying themselves at weddings, others have been putting the latest beta through it's paces :twisted: .

@Martin: Before I got to testing the latest functionality, I tried some transfers of good/formatted/unprotected disks and found a few small differences between the transferred image and the original. Looks like you may have nudged that delicate timing balance somewhere :wink:

I've sent you an email with the test data for your perusal (hope the eyesight is 100% and hasn't deteriorated towards stag-level after the wedding :lol: ).

Hope both weddings went well and all had a blast (and that this post isn't ruining anyone's day :wink: ).

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Sun Aug 07, 2011 4:53 pm

Cheers for that Martin :D

Yes, slowly rejoining the living after getting to bed significantly after 3am (I'm told) :shock:

Got the data and had a quick look - you're right in that it's complete bytes that are missing which actually isn't symptomatic of bit timing errors timing in the strict sense since these tend to manifest as corrupted bytes but not whole byte loss. Looks more like a logical problem such as buffer pointers getting in a knot or something similar and could even be a handshake issue such as the Beeb not respecting PC RTS :-k

One question, is this using USB RS232 or a legacy port?

User avatar
mga1103
Posts: 184
Joined: Mon Jan 24, 2011 4:00 pm
Location: Galway, Ireland
Contact:

Re: Wires? Pah.....

Post by mga1103 » Sun Aug 07, 2011 5:48 pm

This was using the legacy port. (The valid transfers, using the V4 UPURS rom, were completed with the same setup). Didn't actually try the beta with the USB adaptor...yet!

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Sun Aug 07, 2011 6:25 pm

Hmmm, looks like you guys have been busy. I've been off my feet most of the weekend as I appear to have developed non-insertional tendinitis of the Achilles tendon so I've been laid up with ice packs and taking anti-inflammatories to try to take the gob stopper sized lump down a bit :( It developed whilst driving to the reception so no "antics" caused it... Still, off to the quack tomorrow to see if he agrees with my diagnosis...

So, although I haven't done much testing I have managed to get some screenshots and vids that I'm just converting at the moment.

Paul

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Sun Aug 07, 2011 9:46 pm

Bad news on the foot Paul, hope you get it sorted ok :(

Brain functionality mostly restored now so just done some transfer testing and you are indeed correct 'A' Martin, we have a hiccup! I've just tried via USB and got missing data as you discovered so clearly I have unintentionally screwed something up somewhere. I haven't deliberately touched the transfer side as such but I am prone to leaning on the delete key when editing so anything's possible :roll:

I will be back.... :wink:

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Mon Sep 05, 2011 7:38 pm

Wow, it's been three weeks since we updated this thread. Needless to say, MartinB, MartinA and I have all been busy and quieter than normal but that doesn't mean we've been neglecting working on UPURS, far from it in fact.

Since the last posts we've been beavering away doing countless hours of testing and MartinB has tweaked the transfer engine somewhat to make sure it's as good as it can be and we're almost there now but to satiate our collective OCD, we need some outside help.

So, do you run a flavour of Windows on an AMD and related architecture PC or a flavour of Linux on pure Intel architecture PC? If you answered yes to either of these, UPURS needs you!

We've been testing pretty thoroughly and have done hours and hours of successful transfers, testing to destruction but between the three of us, we don't have those two platform combinations and we'd really like to make sure that the transfer engine is tip top in all circumstances :D

There's a specific set of tests that needs to be carried out on these particular hardware/OS combinations to help in the final stages of testing UPXSSD 3.3d. Running the tests should take no more than 20-30 minutes and would help the UPURS project a great deal.

So if you're ready, willing and able, volunteer below :D

Thanks

Paul

User avatar
BigEd
Posts: 2266
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Wires? Pah.....

Post by BigEd » Mon Sep 05, 2011 8:17 pm

I've got linux on intel. I've also got an AMD, but not running windows... could the program run in a minimal windows live CD like BartPE? (or is that too weird and entirely unnecessary?)

Both of those machines have genuine serial ports.

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Mon Sep 05, 2011 8:31 pm

Hi BigEd,

Linux on Intel will do just fine :D

I guess a bootable CD for the Windows environment would work as long as it has hardware access to the serial ports and you can run a serial terminal client like Hercules, I guess HyperTerm will be on the bootable CD image but I'm not sure any of us have played with that and UPURS at all so that would add an extra element to the testing.

Paul

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Mon Sep 05, 2011 8:37 pm

Hyperterminal is poo :evil:. It's unresponsive at high speeds and doesn't support binary receive, just text capture where it is prone to rejecting certain characters :roll:

Doesn't have to be Hercules that's used but please, please not Hyperterminal :wink:

CanonMan
Posts: 477
Joined: Fri Jul 15, 2011 8:41 am
Location: Cheshire
Contact:

Re: Wires? Pah.....

Post by CanonMan » Mon Sep 05, 2011 9:06 pm

Hyperterminal is shite, try RealTerm instead.

User avatar
mga1103
Posts: 184
Joined: Mon Jan 24, 2011 4:00 pm
Location: Galway, Ireland
Contact:

Re: Wires? Pah.....

Post by mga1103 » Mon Sep 05, 2011 9:14 pm

Hi BigEd,
Email sent with details of the various tests we've been performing...

Thanks for your offer of help!
-Martin (A) :wink:

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Mon Sep 05, 2011 9:19 pm

Yes, cheers Ed :D

@ CanonMan : I actually evaluated 23 :shock: different terminal programs (all free ones) whilst looking for a recommendation I could make for use with UPURS including RealTerm. I rejected many because they had some crucial capability deficiency but also many more because they were just too much for the target audience. Ideally, after the settings, there should just be Start & Stop Rx and Start & Stop Tx so as not to overface the non-technical.

Whilst RealTerm is excellent for serious users, it's interface is way too fussy for this application. In the end, that's why I settled on Hercules because there's minimal set-ups required and after a few minutes it becomes very intuitive to use, even for a novice.

User avatar
nOmArch
Posts: 1330
Joined: Fri May 21, 2010 7:27 pm
Location: Gloucestershire
Contact:

Re: Wires? Pah.....

Post by nOmArch » Mon Sep 05, 2011 9:43 pm

I have various Intel and AMD windoze machines all running Windows 7 x86 & x64 if that's any help let me know!
Alex

Back up to 1 Beeb again. \o/

CanonMan
Posts: 477
Joined: Fri Jul 15, 2011 8:41 am
Location: Cheshire
Contact:

Re: Wires? Pah.....

Post by CanonMan » Mon Sep 05, 2011 10:53 pm

MartinB wrote: Whilst RealTerm is excellent for serious users, it's interface is way too fussy for this application. In the end, that's why I settled on Hercules because there's minimal set-ups required and after a few minutes it becomes very intuitive to use, even for a novice.
Okay, you're right about that :)

RealTerm is incredibly powerful but it's way too user-unfriendly for most users.

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Mon Sep 05, 2011 10:58 pm

Alex,

The AMD/Windows combination would complete the set of hardware/OS platform testing so that'd be great if we could sort that out tomorrow :D

Paul

User avatar
1024MAK
Posts: 8202
Joined: Mon Apr 18, 2011 4:46 pm
Location: Looking forward to summer in Somerset, UK...
Contact:

Re: Wires? Pah.....

Post by 1024MAK » Mon Sep 05, 2011 11:22 pm

I have a Pentium 4 that runs Linux. Also an AMD that runs Linux.
I have a number of machines, but I don't think that any of them are AMD running any flavour of Windows (I will have to check, I have W95, ME, W2000, XP and Vista).
I think that all of them have normal serial ports, plus I have four USB to RS232 converters (but one is missing in action somewhere, been looking for it for two months now, on and off :roll: :lol: ).
Two snags :( : first is that I can only do a little bit of limited testing for the next few days.
Will however be able to do a few hours worth of testing Thursday to Sunday.
Second, my favourite BBC B has decided that it would like some TLC of the soldering kind :( , but I have not yet got around to ordering the parts needed. Have found out that another of my Beeb's does not like it's disk interface :evil: . So I hope that my one remaining working Beeb is okay.
I do have a working Master 128 to use for testing.

Let me know what testing you want done... :mrgreen:

Mark K.

User avatar
nOmArch
Posts: 1330
Joined: Fri May 21, 2010 7:27 pm
Location: Gloucestershire
Contact:

Re: Wires? Pah.....

Post by nOmArch » Tue Sep 06, 2011 9:47 am

paulv wrote:Alex,

The AMD/Windows combination would complete the set of hardware/OS platform testing so that'd be great if we could sort that out tomorrow :D

Paul
Yep not a problem :)
Alex

Back up to 1 Beeb again. \o/

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Tue Sep 06, 2011 6:23 pm

@Mark K @Alex

Do you both have a UPURS cable or can you make one up relatively quickly as per the UPURS manual?

If so, can you PM me with your e-mail addresses and we can sort out the testing process with both of you :D

Paul

User avatar
BigEd
Posts: 2266
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: Wires? Pah.....

Post by BigEd » Thu Sep 08, 2011 9:03 pm

Bits flying up and down the wire! After a bit of a shaky start, not knowing one end of a Master from the other, all tests passed. This thing flies - great documentation too!

(And not once did I overwrite my utility disk with test data. One is labelled "Carriers at War" and the other "Russia" so they're easily distinguished. I've no idea where they came from.)

Cheers
Ed

(Only tested on Linux so far - leaving the BartPE adventure for an emergency.)

User avatar
nOmArch
Posts: 1330
Joined: Fri May 21, 2010 7:27 pm
Location: Gloucestershire
Contact:

Re: Wires? Pah.....

Post by nOmArch » Thu Sep 08, 2011 9:10 pm

paulv wrote:@Mark K @Alex

Do you both have a UPURS cable or can you make one up relatively quickly as per the UPURS manual?

If so, can you PM me with your e-mail addresses and we can sort out the testing process with both of you :D

Paul
Unfortunately no but how much and where can I get one and I'll see what I can do.
Alex

Back up to 1 Beeb again. \o/

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Thu Sep 08, 2011 9:52 pm

I've got one spare cable already made up (but it's short at about 50cm) and I have enough spare bits to whip up another if a longer cable is required :D

MartinB charges £6 for the cables so he or I could send you one.

PM one of us for details and to arrange where to send the cable to :D

Paul

User avatar
nOmArch
Posts: 1330
Joined: Fri May 21, 2010 7:27 pm
Location: Gloucestershire
Contact:

Re: Wires? Pah.....

Post by nOmArch » Thu Sep 08, 2011 10:02 pm

PM Sent Paul.
Alex

Back up to 1 Beeb again. \o/

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Sat Sep 24, 2011 8:56 pm

If you’ve been following the thread, you’ll know it’s been quiet for a while apart from a request for help with testing and I’ll let the other UPURS team members update you on that exercise.

Anyway, the time has come to release the all-new, best-ever, shiniest (and probably final) version of the UPURS transfer suite, Version 4.1 :D
The rom is at V4.1 and all the individual disc and rom-based utilities are at Version 4.0x (suffix ‘B’ for disc and suffix ‘R’ for rom).
UPURS V4.1.png
UPURS V4.1.png (8.24 KiB) Viewed 2773 times
The latest round of changes began by adding the capability to deal with damaged, corrupted or ‘sneaky’ discs in the image_to_pc utilities and that capability has now been comprehensibly implemented through the use of the ‘I’ switch. During the development and testing of this capability, my esteemed and tireless project colleagues, paulv and mga1103 (Paul & ‘A’ Martin), discovered some quirks in the behaviour of Windows and it’s interaction with certain PC hardware. Skipping the detail for now, their discovery eventually led to my completely rewriting the Beeb RS232 bit-bang Tx engine in order to ensure maximum compatibility with the vagaries of Windows and again, Paul & ‘A’ Martin will undoubtedly expand on this. Suffice to say though that UPURS now boasts the most compliant RTS/CTS 115k RS232 implementation anywhere ever \:D/

The new suite is attached below and as soon as Paul has dropped the update onto his site together with his awesome user-manual, you’ll be able to get the everything from there.

Enjoy :D

(Lee - WE DD export versions now at the top of my in-tray!)
Attachments
UPURS V4.1 (Disc & Rom) Formal Release.zip
(4.98 KiB) Downloaded 147 times

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Sat Sep 24, 2011 9:32 pm

It's safe to say that UPURS 4.1 is the most thoroughly and expansively tested version of the UPURS suite to date. Collectively we've invested a huge amount of time and literally tested UPURS to destruction. In doing so, and as Martin has already said, he's been able to refine the serial transmission engine and add in some useful features to significantly increase the flexibility of the transfer system. It is now undoubtedly one of the fastest ways to transfer disc and rom images between the Beeb and a PC and unique in it’s capability to maximise data recovery from corrupted or damaged discs.

During the development of UPURS and in performing the recent intensive testing, we actually managed to uncover an I/O issue with various flavours of Windows which can, on rare occasions, cause a transfer not to be successful when Windows is running on a PC with an Intel chipset. This is one of the reasons we asked for extra volunteers a couple of weeks ago to do some blind tests and confirm our findings.

Through the testing of all the different platform configurations of Linux/AMD, Linux/Intel, Windows/AMD and Windows/Intel, we've essentially identified the most probable cause of the problem as being restricted to the Windows/Intel combination and more specifically, a Southbridge driver issue. As such, we've been able to create situations in which we can predictably replicate the problem and from that formulate advice that mitigates the issue should a given user’s PC be susceptible to this potential I/O short-fall under Windows like both Martin A's and mine are. It has to be said, it took me *ages* to actually replicate what Martin A originally found and that's a good thing really as it goes to show just how hard you have to be pushing the PC to cause the problem in the first place.

It basically comes down to this. If the receiving Wintel PC is under a significant I/O load (be it disk, network or anything that goes through the Southbridge driver), then there is scope for the PC to fail to correctly schedule servicing of the serial interface when receiving high-speed data from the BBC Micro and this can lead to individual bytes or groups of bytes not being recorded by the PC during the transfer. Fortunately, the solution is quite simple should you happen to encounter the problem on your set-up. When transferring data to a PC of this type, the user should ideally close all other applications and limit I/O usage throughout the duration of the transfer. In addition, Martin has added a byte count to the export processes of UPURS so you can simply do a visual check on the received data byte count on the PC or check the file size of the received file compared to the sent byte count on the Beeb. Thus, even though it is a rare occurrence, you can never be caught out by this Wintel quirk and unknowingly create an incomplete image of a disc.

At this stage, I'd just like to say thanks to Martin B for putting up with both myself and Martin A as we really pushed UPURS (and probably Martin B too) to its limits and we both carried out hundreds of transfers during the testing. So much so that I semi-automated my testing by programming the function keys of my Beeb with different UPURS commands to speed things up :lol:

I'd also like to extend our thanks again to 1024MAK, BigEd and n0mArch who volunteered for the extra testing that we asked for. They essentially worked blind as we didn't want to affect the outcome of their results and in the end, all their test results confirmed our own so we're very grateful for their help.

The manual is attached to this post and both the full suite and manual are also available from my site here

Paul
Attachments
User Port Utilities Rom Software v4.1.zip
(1.3 MiB) Downloaded 152 times

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Sat Sep 24, 2011 9:36 pm

Ooohh.... Nearly forgot to mention.

FTDI have launched an official BETA of their drivers for the USB-RS232 cable so the "unofficial" version provided by FTDI that Martin B has been distributing is now superseded by the latest BETA version 2.08.17 which is available from the FTDI site

Be careful not to download the official WHQL approved drivers, they're still the buggy ones so as long as your FTDI drivers are 2.08.15 or above, you should be fine :D

Paul

User avatar
mga1103
Posts: 184
Joined: Mon Jan 24, 2011 4:00 pm
Location: Galway, Ireland
Contact:

Re: Wires? Pah.....

Post by mga1103 » Sun Sep 25, 2011 3:16 pm

Well, it was certainly an interesting experience being involved in the UPURS testing, to say the least! The entire suite has evolved into a fantastic set of useful utilities. I take my hat off to Mr. Barr for sticking with it and bringing it to this advanced stage, despite the various attempts by myself and Paul to scuttle his developments =D> .

Just to add a little to Paul's excellent description, as my Windows/Intel system is acting a little differently in certain circumstances. It was by no means an easy or quick task to narrow down the I/O anomaly that Paul referred to above. In fact, at one stage the 6522 VIA's in my beeb even came under suspicion, so much so that I ordered a few spares. (Thanks Mark!). But in the end, it was becoming increasingly obvious that we either had a Windows driver issue or an Intel Southbridge chipset problem, so my 6522's shall remain as spares for now :wink: .

To test the Windows theory, I installed Windows XP on my Linux/AMD system. Afterall, this platform performed flawlessly under Linux - I've never been able to force a single hiccup (either with FTDI or legacy RS232) even with hammering the system with cpuburn and spew to induce unnaturally high workloads.

Windows XP on the same AMD hardware also performed admirably, so this essentially eliminated the core Windows (XP) as a possible suspect.

My Windows/Intel system has also passed all tests when using the FTDI adaptor.

So in my instance, this apparent I/O anomaly only occurs with legacy RS232 port transfers on my Windows/Intel configuration.

Now here's the interesting thing. If I simply disconnect my NIC (a Broadcom NetXtreme Gigabit Ethernet card), it works perfectly! Even stressing the system (with multiple instances of SuperPI running and multiple disk-to-disk transfers) to the extent that a single transfer can take up to 20 minutes, it still delivers perfect transfers every time. So in my case, it's an easy workaround should I ever need legacy port transfers, but boy is it frustrating to not be able to find a permanent fix to this particular case.

During detailed system and process monitoring, I haven't been able see any system resource that's a likely contributor. Even monitoring IRQ's, there's no noticeable difference whether the NIC is plugged in or not. (Initial thinking suggested it may have been IRQ/DPC related). Yes, I/O is obviously higher with the NIC connected, but nowhere near as high as when multiple disk-to-disk operations are being executed.

So as Paul said, it appears to be related to the Super I/O chip in some way, but only occurs under certain unique conditions. Mine is an Intel 82801GB/GR (ICH7 Family) LPC Interface Controller – 27B8.

I'm also acutely aware that going back to UPURS V3.x, legacy transfers worked perfectly on this same configuration (with NIC intact). Thus, I think it's fair to say that something somewhere in my configuration has changed since the earlier V3.x tests and thus it should be possible to tweak that something to avoid this anomaly, without disconnecting the NIC. But what? :?

Is it possible that serial and network I/O are somehow sharing a resource (buffers, path, etc.) that the Super I/O chip just can't handle under certain loads, a resource that doesn't come into play during normal disk and USB (FTDI) I/O operations?

(If anyone knows the engineers that designed this chip, feel free to give them a call - their input would be very useful right now? :wink: :) )

I've totally exhausted my limited knowledge in this Windows/Super I/O chip environment, so if anyone can add to this or offer other theories, I think the entire punch-drunk "UPURS test team" would love to hear them!

Martin.

User avatar
billcarr2005
Posts: 1263
Joined: Fri Sep 09, 2005 3:01 pm
Location: UK
Contact:

Re: Wires? Pah.....

Post by billcarr2005 » Mon Sep 26, 2011 9:27 am

Congratulations on getting all the problems ironed out!
A wonderful set of utilities :)

User avatar
paulv
Posts: 3733
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Wires? Pah.....

Post by paulv » Mon Sep 26, 2011 11:44 pm

So I figured I'd create a video for people that haven't used UPURS to see how it works when exporting an image to a PC.

Enjoy :D



Paul

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Fri Oct 07, 2011 11:07 pm

Partly because of a request by LeeB and partly through technical curiosity, I've written two special versions of the UPURS disc export utilities which allow WE double-density discs to be transferred to PC using the UPURS system. The two utilities, which are supplied as disc-based versions only, are UPXSSDD and UPXDSDD (note the trailing 'D') and are on the attached ssd. They function exactly as documented in paulv's official UPURS manual with one caveat - a 'used sectors' image can only be created if the WE disc has the standard size catalogue. If the disc has the extended (62) file catalogue, you must use the full (F) image switch or one of the full+ignore error (I4 or I8) switches because whilst imaging extended catalogue discs is supported, I haven't re-written the used-space analysis to take account of the WE extended catalogue option.

I have written the utilities to be compatible with both Acorn 177x DFS (tested with 2.26) and WE DDFS (tested with 1.54T) so it isn't necessary to have the WE DDFS rom fitted to image a WE double-density disc. In the case of the Acorn 177x DFS, I manipulate the 177x in double-density mode even though these DFS do not intrinsically offer DD support at user level.

For testing, I used a WE 1.54T machine to create a genuine DD disc and then used the same machine and an Acorn 2.26 machine to export an image with UPXSSDD & UPXDSDD. The resulting image can then be loaded into BeebEm or B-em and manipulated as desired.

These utilities are supplied as UPURS 'bonus' programs to allow recovery and archiving of WE DD floppies but DD support is not offered throughout the full UPURS suite.

@ LeeB : When you get a chance, can you try some transfers and report back because it usually takes me a few versions to get things right :roll:. (Just ask 'A' Martin or Paul :wink:)
Attachments
UPURS WE Double Density Export Utils.zip
(1.41 KiB) Downloaded 141 times

User avatar
MartinB
Posts: 5051
Joined: Mon Mar 31, 2008 9:04 pm
Location: Obscurity
Contact:

Re: Wires? Pah.....

Post by MartinB » Sat Oct 08, 2011 9:34 am

I wrote:...it usually takes me a few versions to get things right
Lol - just noticed that whilst the double-density reading is working fine and I've provisioned for and am transmitting 18 sectors per track to the PC, I missed updating a constant and I'm only reading 10 sectors from the disc. Plonker [-X

Update later.... :wink:

Post Reply