Domesday86 Project

for bbc micro/electron hardware, peripherals & programming issues (NOT emulators!)
User avatar
DutchAcorn
Posts: 1631
Joined: Fri Mar 21, 2014 9:56 am
Location: Maarn, Netherlands

Re: Domesday86 Project

Postby DutchAcorn » Mon Jul 17, 2017 9:16 am

Also supported! :D

Not an owner of the BeebSCSI or Smallymouse, but very much a supporter of conservation projects! =D>
Paul

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Mon Jul 17, 2017 9:31 am

Thanks chaps; it's highly appreciated! I need to get another FPGA development board and some ARM Cortex M4 dev boards, so every donation helps :)

noggin
Posts: 15
Joined: Mon Jul 17, 2017 3:06 pm

Re: Domesday86 Project

Postby noggin » Mon Jul 17, 2017 3:20 pm

simoni wrote:The Domesday86 project is continuing; currently we're working on a Laserdisc copying system to backup the AIV discs properly (an FPGA based system which samples directly from the RF of the laserdisc player - allowing a copy to be made of all the information on the disc without going through 30 year-old LDVP hardware).

/Simon


Fascinating. One thing to note about the Domesday Laserdiscs is that the Philips Laserdisc player and Genlock solution AIV used had some clever PAL decoding (decoding the PAL Laserdisc video to RGB, compositing with the BBC micro output in the RGB domain). I can't remember if the BBC Micro was genlocked to the Laserdisc (BBC Micros could genlock thanks to the BBC involvement!) or whether the Laserdisc player locked to the BBC.

Also the video on the discs themselves was pre-processed to improve the picture quality on still frames. I worked for Quantel R&D briefly as a student and the team there told me how a Quantel Harry had been usde to pre-process the content to reduce interlace twitter, but retain as much vertical resolution as possible. (Quantel Harry included some very nice image processing functionality) Without the pre-processing, still frames could have juddered or twittered quite nastily.

Because the maps and other stills were known to be static, it was possible to optimise the PAL encoding and decoding to exploit the knowledge that there was no motion between fields. (I think PALPlus did something similar - flagging film and video separately)

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Mon Jul 17, 2017 5:42 pm

Fascinating. One thing to note about the Domesday Laserdiscs is that the Philips Laserdisc player and Genlock solution AIV used had some clever PAL decoding (decoding the PAL Laserdisc video to RGB, compositing with the BBC micro output in the RGB domain). I can't remember if the BBC Micro was genlocked to the Laserdisc (BBC Micros could genlock thanks to the BBC involvement!) or whether the Laserdisc player locked to the BBC.


The genlocking and video mixing isn't an issue at this stage; the intent is purely to capture the RF analogue data from the laserdisc. Once that's done it will be post-processed back into it's component parts, namely:

* Video (frame by frame lossless since it's mainly still images)
* Audio (two channels - AIV uses both stereo and dual mono (two individual audio tracks, one per channel)
* Data (Acorn ADFS partition data encoded in the audio 'space' of the RF)
* VIB (this is 'other data' such as lead-in/out information (disc ID, etc) and frame numbering

The current design is using a TVP5150 IC from Texas Instruments to provide a 9-bit sampling of the PAL video coming from the RF output of the LaserDisc player's laser assembly. The intent is to replace this with a TVP5146 which has more accurate sampling. This only solves the video and VIB issue though; the other parts will require additional electronics (but they are easier to achieve). Basically the laser's RF signal will be band-passed into the individual RF streams with separate decoders for each part.

For the video, the TVP5150 provides an ITU-R BT.656 digital stream at a rate of 8 bits, 27 million times per second. That's a bandwidth of 216,000,000 bits a second, or 27 million bytes per second. This is what the FPGA is for; it's not possible to store data practically at that rate - even an SSD would struggle; so the FPGA is required to transcode the video into a 25 fps lossless digital representation which is then streamed to a microcontroller for storage.

So; I know what's required... all I need is some help getting the FPGA and microcontroller dev boards in place (without facing an angry wife :) )- hence the call-out for donations (since the project's inception I've already sunk over 1,500 pounds into it and given all the results away for free). The intent is to have the duplicator functioning before the ABUG meeting in November, so I can bring it along and provide digital backups to anyone that owns a real Domesday disc. The resulting data will also be archived and remixed (best bits of each disc) to form the eventual image used for the full system emulation.

Once that's done it's on to stage 3 (SCSI emulation was stage 1 - hence BeebSCSI, copying the laser discs is stage 2) which is to emulate the video functions of the VP415 and the genlock/video mix of the Master's output (my hope is to reuse the FPGA boards once more to save costs).

There are easier (and less effective ways) to make a copy of the disc (one could simply run the output from a VP415 through a video digitizer) - but the aim is preservation. I want the digital copy to be as good as the information on original disc. This 'extreme' method will preserve the information on the original laserdiscs so they can be recreated in the future. Due to 'Laser rot' most of the original discs are ticking time-bombs of failure. If there's no real backup made in the next 10 years, then there never will be.

dp11
Posts: 707
Joined: Sun Aug 12, 2012 8:47 pm

Re: Domesday86 Project

Postby dp11 » Mon Jul 17, 2017 6:40 pm

Given storing the data is sequential you would be hard pushed to find an ssd drive that can't keep up .

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Mon Jul 17, 2017 7:41 pm

Given storing the data is sequential you would be hard pushed to find an ssd drive that can't keep up .


Raw data write speed (of the storage medium) is just a part of the issue (and the TVP5146 will more than double the required bandwidth); and fast SATA drives plus the interfacing hardware don't come cheap. I'm trying to do it with a 50 quid Cyclone IV board so the project is practically reproducible (the only way to make the AIV laserdiscs more 'available' is to make them simple to copy). The nature of the RF signal from the LVDP laser isn't as simple as a standard PAL capture; there is an element of real-time processing required to analyse the data and respond to drop-outs, errors (including physical damage to the disc) and gain-control (laserdiscs aren't flat as they spin and the RF strength drops towards the outside of the discs) - otherwise I would simply mod a USB3 video capture device and be done with it.

dp11
Posts: 707
Joined: Sun Aug 12, 2012 8:47 pm

Re: Domesday86 Project

Postby dp11 » Mon Jul 17, 2017 8:08 pm

What data rate do you need to your storage? and how much storage do you need?

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Mon Jul 17, 2017 8:23 pm

It very much depends on the amount I can transcode on-the-fly. The more I can get the FPGA to do in real-time, the easier the storage will be. Initially I wanted to use a standard ADC to sample the RF, but the bandwidth requirements were impractical. The TVP5150 simplifies the issue somewhat, but it only handles the video.

I need to get the data stream out of the TVP5150 and into the FPGA's SDRAM first, then attempt to code some video transcoders. After that I'll have better figures on the storage requirements. To simplify the development I'm going to first set up a VGA output from the FPGA board (a DE0-Nano); this will allow me to visualise the data-stream and get the capture system correct; there will also be RS232 serial control to the LVDP to allow the board to control the playback.

You are welcome to take a look at the TVP5146 datasheet and let me know if you have any ideas to simplify the capture process given the constraints I've outlined. It should also be possible to do the maths on the initial ADC bandwidth from the datasheet information too. As a minimum I'd like the 10-bit capture (@ 30MSPS), but the 20-bit 4:2:2 YCbCr with separate syncs would be even better.


User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Mon Jul 17, 2017 8:52 pm

The only ARM based SoC I've found with an open-source tool chain and device library (LibOpenCM3) are the STM32 microcontrollers. Is the cypress SDK open-source?

dp11
Posts: 707
Joined: Sun Aug 12, 2012 8:47 pm

Re: Domesday86 Project

Postby dp11 » Mon Jul 17, 2017 9:01 pm

You can download the tool chain for free. I'm not sure on the exact licensing.

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Wed Jul 26, 2017 7:03 am

A big note of personal thanks to those of you that donated to the domesday86 project; it's really appreciated! You guys are awesome as always :)

User avatar
myelin
Posts: 202
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Domesday86 Project

Postby myelin » Thu Jul 27, 2017 11:51 pm

Cypress EZ-USB is a bit different from STM32. The STM32 series are general purpose Cortex-M0/M3/M4 (ARMv6/ARMv7) microcontrollers, some of which can act as USB2 full speed (12 Mbit, i.e. about 1.2 MB/s) devices, whereas EZ-USB is primarily a USB3 device, with an ARM9 core attached.

ARM9 is an older core (supporting either the ARMv4T or ARMv5TE instruction set; I'm not sure which in this case) but the EZ-USB FX3 is almost certainly going to be the easiest way to get anything over 1.2 MB/s up to a host machine, especially with the GPIF, which is designed to make for easy high speed interfacing to FPGAs etc. If you're just trying to make a one-off capture device to copy all the data from the laser discs onto a hard disk, this plus an FPGA is probably what you want.

Not sure on how open source everything is. In general I've found that ARM microcontroller dev includes *mostly* open source code, but there's often a bit of proprietary stuff mixed in.
SW/EE from New Zealand, now in San Francisco: http://myelin.nz/
Having fun making hardware projects for the Electron!
So far: 32k flash cart, USB cart interface, 3-cart expansion, Elk PiTubeDirect. Later: Dual ported ram cart.

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Fri Jul 28, 2017 4:13 am

In the end I decided to try with some STM32 devices since there is a HAL called libopencm3. Using that I can go end to end GPL although the IDE situation for ARM development seems to be extremely disjointed. I have got a STM32f103 (ARM Cortex M3) and STMf411 (M4), both run at around 72Mhz. In addition I have a F407 which can do 168MHz.

I'm thinking that, if I run the TVP5150 through an FPGA/CPLD front-end which does some of the initial processing, then I can get the data-stream down to a level that the STM can handle and store/USB stream.

Just need to learn the ARM Cortex architecture first - I've been mainly using 8-bit AVRs so far; I've tried ARM Cortex before, but the mess of licenses has always put me off - perhaps libopencm3 will light the way forward :)

The licensing of the software may seem 'unimportant' to some, but to me it's extremely important. Domesday86 is, above all things, a preservation project - everything produced for it should act as free and open 'documentation', even the source code. The benefit of the approach is demonstrated with sub-projects like BeebSCSI, which is a very clear and detailed description of how SCSI and ADFS works for both the laser disc player and winchester drives (as well as documenting how the VP415 works).

noggin
Posts: 15
Joined: Mon Jul 17, 2017 3:06 pm

Re: Domesday86 Project

Postby noggin » Thu Aug 03, 2017 4:32 pm

simoni wrote:The genlocking and video mixing isn't an issue at this stage; the intent is purely to capture the RF analogue data from the laserdisc. Once that's done it will be post-processed back into it's component parts, namely:

* Video (frame by frame lossless since it's mainly still images)
* Audio (two channels - AIV uses both stereo and dual mono (two individual audio tracks, one per channel)
* Data (Acorn ADFS partition data encoded in the audio 'space' of the RF)
* VIB (this is 'other data' such as lead-in/out information (disc ID, etc) and frame numbering


That makes sense - my point was that the PAL encoding and decoding used in the Domesday discs for still images was optimised - and if you aim to decode to Rec 601 SD 4:2:2 component video, you may be able to get improved results if your PAL decoder algorithm takes this into account, rather than using one optimised for conventional, moving, PAL video. (Richard Russell who lurks here may well be able to assist - he and his colleagues developed a very high quality PAL decoder and know a lot about this stuff)

The current design is using a TVP5150 IC from Texas Instruments to provide a 9-bit sampling of the PAL video coming from the RF output of the LaserDisc player's laser assembly. The intent is to replace this with a TVP5146 which has more accurate sampling. This only solves the video and VIB issue though; the other parts will require additional electronics (but they are easier to achieve). Basically the laser's RF signal will be band-passed into the individual RF streams with separate decoders for each part.

For the video, the TVP5150 provides an ITU-R BT.656 digital stream at a rate of 8 bits, 27 million times per second. That's a bandwidth of 216,000,000 bits a second, or 27 million bytes per second. This is what the FPGA is for; it's not possible to store data practically at that rate - even an SSD would struggle; so the FPGA is required to transcode the video into a 25 fps lossless digital representation which is then streamed to a microcontroller for storage.


Sorry - I don't agree with that.

Your numbers for SDI bit rate for the active picture are broadly right (my back of envelope - ignoring H and V blanking is nearer 207Mbs for 720x576x25x8x2 4:2:2 sampled 8 bit data)

However going with your figures 216Mb/s = 27MB/s - is well within the scope of a normal modern SATA hard drive. I've regularly captured uncompressed SD 601 video (i.e. the video that would be carried in an 8 bit Rec 656 parallel stream) to cheap domestic hard drives.

SSDs would eat that bitrate for breakfast, even relatively old HDDs would be fine. Uncompressed ITU 601/656 stuff will be no major issue for modern storage.

(When I first started doing this in 1990 we used early RAID arrays - but we were running triple bandwidth so we could replay 2 uncompressed streams and record 1 uncompressed stream back to the same array simultaneously)

(As a side note I recently recorded some 1080/50i 4:2:2 8 bit HD content uncompressed 4:2:2 8-bit rather than using ProRes HQ - to a cheap USB3 HDD. Didn't drop a single frame. I did a speedtest on my Mac using BlackMagic's disk speed test app afterwards and it confirmed it would record 1920x1080 4:2:2 8 bit uncompressed (i.e. ITU 709 HD) uncompressed (which is around 103MB/s or 824Mb/s).

What lossless codec were you considering transcoding to? There aren't that many lossless codecs in widespread use - most codecs used - even for high quality storage, use lossy compression. (That said if you use a high-quality lossy scheme - you can achieve something that is effectively visually lossless. MPEG4 SStP - as used by HD Cam SR is one example)

For SD content - if you are playing from local storage - I'd probably stick with uncompressed if quality is your main aim unless bandwidth is an issue.

(As a side note - I wonder if the original 1" tape used to master the discs was transferred in the BBC's archive project. That would be on both Digtal Betacam and uncompressed SD 601 on LTO tape - in component 4:2:2 format decoded via a BBC TRANSFORM PAL decoder - the highest quality PAL decoder available to my knowledge. That would be the highest quality source - though I don't know how any VBI data would have been handled. I assume that was present on the 1" but won't be on the 4:2:2 component copy as that will only contain active video)

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Thu Aug 03, 2017 5:01 pm

It's a little moot now since the TVP5150 idea didn't work out so I'm no longer trying to capture the video individually :)

My new idea is to sample the RF directly using a 40Msps 10-bit ADC and just capture the raw RF from the disc for later processing. This has been done by another project (for NTSC discs though) - so I can build on that for the non LV-ROM parts. Their problem seems to be they are using an off-the-shelf video capture card as the ADC (to capture the RF) - the 8-bit resolution and 27Msps limit of the card was hampering the results though.

I'm also toying with the idea of demodulating the streams on the fly using DSP - and I wouldn't loose any information by doing this.

But... it's early days; I'm prototyping to see what is practical. Without actually building some test systems and analysing the results it's hard to say what - in practice - is the right approach.

dp11
Posts: 707
Joined: Sun Aug 12, 2012 8:47 pm

Re: Domesday86 Project

Postby dp11 » Thu Aug 03, 2017 5:38 pm

I would capture raw data to SSD. Then do post processing in software. Far easier to correct a bugs that might creep into DSP code. Its also might be easier to have the raw data when it comes to merging multiple discs to fix dropouts in frames .

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Thu Aug 03, 2017 5:51 pm

I was thinking along the same lines; but it comes down to cost; buying yet another development board for SATA interfacing (along with yet another processor/SoC architecture to learn) is stretching things a little for a one-off project :) I'm going to build the ADC, do some testing and then move on to the storage part once I'm sure I have a workable front-end.

dp11
Posts: 707
Joined: Sun Aug 12, 2012 8:47 pm

Re: Domesday86 Project

Postby dp11 » Thu Aug 03, 2017 6:20 pm

I realise it is money and time but the FX3 with a little bit of code will stream to your computer SATA drive. I suspect that will be cheaper than any other SATA dev board. It will almost certainly interface directly to the ADC without any other logic .

SteveBagley
Posts: 129
Joined: Sun Mar 15, 2015 8:44 pm

Re: Domesday86 Project

Postby SteveBagley » Thu Aug 03, 2017 7:40 pm

noggin wrote:(As a side note - I wonder if the original 1" tape used to master the discs was transferred in the BBC's archive project. That would be on both Digtal Betacam and uncompressed SD 601 on LTO tape - in component 4:2:2 format decoded via a BBC TRANSFORM PAL decoder - the highest quality PAL decoder available to my knowledge. That would be the highest quality source - though I don't know how any VBI data would have been handled. I assume that was present on the 1" but won't be on the 4:2:2 component copy as that will only contain active video)


They have -- details here:

http://www.atsf.co.uk/dottext/domesday.html

And I suspect those Digibetas have since been archived as uncompressed 8-bit MXF files as part of the BBC's archiving project.

Steve

User avatar
simoni
Posts: 317
Joined: Wed May 25, 2016 6:18 pm

Re: Domesday86 Project

Postby simoni » Thu Aug 03, 2017 8:26 pm

I'm aware that there is an archived master for the national and possibly the community discs video/picture content, but that link (like most other 'Domesday' information) missing some important things.

The Domesday disc set is just 2 of the AIV LV-ROMs, there's also volcanoes, ecodisc, countryside, north pole and city. Isn't preserving those important too? Where are the masters for them? There are also 2 versions of National; national and national+... what's the difference? (there are also several interactive training disc sets based on the AIV system, so far I've seen SPC from Ford motor company and I have a lead on another produced by British Railways).

Even though masters for the domesday discs exist; there doesn't seem to be anyway to access them (if anyone knows how to get a copy, I'd really love to know!).

The pictures and video are only one part of a domesday disc. The aim of Domesday86 is to recreate the experience of the AIV system: video, pictures, sound, text, data, etc. All aspects of the disc contents need to be preserved to allow the 'system' to be recreated in full (and the system worked from a laserdisc, not a digital betamax tape).

The LV-ROMs are a central part of the Domesday system (all of them, not just National and Community).

If it was just a case of getting a 'good enough' emulation of the system together I could have stopped months ago... I already did that using BeebSCSI in January with full support for video and sound (surpassing the functionality of some previous attempts at simple emulation):

https://www.youtube.com/watch?v=PhliQ5scHzY

The Laserdiscs are falling apart from age... To me it's really important to capture their contents whilst it's still possible :)

User avatar
martinw
Posts: 1214
Joined: Sat Nov 13, 2010 10:31 am
Location: Aberdeenshire, Scotland

Re: Domesday86 Project

Postby martinw » Thu Aug 03, 2017 9:11 pm

I used this for an MSc project, got it at student rates ;)

IMG_5199.JPG

IMG_5200.JPG


Cool thing was it came with a reference Verilog project.

PAL to VGA, not sure it's of interest, but thought I'd mention it.

Martin

noggin
Posts: 15
Joined: Mon Jul 17, 2017 3:06 pm

Re: Domesday86 Project

Postby noggin » Fri Aug 04, 2017 12:46 am

SteveBagley wrote:
noggin wrote:(As a side note - I wonder if the original 1" tape used to master the discs was transferred in the BBC's archive project. That would be on both Digtal Betacam and uncompressed SD 601 on LTO tape - in component 4:2:2 format decoded via a BBC TRANSFORM PAL decoder - the highest quality PAL decoder available to my knowledge. That would be the highest quality source - though I don't know how any VBI data would have been handled. I assume that was present on the 1" but won't be on the 4:2:2 component copy as that will only contain active video)


They have -- details here:

http://www.atsf.co.uk/dottext/domesday.html

And I suspect those Digibetas have since been archived as uncompressed 8-bit MXF files as part of the BBC's archiving project.

Steve


Indeed. I suspect they are on INFAX or FABRIC then...


Return to “hardware”

Who is online

Users browsing this forum: No registered users and 6 guests