Docker & Acorn 101

discuss PC<>Acorn file transfer issues & the use of FDC, XFER, Omniflop/disk etc.
Post Reply
User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Docker & Acorn 101

Post by Elminster » Tue May 01, 2018 11:02 pm

As I have metioned in the B2 and B-EM threads that I have dockerised these for Linux x86 (rn Linux, Mac and indows) and Linux ARM (RPi). I thought I would mention what it was, how to use it etc.

So I started to write a short post, this got to be a very long so deiced to do it it in 2 and then 3 parts. Then I changed my mind and decided to do it in Markdown and stuck it in this post here

I wrote most of it last week but then got distracted by deciding to rebuild my Mac for the first time in 10 years (i.e. I have just migrated everything from 1 Mac to another, twice) so had loads of junk and artifacts. Eventually I shall condense all the posts and create something a bit better. Will add the links for teh other parts when I do them.

Edit:
Part 2 found here
Last edited by Elminster on Sun May 20, 2018 9:38 pm, edited 1 time in total.

User avatar
myelin
Posts: 420
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Docker & Acorn 101

Post by myelin » Wed May 02, 2018 5:17 pm

Very nice!

Something I've been playing with recently is using Google Cloud Container Builder to build projects. It's designed for building Docker images (we use it at work to build untrusted code), but all it does is execute Docker containers in sequence, so you can do all sorts of not-really-related-to-Docker things with it.

I have an experimental 3GB Docker image containing the Xilinx tools, which I can use to build a CPLD bitstream for my Acorn projects. A bit slow to run, but it means I can build directly from my Mac without keeping a Linux VM running the whole time. Something similar with beebasm etc might be useful.
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Wed May 02, 2018 8:22 pm

myelin wrote: I have an experimental 3GB Docker image containing the Xilinx tools, which I can use to build a CPLD bitstream for my Acorn projects. A bit slow to run, but it means I can build directly from my Mac without keeping a Linux VM running the whole time. Something similar with beebasm etc might be useful.
That was on my todo list. Are you going to upload to the image to a container repostiry and/or the buildfiles to github. I am assuming you end up with something compatable if it is Docker under the hood. Saves me having to do much then.

I cant remember if I said in the blog now but I have a jenkins container that auto build from github and then push back out to docker hub. Will have to add tha tin part 2 or 3.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Wed May 02, 2018 8:45 pm

I just rememebered I think you worked for that largest of container instantiator. So not surprising you woudl use them.

User avatar
myelin
Posts: 420
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Docker & Acorn 101

Post by myelin » Wed May 02, 2018 9:20 pm

Ah, Jenkins + Docker Hub, that's right. At work of course it's G-everything, so we use gcr.io :) Docker Hub being free for public projects is awesome; it makes it the obvious place to store your images if you don't have other reasons to keep them inside the Google Cloud.

I haven't looked at making a container with beebasm and friends; just think it would be a good idea. As for Xilinx, I'm not sure about the legality of me publishing a Xilinx Docker image, but I can give you my Dockerfile, which basically just downloads and untars the entire Xilinx install from my Linux VM. If you have it installed for Linux, tar up /opt/Xilinx and exclude as much as you can (I got rid of everything outside ISE_DS, I think). I pushed my tarballs up to GCS, but any web storage will work. You might be able to just drop them in the folder with your Dockerfile too, although that may result in them getting copied into the image and bloating it even further.

As you can see below, I didn't want to open my storage bucket up to the world but was too lazy to figure out the authentication; I just ran 'gsutil signurl https://storage.googleapis.com/myelin-n ... x/whatever' for each file, and pasted the signed URLs into my Dockerfile.

Code: Select all

FROM ubuntu

RUN apt-get update && apt-get install -y curl make

# Untar minimal-ish (~5G) ISE image into /opt/Xilinx
RUN curl -s 'https://storage.googleapis.com/myelin-nz-xilinx/opt-xilinx-minimal.tar.gz?GoogleAccessId=[access key redacted]' | tar -C / -zx

# Copy in the license file
RUN curl 'https://storage.googleapis.com/myelin-nz-xilinx/Xilinx.lic?GoogleAccessId=[access key redacted]' > /opt/Xilinx/Xilinx.lic

# Xilinx license location
ENV LM_LICENSE_FILE /opt/Xilinx/Xilinx.lic

# Set PATH/LD_LIBRARY_PATH for convenience (cut-down version of settings64.sh)
ENV PATH="/opt/Xilinx/14.7/ISE_DS/ISE/bin/lin64:/opt/Xilinx/14.7/ISE_DS/ISE/sysgen/util:/opt/Xilinx/14.7/ISE_DS/common/bin/lin64:${PATH}"
ENV LD_LIBRARY_PATH /opt/Xilinx/14.7/ISE_DS/ISE/lib/lin64:/opt/Xilinx/14.7/ISE_DS/common/lib/lin64

# Kick off user code
WORKDIR /workspace
CMD /workspace/build.sh
Here's the cloudbuild.yaml, if you're using Container Builder:

Code: Select all

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/myelin-nz/xilinx-ise', '.']

images: ['gcr.io/myelin-nz/xilinx-ise']
To kick off a build, I run 'gcloud container builds submit . --config=cloudbuild.yaml' from the folder with the Dockerfile and cloudbuild.yaml, and the build happens remotely and stores the resulting image on gcr.io.
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

Phlamethrower
Posts: 52
Joined: Fri Nov 24, 2017 1:35 pm
Contact:

Re: Docker & Acorn 101

Post by Phlamethrower » Wed May 02, 2018 10:07 pm

Alpine Linux is a good choice if you don't want gigabytes of bloat in your final images.

https://hub.docker.com/_/alpine/

At work we typically use a two-stage process - one docker container which does the build, and another which the compiled binary gets copied into (which generally requires the binary to be statically linked). Results in nice minimal images perfect for redistribution.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Wed May 02, 2018 10:23 pm

Phlamethrower wrote:Alpine Linux is a good choice if you don't want gigabytes of bloat in your final images.

https://hub.docker.com/_/alpine/

At work we typically use a two-stage process - one docker container which does the build, and another which the compiled binary gets copied into (which generally requires the binary to be statically linked). Results in nice minimal images perfect for redistribution.
Yes the emulator are all 2 stage multibuilds. The get build is in first stage and then copied to second build. Although they are Ubuntu, alone for another day as got bored working out dependencies. Will move the second stage to alpine at some Point.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Wed May 02, 2018 10:33 pm

myelin wrote: As for Xilinx, I'm not sure about the legality of me publishing a Xilinx Docker image, but I can give you my Dockerfile, which basically just downloads and untars the entire Xilinx install from my Linux VM.
True, it probably doesn’t have a repackage license, but a Docker file is fine, and should be pretty portable, not quite as good as a prebuilt image (maybe Xilinx will produce there own, actually they might, I never checked)
As you can see below, I didn't want to open my storage bucket up to the world but was too lazy to figure out the authentication; I just ran 'gsutil signurl https://storage.googleapis.com/myelin-n ... x/whatever' for each file, and pasted the signed URLs into my Dockerfile.
Will have a peruse. On my todo list honest. :)
To kick off a build, I run 'gcloud container builds submit . --config=cloudbuild.yaml' from the folder with the Dockerfile and cloudbuild.yaml, and the build happens remotely and stores the resulting image on gcr.io.
Assume gCloud is equiv of Docker-compose. I use those to start Jenkins container. Must remember to push that to my Jenkins-Docker fork.

User avatar
myelin
Posts: 420
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Docker & Acorn 101

Post by myelin » Wed May 02, 2018 11:38 pm

Elminster wrote:Assume gCloud is equiv of Docker-compose. I use those to start Jenkins container. Must remember to push that to my Jenkins-Docker fork.
Looks like docker compose is something to start up a bunch of services... gcloud is Google's tool that you use for pretty much all interaction with services like Compute Engine, App Engine, Cloud Storage and all that stuff.

That last part of my post is only relevant if you want to use Container Builder to build your image. 'docker build .' will work just fine too :)
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Thu May 03, 2018 9:12 am

myelin wrote:
Elminster wrote:Assume gCloud is equiv of Docker-compose. I use those to start Jenkins container. Must remember to push that to my Jenkins-Docker fork.
Looks like docker compose is something to start up a bunch of services... gcloud is Google's tool that you use for pretty much all interaction with services like Compute Engine, App Engine, Cloud Storage and all that stuff.

That last part of my post is only relevant if you want to use Container Builder to build your image. 'docker build .' will work just fine too :)
You can use docker-compose to start stuff but you can also use it to build stuff. It is just basically a way of cutting out trying to rememebr a millon command line options when you are in Development. In prod you use stacks on swarms, which is basically the same yaml file but it ignores all the build stuff. And any secrets in the yaml file only work with swarms and not docker-compose.

So I guess the functions of gcloud is a superset of those in docker-compose. I will put 'learn googles container control stuff' on my todo list. Along with design that time machine from a previous thread. :)

I uploaded the Jenkins-Docker docker-compose file to github so you can have a look and compare it to all that mystical G stuff. It is here.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Thu May 03, 2018 1:09 pm

Phlamethrower wrote:Alpine Linux is a good choice if you don't want gigabytes of bloat in your final images.

https://hub.docker.com/_/alpine/

At work we typically use a two-stage process - one docker container which does the build, and another which the compiled binary gets copied into (which generally requires the binary to be statically linked). Results in nice minimal images perfect for redistribution.
Forgot to say issue is more Xilinx, alpine docker image is about 100MB smaller than ubuntu image. But the Xilinx full application install uncompressed is 6GB!

Code: Select all

ubuntu                         17.10               e4422b8da209        5 days ago          99.2MB
ubuntu                         latest              f975c5035748        8 weeks ago         112MB
alpine                         latest              3fd9065eaf02        3 months ago        4.15MB

User avatar
myelin
Posts: 420
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Docker & Acorn 101

Post by myelin » Thu May 03, 2018 4:33 pm

A few months back I read a blog post where someone ran a Xilinx build on a filesystem that tracked all accesses, then redid it with just the files that it touched, and it was more like ~150MB. I can't find it any more, unfortunately!
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
Most popular: fast serial port, FX2+PiTubeDirect Tube/Cartridge adapter, USB cart interface.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Thu May 03, 2018 5:05 pm

myelin wrote:A few months back I read a blog post where someone ran a Xilinx build on a filesystem that tracked all accesses, then redid it with just the files that it touched, and it was more like ~150MB. I can't find it any more, unfortunately!
I cannot image it really needs all 6GB, so wouldnt suprise me if it was 5GB of bloat. That would be a very useful note if you could find it. Would make a container much more useful when it doesnt take 10 mins to download it.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Mon May 07, 2018 10:03 pm

Part 2 drafted just need to tidy it.

In the meantime Elkulator and Atomulator are now in the Docker hub.

User avatar
Elminster
Posts: 2408
Joined: Wed Jun 20, 2012 8:09 am
Location: Essex, UK
Contact:

Re: Docker & Acorn 101

Post by Elminster » Sun May 20, 2018 8:53 pm

Took very slightly longer to write part 2 than I had planned. Got a bit long.

Can now be found here

Post Reply