Arduino Filestore

for bbc micro/electron hardware, peripherals & programming issues (NOT emulators!)
gazzaD
Posts: 20
Joined: Sun Jun 18, 2017 11:37 am

Arduino Filestore

Postby gazzaD » Mon Oct 23, 2017 11:06 pm

I mentioned a while back that I was working on getting an Arduino talking to an Econet card, in the last couple of days I've just about managed to get a reliable 4 way handshake working, which means I've been able to start implementing the file server operations:

Image

*I AM, examine folder, read object information, read user environment, read disc information and log off is all it does at the moment, which seem to be the minimum required to get the home folder to open on an A3000. Examine folder also needs a bit of work to populate the file attributes, just as soon as I decide on a sensible way to store them on the FAT32 formatted SD card.

The hardware involved is very simple, just an Arduino Due with some level shifters to communicate with an Econet card. It's currently sitting on breadboard at the moment:

Image

I've got a lot of work to do to implement the remainder of the FS calls and get the Arduino working in a stable manor, but I can see the potential for doing lots of cool things using the Ethernet port. One idea I have is virtual HTTP and FTP disks on the file server, that allow you to pull files off of the Internet with just a NetFS call. Or maybe routing between the two networks without all the machines needing AUN?

Right now this is very much a work in progress, but I'm sure it will be interesting to the community :)
Last edited by gazzaD on Thu Oct 26, 2017 12:25 am, edited 1 time in total.
Gareth

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Mon Oct 23, 2017 11:32 pm

gazzaD wrote:I mentioned a while back that I was working on getting an Arduino talking to an Econet card, in the last couple of days I've just about managed to get a reliable 4 way handshake working, which means I've been able to start implementing the file server operations:


Image

gazzaD wrote:*I AM, examine folder, read object information, read user environment and log off is all it does at the moment, which seem to be the minimum required to get the home folder to open on an A3000. Examine folder also needs a bit of work to populate the file attributes, just as soon as I decide on a sensible way to store them on the FAT32 formatted SD card.

The usual method is:
* Unix-y or DOS-y paths on the server are translated by flipping all '.'s with '/'s, eg "docs/6502ops.txt" is served as "docs.6502ops/txt" (and a few other minor flips, see wiki)
* a file on the host with ,xyz suffix translates to a served file without the suffix with a RISC OS filetype of xyz, ie load=&FFFxyzss, exec=ssssssss with ssssssssss being the centisecond timestamp translated from the host system's timestamp.
* a file with a .xyz extension is translated via a MimeMap lookup on the server to a RISC OS filetype
* a .inf file on the server holding the load and execution addresses
* the modification date, modification time, creation date, creation time(*) come from the host system's metadata for the served file.
ie: the data returned by Examine, ReadDate&Time, ReadInfo and WriteInfo. By coincidence, a few days ago I uploaded an updated version of NetFS.txt summarising the NetFS operations.
(Edit: see the DEFFNfile_info() and DEFFNf_info() functions in TubeHost)

gazzaD wrote:One idea I have is virtual HTTP and FTP disks on the file server, that allow you to pull files off of the Internet with just a NetFS call. Or maybe routing between the two networks without all the machines needing AUN?

The standard method is that network 128+.xxx map to a pre-specified IP address, but that's for working at the network level. You need something so that a machine comminated to over the Econet using NetFS calls to the server translate to ftp calls over whatever media link the server's ftp link uses.

So maybe something along the lines of
*I AM 0.254 gazza
*NETMOUNT :nvg ftp://bbc.nvg.org/
*DIR :nvg.docs
*TYPE 6502OpList/txt

etc.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
IanS
Posts: 477
Joined: Mon Aug 31, 2009 6:02 pm

Re: Arduino Filestore

Postby IanS » Mon Oct 23, 2017 11:41 pm

gazzaD wrote:I mentioned a while back that I was working on getting an Arduino talking to an Econet card, in the last couple of days I've just about managed to get a reliable 4 way handshake working, which means I've been able to start implementing the file server operations:
...

Right now this is very much a work in progress, but I'm sure it will be interesting to the community :)

--
Gareth


Awesome, can't wait to see how this develops.

User avatar
richardtoohey
Posts: 3378
Joined: Thu Dec 29, 2011 5:13 am
Location: Tauranga, New Zealand

Re: Arduino Filestore

Postby richardtoohey » Tue Oct 24, 2017 5:18 am

=D> and love the photos ... :D

User avatar
danielj
Posts: 5364
Joined: Thu Oct 02, 2008 4:51 pm
Location: Manchester

Re: Arduino Filestore

Postby danielj » Tue Oct 24, 2017 6:52 am

Great work on this! :)

d.

gazzaD
Posts: 20
Joined: Sun Jun 18, 2017 11:37 am

Re: Arduino Filestore

Postby gazzaD » Tue Oct 24, 2017 10:05 pm

jgharston wrote:The usual method is:
* Unix-y or DOS-y paths on the server are translated by flipping all '.'s with '/'s, eg "docs/6502ops.txt" is served as "docs.6502ops/txt" (and a few other minor flips, see wiki)
* a file on the host with ,xyz suffix translates to a served file without the suffix with a RISC OS filetype of xyz, ie load=&FFFxyzss, exec=ssssssss with ssssssssss being the centisecond timestamp translated from the host system's timestamp.
* a file with a .xyz extension is translated via a MimeMap lookup on the server to a RISC OS filetype
* a .inf file on the server holding the load and execution addresses
* the modification date, modification time, creation date, creation time(*) come from the host system's metadata for the served file.

I had been been reading up on how Fileswitch works before posting , and after (mumble) years of using RISC OS machines I'd never noticed you can either have a load and exec address, or a filetype and date stored with a file. I've got an NFS mount on my current network, and used them back in 1996 - so am aware of the ,filetype notation. I'm not planning on building a system where the SD cards are interchangeable with other devices, so there will be a hybrid thing happening.

Having a map file for foreign files coming in over the Ethernet is a good idea, whether using the content-type header or the .xxx extension.
gazzaD wrote:One idea I have is virtual HTTP and FTP disks on the file server, that allow you to pull files off of the Internet with just a NetFS call. Or maybe routing between the two networks without all the machines needing AUN?

The standard method is that network 128+.xxx map to a pre-specified IP address, but that's for working at the network level. You need something so that a machine comminated to over the Econet using NetFS calls to the server translate to ftp calls over whatever media link the server's ftp link uses.

So maybe something along the lines of
*I AM 0.254 gazza
*NETMOUNT :nvg ftp://bbc.nvg.org
*DIR :nvg.docs
*TYPE 6502OpList/txt

etc.


What I had in mind was something like:
copy net#ftp:$.bbc.nvg.org/pub/bbc/doc/6502.txt adfs:$
But it depends on how NetFS deals with file names as they pass through, which is something I've not looked at yet.
Gareth

crj
Posts: 328
Joined: Thu May 02, 2013 4:58 pm

Re: Arduino Filestore

Postby crj » Tue Oct 24, 2017 10:26 pm

gazzaD wrote:What I had in mind was something like:
copy net#ftp:$.bbc.nvg.org/pub/bbc/doc/6502.txt adfs:$

I'd strongly recommend net#ftp::bbc/nvg/org.pub.bbc.doc.6502/txt as the filename. You want to be using dots as directory specifiers and slashes in place of dots. And it feels "right" to have the hostname being the disc name.

User avatar
paulv
Posts: 3606
Joined: Tue Jan 25, 2011 6:37 pm
Location: Leicestershire
Contact:

Re: Arduino Filestore

Postby paulv » Tue Oct 24, 2017 10:29 pm

This looks very neat :D =D>

Paul

crj
Posts: 328
Joined: Thu May 02, 2013 4:58 pm

Re: Arduino Filestore

Postby crj » Tue Oct 24, 2017 10:35 pm

jgharston wrote:The usual method is:[...]
* a .inf file on the server holding the load and execution addresses

While it obviously makes complete sense to support the existing conventions for backwards compatability, two ways to finesse this have occurred to me. They're mutually incompatible, so one couldn't do both.

The first would be to use an extended attribute instead of a .INF file. That would be quicker and tidier, and support is pretty widespread nowadays.

The second would be to put Acorn-specific data for every object in a directory in a specially-named sqlite database. That would be robust, and would also solve the headache of finding files in a case-insensitive way on a case-sensitive underlying filesystem.

Any thoughts?

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Wed Oct 25, 2017 12:10 am

gazzaD wrote:I had been been reading up on how Fileswitch works before posting, and after (mumble) years of using RISC OS machines I'd never noticed you can either have a load and exec address, or a filetype and date stored with a file.

That's how RISC OS stores datestamps - in the load/exec addresses, as the ADFS on-disk file system doesn't have seperate timestamp fields. If the load address is &FFFxxxxx then it is impossible to use as a load address, and indicates that the load/exec addresses hold a filetype and datestamp.

There is a complication that the NetFS protocols implement both load/exec addresses (what you get at XY!2, XY!6 from OSFILE 5) and datestamps (what you get at XY!15 from OSFILE 5), as that's what BBC systems expect, eg:

>*INFO ROMS
ROMS WR/r FFFF0900 FFFF0918 0001F8 28/05/1992
>


The standard method when serving files from a non-load/exec filing system is:
* Fill in the NetFS date/time fields from host system's date/time fields
* Translate host system's date/time fields to centisecond time and use to fill in NetFS load/exec fields
* Look for any extension/suffix* on the host system object to a filetype and use it to fill in the NetFS load field and remove the suffix from the name
* Look for a sidecar object* supplying load/exec address, and if it exists, use it to fill in NetFS load/exec fields
* Supply the metadata to the client

(*)nomenclature: an extension is dot-something-something, so ",xyz" isn't an extension as it doesn't start with a dot.
(*)newly-discovered useful term: sidecar object/sidecar file: an additional file or other object supplying additional metadata that cannot be stored in the filesystem, such as our .inf files, Windows .EA files.

An extension I've found to be really useful is to examine the object, and if it has an Acorn ROM header, use that to supply the load/exec addresses, see the wiki.

crj wrote:
gazzaD wrote:What I had in mind was something like:
copy net#ftp:$.bbc.nvg.org/pub/bbc/doc/6502.txt adfs:$
I'd strongly recommend net#ftp::bbc/nvg/org.pub.bbc.doc.6502/txt as the filename. You want to be using dots as directory specifiers and slashes in place of dots. And it feels "right" to have the hostname being the disc name.

More than "strongly recommend". It's *required* that directory seperators are dots.

When serving objects you are universally* serving leafnames, so the served name translation is easy. From a non-dot-directory file system, convert:
* case '.': b='/'; break;
* case '#': b='?'; break;
* case '$': b='<'; break;
* case '^': b='>'; break;
* case '/': b='\\'; break;
* case ' ': b='\xA0'; break;
* strip any ",xyz" suffix

So, for example, a host file called "My House.jpg" is served as "My"+CHR$160+"House/jpg". A host file "readme,fff" is served as "readme".

(*)From a quick skim through NetFSOps.txt, all served names are "the names in the specified directory" or similar.

When receiving pathnames you have to translate full pathnames as you may receive, eg *INFO :User.$.JGH.Docs.index/htm. For a non-dot filing system you need to convert:
* case '\xA0': b=' '; break;
* case '.': b='/'; break; // directory seperator
* case '?': b='#'; break;
* case '#': b='?'; break; // 1-char wildcard
* case '/': b='.'; break; // extension seperator
* case '@': b='.'; break; // current directory
* case '<': b='$'; break;
* case '>': b='^'; break;
* case '^': b=".."; break; // parent directory
* case '&': b=home directory;
* case '%': b=library directory; // only as a single character, eg "%" or "%.file", not "%filename" or "file/%%%"
* case '%': eg "%filename" can't remember off the top of my head

Root references are a bit fiddly.
* :drive -> translate to (mount point for drive)
* :drive.$ -> translate to (mount point for drive) and skip past the $
* '$' -> translate to the root of the current mount point
* Be careful with :drive vs :drive.$ which are the same, vs :drive.$.object which should give eg drive:/object not drive://object

See 'host.ccp' in the BeebEm source or the BBCUnZip source.

I need to check in detail, but you have to be able to expect pathnames that are space terminated with extranous stuff after the space that you need to ignore. For instance, I can't remember but some systems may process "*LOAD fred 900" and pass the filename via NetFS as "fred 900". You certainly need to look for spaces when passed commands such as "*ACCESS FRED thing" and "*RENAME OLD NEW".

gazzaD wrote:What I had in mind was something like:
(edit) net#ftp::bbc/nvg/org.pub.bbc.doc.6502/txt
But it depends on how NetFS deals with file names as they pass through, which is something I've not looked at yet.

RISC OS NetFS uses the #name part as the name of the file server to connect to. RISC OS NetFS treats all served disks as semi-seperate file servers. The result is that the NetFS #name part is functionally the disk name, unless you then specify a disk name in the path.

So, "net#ftp::bbc/nvg/org.pub.bbc.doc.6502/txt" would result in NetFS talking to fileserver "ftp" and asking for the file ":bbc/nvg/org.pub.bbc.doc.6502/txt". The server doesn't see the client asking for the "ftp" part, that is just used by NetFS to select the file server to talk to, it is essentially an alias for, eg, 0.254.

Also, I think: a typo by cjr:
* ":bbc/nvg/org.pub.bbc.doc.6502/txt" should be
* ":bbc/nvg/org/pub.bbc.doc.6502/txt"
as pub.bbc.doc (/pub/bbc/doc) is the directory on the server, bbc/nvg/org (bbc.nvg.org) is the "disk name".

cjr wrote:The first would be to use an extended attribute instead of a .INF file. That would be quicker and tidier, and support is pretty widespread nowadays.
The second would be to put Acorn-specific data for every object in a directory in a specially-named sqlite database. That would be robust, and would also solve the headache of finding files in a case-insensitive way on a case-sensitive underlying filesystem.

Using .inf files makes it transparent and file system independent, and copying a directory structure on the host system automatically copies the metadata. Certainly support other schemes, but I think supporting .inf files must be a core feature.

If you want to put the metadata for all the objects in a directory in a single structure, what I do with SoftMDFS is have a file called "!!MetaData" or CHR$160+CHR$160+"MetaData" which contains .inf type information for all the objects in the directory, eg:
!BOOT FFFFFF50 87352628
Hello FFFFFB4F 028463FE
screen FFFE3000 FFFE3000

In all cases, when serving an "enumerate objects in directory" request, the metadata files' names should not be served.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

crj
Posts: 328
Joined: Thu May 02, 2013 4:58 pm

Re: Arduino Filestore

Postby crj » Wed Oct 25, 2017 1:51 am

jgharston wrote:Also, I think: a typo by cjr:
* ":bbc/nvg/org.pub.bbc.doc.6502/txt" should be
* ":bbc/nvg/org/pub.bbc.doc.6502/txt"
as pub.bbc.doc (/pub/bbc/doc) is the directory on the server, bbc/nvg/org (bbc.nvg.org) is the "disk name".

Firstly, ironically, I'm "crj" not "cjr". :-p

Secondly, I think I was correct. I intended that bbc/nvg/org be the disk name, separated by a dot from the path within the "disc". You've got bbc/nvg/org/pub as the disk name, no?

(I was, incidentally, assuming that "ftp" was the name of the fileserver dedicated to providing ftp-to-NetFS service.)

User avatar
sydney
Posts: 1987
Joined: Wed May 18, 2005 9:09 am
Location: Newcastle upon Tyne

Re: Arduino Filestore

Postby sydney » Wed Oct 25, 2017 11:05 am

Great work.
I've always been of the opinion that econet is interesting but not that useful to me. I was thinking of trying to connect a ch375b usb interface to my beeb via the 68b54 socket but your arduino filestore makes econet useful so I may hold off for a while.

User avatar
myelin
Posts: 204
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Arduino Filestore

Postby myelin » Wed Oct 25, 2017 6:09 pm

This is really really cool. I know it's early, but are you planning on open sourcing this? I'd love to have a play with it. I have some ideas about alternative Econet interfaces (homebrew versions of the Master Econet module you're using) and this would be a great platform to try them out.
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
So far: fast serial port, 32k flash cart, USB cart interface, 3-cart expansion, Elk PiTubeDirect.

gazzaD
Posts: 20
Joined: Sun Jun 18, 2017 11:37 am

Re: Arduino Filestore

Postby gazzaD » Thu Oct 26, 2017 12:02 am

jgharston wrote:More than "strongly recommend". It's *required* that directory seperators are dots.


Noted! And I'm having to do the translation for local file access anyway. But the filename net#http://www.bbc.co.uk/ does make it through to the server just about unscathed, as there isn't a $ sign in it you actually get a request relative to the CSD.

jgharston wrote:
gazzaD wrote:What I had in mind was something like:
(edit) net#ftp::bbc/nvg/org.pub.bbc.doc.6502/txt
But it depends on how NetFS deals with file names as they pass through, which is something I've not looked at yet.

RISC OS NetFS uses the #name part as the name of the file server to connect to. RISC OS NetFS treats all served disks as semi-seperate file servers. The result is that the NetFS #name part is functionally the disk name, unless you then specify a disk name in the path.

The plan of sorts was to separate the protocols by drive, and so the HTTP 'drive' would fetch via HTTP etc. But I've already seen the drive names being stripped off on my 'single drive' server, so will need to look at how this works with the additional drives added. This is a long way down the list though, I need to read and write to the local SD card first.

jgharston wrote:
cjr wrote:The first would be to use an extended attribute instead of a .INF file. That would be quicker and tidier, and support is pretty widespread nowadays.
The second would be to put Acorn-specific data for every object in a directory in a specially-named sqlite database. That would be robust, and would also solve the headache of finding files in a case-insensitive way on a case-sensitive underlying filesystem.

Using .inf files makes it transparent and file system independent, and copying a directory structure on the host system automatically copies the metadata. Certainly support other schemes, but I think supporting .inf files must be a core feature.

I remember having separate .info metadata files on the Amiga, was not really a huge fan. For locally stored files I'm going to try extending the filenames beyond the existing case of ,XXX where XXX is the filetype to additionally have a ,YYYYYYYYZZZZZZZZ extension where a real load and exec address are required. FAT32 supports timestamps, so they can be used natively. Access attributes could become entertaining, as there are only 3 available on FAT32 and I need 5. In a network environment I'd suggest that the public access bits are probably the most important ones (which i can map to 'system' or 'hidden' in FAT speak) leaving the FAT locked attribute to map to locked on the Acorn side. As long as the translations are consistent both ways, what gets laid down on the disk isn't really important. To get around the case problem I'll continue converting everything internally to uppercase, although I may need to review what the performance penalty is on this once directories start getting large.

Thanks for the input people.
Last edited by gazzaD on Thu Oct 26, 2017 12:25 am, edited 1 time in total.
Gareth

gazzaD
Posts: 20
Joined: Sun Jun 18, 2017 11:37 am

Re: Arduino Filestore

Postby gazzaD » Thu Oct 26, 2017 12:22 am

myelin wrote:This is really really cool. I know it's early, but are you planning on open sourcing this? I'd love to have a play with it. I have some ideas about alternative Econet interfaces (homebrew versions of the Master Econet module you're using) and this would be a great platform to try them out.

Absolutely, although the code does need some tidying up first.

It's been a very interesting exercise for me, as I've never needed to develop something as 'real time' as this before. Keeping the code readable, while also keeping it working is proving to be a challenge in itself. Some operations on the ADLC have to happen in a period of less than a microsecond, and so even with an 86 Mhz CPU available, when you are programming from a high level language (Arduino uses C) it's fairly easy to run out of clock cycles when trying to do things. It took several attempts at refactoring the transmission loop before I had something that had complete error checking, and executed fast enough, to not cause a buffer under run by the 4th byte.
Gareth

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Thu Oct 26, 2017 1:54 am

gazzaD wrote:I remember having separate .info metadata files on the Amiga, was not really a huge fan. For locally stored files I'm going to try extending the filenames beyond the existing case of ,XXX where XXX is the filetype to additionally have a ,YYYYYYYYZZZZZZZZ extension where a real load and exec address are required.

A single file per directory with all the metadata for all the objects in that directory in .inf format is the best way to go, and is a pre-existing method: eg:
$ ls
!!MetaData
!Boot
Hello
Screen
$ cat !!MetaData
!Boot FFFFFF50 892BFE12
Hello FFFFFB51 428AB40
Screen FFFF3000 FFFF3000
$


Also, having .inf files means that you can raw copy stuff from other sources that have .inf files, you can extract files from disk images with tools that create .inf files for the extracted files. Inventing a different shaped wheel is all ok as long as you accept the pre-existing wheels as well.

To minimise the creation of too many additional .inf files, what you should do when writing a file's metadata is read the file's metadata through the translation processes described upthread, and only if the new metadata is different from the existing do you metadata create a .inf file.

So, for example, if the file is called house.jpg and the server's mimemap translates ".jpg" to &C85 (JPEG), then if the client does a "write load/exec addresses" call to set the load address to &FFFC85xx you don't create a .inf file as the file already translates to &FFFC85xx.

Similarly, if the file on the host system has a ,xyz suffix, when the client does a "write load/exec addresses" call, if it sets the load addres to &FFFxyz## then just rename the ,xyz suffix to the new ,xyz suffix.

Only create a .inf file (or create an entry in a single metadata file) if the "write load/exec" call falls outside these actions.

See TubeHost and host.ccp in BeebEm.

I'm away from my network at the moment, but when I get back I'll do some sample netfs transactions to check what gets chucked about. One thing you may have discovered is that RISC OS 3 NetFS translates LOAD and SAVE to OPEN/OSGBPB/CLOSE, so you always need one handle spare.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Thu Oct 26, 2017 1:56 am

Win/DOS/Unix file access attributes are translated to RISC OS attributes as per this WIki article.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Thu Oct 26, 2017 2:48 am

When you do a GetFileAttributesEx call to examine a FAT object (or non-Windows equivalent) and get:

DWORD dwFileAttributes;
FILETIME ftCreationTime;
FILETIME ftLastAccessTime;
FILETIME ftLastWriteTime;
DWORD nFileSizeHigh;
DWORD nFileSizeLow;

ftLastWriteTime is what should be translated to the 5-byte centisecond time for objects with load=&FFFxxxxxx, and should be translated and returned when the 2-byte "last modification date" and 3-byte "last modification date" is asked for.

ftCreationTime is what should be translated to the 2-byte "file creation date" and the 3-byte "file creation time".

You should ignore ftAccessTime, there are no NetFS calls that would read it.

If nFileSizeHigh is non-zero, or nFileSizeLow>&00FFFFFF then if the returned file size should be &00FFFFFF. You should refuse to serve a file that is longer than &00FFFFFF and should refuse move PTR within an open file past &00FFFFFF.

dwFileAttributes should be translated as follows:

0x10 DIRECTORY -> clear: object type=1, set: object type=2

Always return PublicRead and OwnerRead
0x01 READONLY -> OwnerLocked

The standard is to always return PublicWrite and OwnerWrite, but I'd be inclinded to return (maybe as a configuration option):
0x01 READONLY -> NOT (PublicWrite) and NOT (OwnerWrite)

Now, that scheme means that the public access is always a duplicate of the owner access. Something I've thought about in the past is to use the HIDDEN access to seperate owner access and public access. So, if HIDDEN is set, there is no public access, if HIDDEN is clear the public access is the same as the owner access, giving:

-- -> WR/wr
-R -> LR/r
H- -> WR/
HR -> LWR/

The NetFS access byte is %MPDLwrWR containg the object type and access, so the FAT access byte translates to the NetFS access byte as:

FAT AND &13 -> NetFS
----- 0x00 -> 0x0F WR/wr
----R 0x01 -> 0x15 LR/r
---H- 0x02 -> 0x03 WR/
---HR 0x03 -> 0x13 LR/
D---- 0x10 -> 0x2F DWR/wr
D---R 0x11 -> 0x35 DLr/r
D--H- 0x12 -> 0x23 DWR/
D--HR 0x13 -> 0x33 DLR/
(I worked that out in my head, there could be errors!)

You could extend that to use the SYSTEM attribute as well, but you must ensure that the client is never allowed to set LSHR (&xF) as that indicates a FAT LongFileName entry.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

gazzaD
Posts: 20
Joined: Sun Jun 18, 2017 11:37 am

Re: Arduino Filestore

Postby gazzaD » Thu Oct 26, 2017 6:40 pm

jgharston wrote:Win/DOS/Unix file access attributes are translated to RISC OS attributes as per this WIki article.

They seem to be based on ignoring the public access bits, which on a network server I'd consider the most important. I know we are dealing with a system that send the username and password in the clear across the wire as you log on, but I'd like to put something together that isn't any worse than the existing servers for security :)

Thanks for the detail on the file time mappings, it's saved me looking it up. I don't think the attribute mapping i was thinking of using was going to be too different to your suggestion. I'm also going to dig into the Arduino SD card libraries a bit, as alongside the hidden and system attributes there is also an archived bit - but it's considered an MS-DOS legacy so not exposed.

jgharston wrote:Inventing a different shaped wheel is all ok as long as you accept the pre-existing wheels as well.

Ah, I think I see why we are talking across each other here. At the moment I'm only interested in providing file storage, and gateway services, to an Econet network. Being able to mount SD cards from other platforms isn't really in scope for now, it would only encourage an attempt at hot swapping which doesn't always end well on an Arduino.

I'm expecting (famous last words) all files in and out to pass through one of the network interfaces, so that gives the opportunity to fettle the attributes into a format the receiver is expecting on the way through, regardless of what is on the disk. As long as it all reaches the target machine correctly, then it can be converted to whatever there.
Gareth

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Thu Oct 26, 2017 11:43 pm

gazzaD wrote:
jgharston wrote:Win/DOS/Unix file access attributes are translated to RISC OS attributes as per this WIki article.

They seem to be based on ignoring the public access bits, which on a network server I'd consider the most important.

On a FAT filesystem there are only one set of access bits, so you either have to duplicate owner access bits to the public access bits, or come up with a method for synthesising them such as using the H bit to indicate "don't let public see owner's access".

The NetFS access byte is MPDLwrWR. On a FAT filesystem you have --ADLSHR, you must ensure LSHR are never set together as that flags a LongFileNames entry, so from the remaining 15 possible bit states you have to come up with a method to translate those to the 32 possible NetFS values. If you go safe and avoid fiddling with L entirely that results in 8 possible FAT bit values to translates to 32 possible NetFS bit values.

You could use the Archive bit instead of (or as well as the Label bit), giving --A--SHR to translates to ---LwrWR while ensuring that any object with LSHR=%1111 is never changed, and any object with LSHR<>%1111 is never changed to %1111. That gives you 16 FAT access bits to fit 32 NetFS access bits into. However, I'm wary of using the Archive bit as various file operations automatically change it.

The easiest NetFS bit to ignore is the OwnerRead and assume/enforce that OwnerRead is always set. That is consistant with mapping methods in other systems, eg DOSFS, WinFS, LanMan, etc. That then reduces it to 16 NetFS access states you need to encode:
R/ WR/ R/r WR/r R/w WR/w R/wr WR/wr
LR/ LWR/ LR/r LWR/r LR/w LWR/w LR/wr LWR/wr

The next easiest is to make the Locked bit always mirror the Write bit, so that if you can't delete the object you also can't write to the object, and if you can't write to the object you also can't delete the object, giving you 8 NetFS access states to encode:
LR/ WR/ LR/r WR/r LR/w WR/w LR/wr WR/wr

That fits into the 8 FAT access bit states you have with S-H-R. That gives a better set of NetFS states as the "H bit to indicate don't let public see owner's access" doesn't have WR/r available, which is the second most common access on my server. The access states I have on files on my server (ignoring a few Ps) are LR/ (eg system files), LR/r (eg library files), WR/, WR/r, WR/wr, WR/w (mailbox files).

Thinking of the symantics of the FAT access bits, I think I'd do:
1 -----------> R
R -+--NOT-> W
....\--------> L
H ---NOT---> r
S ---NOT---> w
ie, if the object is ReadOnly you can't Write to it and it is Locked, 'cos it's read-only
if the object is not Hidden, public can Read it, 'cos it's not hidden
if the object is not System, public can Write to it, and I've run out of analogies for this bit
note: only the owner of the directory an object is in can delete or rename an object, regardless of who owns the object itself.

As you're using a FAT file system you'll have to implement any ownership system in the server itself, which is reasonable as even with a server running on a file system that implements ownership in the file system a server has to run as root and deal with at least some aspects of ownership, as if the server doesn't run as root it can't serve objects the server process doesn't own.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
oss003
Posts: 2550
Joined: Tue Jul 14, 2009 11:57 am
Location: Netherlands
Contact:

Re: Arduino Filestore

Postby oss003 » Fri Oct 27, 2017 9:54 am

Hi Gareth,

great job to get the Arduino filestore working....... =D>

I'm also interested in the code because at the moment I disassembled the Atom Econet rom to understand how Econet communication is working.
My goal is to get Econet package transfer working in an Atom emulator and write a Javascript fileserver for exchanging files over a network.
It will be a very basic fileserver to support Atom Econet but maybe later it can be expanded.
So I would be very happy if you want to share any information about Econet package formats and Econet communication.

Greetings
Kees

User avatar
jgharston
Posts: 2764
Joined: Thu Sep 24, 2009 11:22 am
Location: Whitby/Sheffield

Re: Arduino Filestore

Postby jgharston » Sun Oct 29, 2017 6:05 pm

I've recovered from getting home late from work and being completely confuddled by the changed clocks, and done some testing.

NetFS 5.50 onwards (RISC OS 3 onwards) does extra low-level fiddling to provide an easier user interface, so can complicate things at the server end.

When the client logs on, NetFS reads the disk name and the directory name of the URD and uses them when passing certain paths later on.

NetFS asks the file server for a list of the disks on that server, and uses that to select net#name points. This list is listed with *ListFS

Taking the command *Copy net#ftp::bbc/nvg/org.pub.docs.readme/txt adfs::4.temp.readme/txt
NetFS looks in the ListFS list for a file server serving a disk called "ftp"
It then looks in the ListFS list checking the servers with the net.stn found above for a disk called "bbc/nvg/org"
NetFS then inserts a "$." into the pathname it then uses to talk to the file server (nb: see below)
It then does NetFS_ReadAttrs on ":bbc/nvg/org.$.pub.docs.readme/txt"
If the object is not found, NetFS lops the leaf off the path and does ReadAttrs repeatedly until it gets a match, which will always happen when it eventually gets to ":bbc/nvg/org.$", it then does a ReadAttrs again on the full path before giving a Not found error. I think this is so it can report Object 'xxx' not found refering to the first part of the path that is not there.

So, assuming the object is present, the server never sees the #ftp part, if it needs to the server has to remember this part from the logon process.

If you use a path that starts from @ (CSD), & (URD) or % (LIB), NetFS passes the relative path after the @/&/%, with the 'CSD context' passed in the transation pointing to the CSD, URD or LIB as appropriate. So:
*Info @.!Choices looks for CSD=CSD:"!Choices"
*Info &.!Choices looks for CSD=URD:"!Choices"
*Info %.!Choices looks for CSD=LIB:"!Choices"

If you use a path that starts $, but has no disk name in it, NetFS prepends it with the URD's disk name found when logging on. So (assuming my URD is on "User"),
*Info $.NetMail looks for ":User.$.NetMail"
*Info :Info.NetMail looks for ":Info.$.NetMail"

Note, there is an oddity in that a reference to "%" on its own as a directory is passed as a reference to "^.<URD name>", eg (assuming my URD is called JGH):
*Info % looks for "^.JGH"

A method for the server to decide what #name protocol the client has used is for the server to remember a protocol for each disk name, so when the server get a request for
":bbc/nvg/org.$.blah... the server has previously noted this is a ftp method, and
":mdfs/net.$.index/htm the server has previously noted this is a http method.

This fails if a particular host name can be used with different protocols. For example the server cannot see the difference between:
*Copy net#ftp::mdfs/net.$.index/htm .... and
*Copy net#http::mdfs/net.$.index/htm ...

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.25
(C) Copyright J.G.Harston 1989,2005-2015
>_

User avatar
myelin
Posts: 204
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Arduino Filestore

Postby myelin » Tue Nov 07, 2017 11:21 pm

gazzaD wrote:It's been a very interesting exercise for me, as I've never needed to develop something as 'real time' as this before. Keeping the code readable, while also keeping it working is proving to be a challenge in itself. Some operations on the ADLC have to happen in a period of less than a microsecond, and so even with an 86 Mhz CPU available, when you are programming from a high level language (Arduino uses C) it's fairly easy to run out of clock cycles when trying to do things. It took several attempts at refactoring the transmission loop before I had something that had complete error checking, and executed fast enough, to not cause a buffer under run by the 4th byte.

Can you elaborate on this? I'm working out the details right now for an Econet module that doesn't use the MC68B54, and it looks like at the max 6502-friendly clock rate of 250 kHz, we're sending one byte every 32 us, and on an Archimedes at the max of 660 kHz, you should still have 12 us per byte. The most latency-sensitive thing I can find is the need to flag-fill right after receiving a scout packet, but there should be a bit of leeway even there. So now I'm wondering what I missed :)
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
So far: fast serial port, 32k flash cart, USB cart interface, 3-cart expansion, Elk PiTubeDirect.

User avatar
sweh
Posts: 1847
Joined: Sat Mar 10, 2012 12:05 pm
Location: New York, New York
Contact:

Re: Arduino Filestore

Postby sweh » Wed Nov 08, 2017 12:14 am

gazzaD wrote:Some operations on the ADLC have to happen in a period of less than a microsecond, and so even with an 86 Mhz CPU available, when you are programming from a high level language (Arduino uses C) it's fairly easy to run out of clock cycles when trying to do things. It took several attempts at refactoring the transmission loop before I had something that had complete error checking, and executed fast enough, to not cause a buffer under run by the 4th byte.


You don't have to write in C; it's possible to do stuff in assembler.

FWIW, back in 2014 I took a 16Mhz Arduino and tried to see how quickly I could toggle a bit.

First naive approach:

Code: Select all

void loop() {
  boolean x=false;
  int y;
  while(1)
  {
    x=!x;
    if (x) { y=HIGH; } else { y=LOW;}
    digitalWrite(LED_BUILTIN,y);
  }
}

That only got me 70KHz.

Then I was given a clue; don't use digitalWrite(), but write to the port directly:

Code: Select all

  DDRD=B11111111;
  boolean x=false;
  while(1)
  {
     x=!x;
     if (x) { PORTD=255; } else { PORTD=0;}
  }
 

That got me 748Khz.

Removing the test:

Code: Select all

  while (1)
  {
    PORTD=255;
    PORTD=0;
  }

3.9Mhz... but the on/off cycle was asymetric.

Code: Select all

  while (1)
  {
    PORTD=255;
    asm("nop");
    asm("nop");
    PORTD=0;
  }

gave me a 2.6Mhz square.

Image

Sometimes the libraries are very inefficient and bypassing them gives much better performance :-)
Rgds
Stephen

crj
Posts: 328
Joined: Thu May 02, 2013 4:58 pm

Re: Arduino Filestore

Postby crj » Wed Nov 08, 2017 1:08 am

myelin wrote:Can you elaborate on this? I'm working out the details right now for an Econet module that doesn't use the MC68B54, and it looks like at the max 6502-friendly clock rate of 250 kHz, we're sending one byte every 32 us

That assumes that, without the 6854, you have some alternative form of serialise/deserialise logic (with at least a couple of bytes of FIFO) so that you can operate byte by byte rather than bit by bit.

User avatar
myelin
Posts: 204
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Arduino Filestore

Postby myelin » Wed Nov 08, 2017 5:25 am

crj wrote:That assumes that, without the 6854, you have some alternative form of serialise/deserialise logic (with at least a couple of bytes of FIFO) so that you can operate byte by byte rather than bit by bit.

That's exactly what I have in mind (using a small CPLD to handle clocking bits onto and off the wire, flag matching, and zero insertion/removal) :) I'm curious about operations that have to happen in less than 1us, though, because as far as I can tell, nothing should be anywhere near as latency sensitive as that.
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
So far: fast serial port, 32k flash cart, USB cart interface, 3-cart expansion, Elk PiTubeDirect.

philb
Posts: 118
Joined: Sat Aug 05, 2017 6:05 pm

Re: Arduino Filestore

Postby philb » Wed Nov 08, 2017 9:21 am

myelin wrote:
crj wrote:That assumes that, without the 6854, you have some alternative form of serialise/deserialise logic (with at least a couple of bytes of FIFO) so that you can operate byte by byte rather than bit by bit.

That's exactly what I have in mind (using a small CPLD to handle clocking bits onto and off the wire, flag matching, and zero insertion/removal) :) I'm curious about operations that have to happen in less than 1us, though, because as far as I can tell, nothing should be anywhere near as latency sensitive as that.


No, I don't think there is anything that sensitive. 1µs is less than one bit time at any feasible clock rate, for a nontrivial network at least.

Once you receive a scout you need to start outputting flags quickly enough that the guy at the other end doesn't see an abort/idle condition, but you have at least six bit times to do that. At a clock rate of 500kHz that'd be 12µs. I can't think of anything that would need to be done more quickly than that.

gazzaD
Posts: 20
Joined: Sun Jun 18, 2017 11:37 am

Re: Arduino Filestore

Postby gazzaD » Tue Nov 14, 2017 12:30 am

philb wrote:
myelin wrote:
crj wrote:That assumes that, without the 6854, you have some alternative form of serialise/deserialise logic (with at least a couple of bytes of FIFO) so that you can operate byte by byte rather than bit by bit.

That's exactly what I have in mind (using a small CPLD to handle clocking bits onto and off the wire, flag matching, and zero insertion/removal) :) I'm curious about operations that have to happen in less than 1us, though, because as far as I can tell, nothing should be anywhere near as latency sensitive as that.


No, I don't think there is anything that sensitive. 1µs is less than one bit time at any feasible clock rate, for a nontrivial network at least.

Once you receive a scout you need to start outputting flags quickly enough that the guy at the other end doesn't see an abort/idle condition, but you have at least six bit times to do that. At a clock rate of 500kHz that'd be 12µs. I can't think of anything that would need to be done more quickly than that.


Remember the 68B54 is clocked at 2Mhz, regardless of what speed the network is. If you look at the bus timing diagram on the data sheet, you will see you can only read and write to the ADLC for part of that 2Mhz cycle. Too short a write and the byte is not detected giving you an underrun, and if you assert the write for too long you get corrupted data appearing on the network - or a double posting of the byte if the FIFO wasn't full. The ADLC is very fussy about timing.

In between writes you have to reverse the bus direction, and repeatedly poll (and decode) the status registers until the ADLC indicates it is ready for another byte (TDRA), or an error like an underrun or carrier loss occurs. When you have TDRA you then have to swap the bus direction and start the process to write the next byte.

As it happens, there are a couple of places in the code where I'm firing off a large set of assembler NOPs to provide those sub microsecond timings, and I am having to address the Atmel registers directly, as it's not possible to get a fast enough throughput using the Arduino library routines.

These are only considerations if you are using a real ADLC though. And having now discovered them, i am considering a step back to try bit banging the Econet bus from the Arduino via a LM319 / 75159 combo instead of using the Econet card. Although I'm a bit short of time for my various projects at the moment.
Gareth

User avatar
myelin
Posts: 204
Joined: Tue Apr 26, 2016 9:17 pm
Location: San Francisco, CA, USA
Contact:

Re: Arduino Filestore

Postby myelin » Tue Nov 14, 2017 1:09 am

gazzaD wrote:Remember the 68B54 is clocked at 2Mhz, regardless of what speed the network is. If you look at the bus timing diagram on the data sheet, you will see you can only read and write to the ADLC for part of that 2Mhz cycle. Too short a write and the byte is not detected giving you an underrun, and if you assert the write for too long you get corrupted data appearing on the network - or a double posting of the byte if the FIFO wasn't full. The ADLC is very fussy about timing.

Ahhhhhhh that makes a whole lot of sense. Thanks for the explanation!

Bit banging the bus straight from your chip is a clever idea. That'll be challenging too, but maybe not as challenging as having to sync up with the ADLC's 2MHz clock!

I'm nearly done with the PCB for my project, which uses an SN65C1168 dual RS422 transceiver chip to handle both data and clock transmission and reception. That one might work well for your bit banging project also.
SW/EE from New Zealand, now in San Francisco, making BBC/Electron hardware projects for fun.
So far: fast serial port, 32k flash cart, USB cart interface, 3-cart expansion, Elk PiTubeDirect.

User avatar
fordp
Posts: 923
Joined: Sun Feb 12, 2012 9:08 pm
Location: Kent, England

Re: Arduino Filestore

Postby fordp » Tue Nov 14, 2017 7:58 am

If you use a timer for the Chip Select generation then that should remove your 1us sensitivity. However with interrupts off 1us should be easy to to on the microcontroller.
FordP (Simon Ellwood)
Time is an illusion. Lunchtime, doubly so!


Return to “hardware”

Who is online

Users browsing this forum: Bing [Bot] and 8 guests