New beebjit release: v0.9.1

discuss bbc micro and electron emulators (including mame) here!
User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Wed May 13, 2020 8:14 am

vanekp wrote:
Tue May 05, 2020 6:05 am
unimplemented:instruction:pc $1c0e opcode $cb
DuneRider(MicroPower).uef
Ah, that Dune Rider protected loader is a monster and deserves a reply all to itself. It's pulling all sorts of tricks. I checked in some notes as I added the missing opcodes. They are pasted below in case anyone in interested.

And then, its simple-ish teletext loading screen exposed an incorrect rendering bug in both beebjit _and_ jsbeeb (both fixed).

Perhaps it deserves the coveted title "tricky emulator test case". Keep them coming :)


Cheers
Chris

---
Tons of very unusual undocumented opcodes at various points in the loader.
First discovery of AXS.
Uses them not only to obfuscate but also compute:
[ITRP] 1C02: LAX $81
[ITRP] 1C04: SAX $0213
[ITRP] 1C07: LDA #$13
[ITRP] 1C09: LDX #$FF
[ITRP] 1C0B: SAX $0212
[ITRP] 1C0E: AXS #$A1
[ITRP] 1C10: STX $80
[ITRP] 1C12: RTS
...
Page crossing JMP (ind):
[ITRP] 1C6F: JMP ($04FF) [A=C8 X=03 Y=00 S=FB F=C 1 N]
...
First ever instance of LAX imm (unstable!):
[ITRP] 0401: LAX #$00
[ITRP] 0403: SEI
...
Hits tape / serial hardware registers with undocumented instruction:
[ITRP] 041A: LDA #$03
[ITRP] 041C: SAX $FE08
[ITRP] 041F: LDA #$9D
[ITRP] 0421: SAX $FE10
[ITRP] 0424: LDA #$91
[ITRP] 0426: SAX $FE08
...
Undocumented alias for SBC imm at $0463 (opcode $EB).
---

User avatar
vanekp
Posts: 772
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: New beebjit release: v0.9.1

Post by vanekp » Wed May 13, 2020 8:28 am

ApplePie loads fine on BeebEm, but then Beebem does not care about baud rate or parity they are just ignored. was also mention in this thread viewtopic.php?f=1&t=19220#p268151
its one that switches to 300 baud for the 2nd small loader program.
ApplePie.png
Oh and the reason it hangs on any of the emulators it you need to set page to &E00 before loading it or it will hang which is why I find it handy with beebjit to mount a blank ROM in place of the DFS then I always have page at &E00
and it works fine on beebjit as long as I take the &0117 chunks out and beebjit also does not care about baud rate.
Peter.

User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Wed May 13, 2020 8:44 am

vanekp wrote:
Wed May 13, 2020 8:28 am
Oh and the reason it hangs on any of the emulators it you need to set page to &E00 before loading it or it will hang which is why I find it handy with beebjit to mount a blank ROM in place of the DFS then I always have page at &E00
and it works fine on beebjit as long as I take the &0117 chunks out and beebjit also does not care about baud rate.
Thanks. Seems to work fine with latest beebjit master. Latest master also includes the new -no-dfs command line option which should be less ugly that loading a blank ROM on top of DFS.


Cheers
Chris

User avatar
vanekp
Posts: 772
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: New beebjit release: v0.9.1

Post by vanekp » Wed May 13, 2020 9:20 am

:lol: hehe well it was the only way I could work out how to start the emulator without a DFS. but the noDFS is a nice new option for that thanks :D
Peter.

User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Thu May 14, 2020 11:42 am

vanekp wrote:
Tue May 05, 2020 6:05 am
and one with a crazy long gap at the end :- BAILING: uef file strange float gap 907.934021
CompleteBBC,The(Audiogenic)[tape-2][side-1]_hq.uef
Fixed as of latest master.
unimplemented:instruction:pc $1c0e opcode $cb
AndroidAttack(ComputerConcepts).uef
Another fantastic find! This exploded the JIT engine itself. The protected loader is obfuscating by using 16-bit address space wraps to store into the zero page. Most of these were working but it does this using different addressing mode, and indexed Y blew up.
Also fixed as of latest master.

Thanks again. Happy to fix any more of these pesky protected loaders you find :)


Cheers
Chris

User avatar
vanekp
Posts: 772
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: New beebjit release: v0.9.1

Post by vanekp » Thu May 14, 2020 4:31 pm

good to hear, I think we have more or less had all the ones with any weard and wonderful loaders that could cause problems.
Will have to give it a test once the new version is released.
Peter.

User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Fri May 15, 2020 7:34 am

vanekp wrote:
Sun May 10, 2020 5:05 am
move my response to this thread its more appropriate :wink:
ahh okay easy enough if you know how.
things like trace till a register is equal to some value could be a useful function i.e. "trace till y =FF" same for A and X registers.
"m" only shows 16 bytes of memory be nice it it could dump more.
"D" disassemble at <a> keeps disassembling the same block (when you pres enter) would be useful if it disassembled the next block of code.
I notices n also did some sort of trace but not mentioned in the help ?
Thanks for these suggestions. I made "d" behave how you describe, and also gave "m" the same treatment. Furthermore, "m" now shows 4 lines of memory by default, as per suggestion.

"n" is "next instruction", i.e. use it to jump over a JSR.
"f" is "finish", it attempts to guess where an RTS might land and stop there.

I've added a TODO to add generic conditional breakpoints. Shouldn't be too hard but should be pretty powerful and help with my protected loader debugging to boot!


Cheers
Chris

User avatar
vanekp
Posts: 772
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: New beebjit release: v0.9.1

Post by vanekp » Fri May 15, 2020 10:02 am

sounds good, will give it a test when it is ready :D
Peter.

User avatar
BigEd
Posts: 2995
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: New beebjit release: v0.9.1

Post by BigEd » Fri May 15, 2020 10:12 am

I've sometimes wondered if it would be handy to have
- step until next untaken backward branch
or
- step until next taken forward branch
or possibly a combination: as a shortcut for exiting loops.

User avatar
tricky
Posts: 4245
Joined: Tue Jun 21, 2011 9:25 am
Contact:

Re: New beebjit release: v0.9.1

Post by tricky » Fri May 15, 2020 11:51 am

Can beebjit import labels from beebasm?
I posted my code to export nested labels, which I think is now in "main".
For my beebem, I list them as you step and allow watches, breakpoints and databreakpoints to use them as targets.

User avatar
Arcadian
Site Admin
Posts: 3276
Joined: Fri Nov 24, 2000 12:16 pm
Contact:

Re: New beebjit release: v0.9.1

Post by Arcadian » Fri May 15, 2020 7:33 pm

scarybeasts wrote:
Thu May 14, 2020 11:42 am
vanekp wrote:
Tue May 05, 2020 6:05 am
unimplemented:instruction:pc $1c0e opcode $cb
AndroidAttack(ComputerConcepts).uef
Another fantastic find! This exploded the JIT engine itself. The protected loader is obfuscating by using 16-bit address space wraps to store into the zero page. Most of these were working but it does this using different addressing mode, and indexed Y blew up.
Also fixed as of latest master.
Just announced - the author of Android Attack, Paul Hiscock - will be speaking at 6pm during tomorrow's Virtual ABug session, so you can quiz him about his pesky tape loader in the Q&A section if you want!

(I had the original Android Attack tape as a kid and it wouldn't even load on our BBC due to the copy protection - it worked on other BBCs though!).
For a "Complete BBC Games Archive" visit www.bbcmicro.co.uk

SteveF
Posts: 564
Joined: Fri Aug 28, 2015 9:34 pm
Contact:

Re: New beebjit release: v0.9.1

Post by SteveF » Fri May 15, 2020 10:45 pm

tricky wrote:
Fri May 15, 2020 11:51 am
Can beebjit import labels from beebasm?
I posted my code to export nested labels, which I think is now in "main".
For my beebem, I list them as you step and allow watches, breakpoints and databreakpoints to use them as targets.
At the moment it's just on my personal fork of the beebasm repo here: https://github.com/ZornsLemma/beebasm/c ... bel-export
I've also (just now) created https://github.com/stardot/beebasm/issues/42 so this doesn't get forgotten about the next time there's a push to merge any pending changes and make a new "official" beebasm release.

User avatar
BigEd
Posts: 2995
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: New beebjit release: v0.9.1

Post by BigEd » Sat May 16, 2020 9:39 pm

Thanks for your talk at the virtual ABUG Chris. Some points arising:

Following on from Dominic's question about self-modifying code being cache-unfriendly on ARM: how much speedup comes from just the inlining and collapsing of uninterrupted blocks - in other words, if self-modifying code is run with the slow path, how much gain is left?

Is JS, or web assembly, a useful and feasible target for these techniques?

I have a couple of use-cases in mind: if the CPU load of emulation can be decreased, then a cheaper Pi can run a smoother emulation, and JSBeeb can emulate a full speed second processor on a modest laptop.

And a possible use-case for speed: testing of difficult 6502 code, perhaps, like a Reed-Solomon decoder, or a floating point library. More speed means more test cases in less time.

User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Sun May 17, 2020 2:25 am

Hi,
BigEd wrote:
Sat May 16, 2020 9:39 pm
Thanks for your talk at the virtual ABUG Chris. Some points arising:

Following on from Dominic's question about self-modifying code being cache-unfriendly on ARM: how much speedup comes from just the inlining and collapsing of uninterrupted blocks - in other words, if self-modifying code is run with the slow path, how much gain is left?
Not sure I understand the question fully. The slow path is to enter the compiler, which is very slow relative to execution and should be avoided where possible.
With a re-jig of how things work, It may be possible to use a different scheme for self-modification which is more friendly with incoherent caches.
Is JS, or web assembly, a useful and feasible target for these techniques?
I don't know much about WebAssembly, unfortunately. I'll see if I can get any jsbeeb developers to weigh in.


Cheers
Chris

User avatar
BigEd
Posts: 2995
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: New beebjit release: v0.9.1

Post by BigEd » Sun May 17, 2020 8:55 am

Ah, by slow path I was thinking of a conventional instruction-at-a-time emulator.

I had another thought: outside of copy protection, what forms of self-modifying code do we see? I would expect it's mostly a tactic used to save a few cycles, by patching operands to avoid indirection, possibly doing it in zero page to save a few more. Would it be possible to look at these routines macroscopically, and perform the actions they perform, separating out the incidental write to the code stream from the important writes to memory or peripheral?

That is to say, a self-modifying sprite routine could be recognised and special-cased to run at full speed. A self-modifying decryption routine will be emulated conventionally.

User avatar
tricky
Posts: 4245
Joined: Tue Jun 21, 2011 9:25 am
Contact:

Re: New beebjit release: v0.9.1

Post by tricky » Sun May 17, 2020 9:33 am

I would have thought that the most common form of self modifying code is just loading a program!

User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Sun May 17, 2020 9:45 am

BigEd wrote:
Sun May 17, 2020 8:55 am
Ah, by slow path I was thinking of a conventional instruction-at-a-time emulator.

I had another thought: outside of copy protection, what forms of self-modifying code do we see? I would expect it's mostly a tactic used to save a few cycles, by patching operands to avoid indirection, possibly doing it in zero page to save a few more.
Sprite loops tend to use a lot self-modification. As you note, hosting in the zero page is a useful trick! I hit that with both Wizadore and Stryker's Run (same author) and that was annoying to implement in the JIT :)
But it's not always so simple. If we take Exile, which is optimized to within an inch of its life, there's a large variety of self-modifcation, including a lot of opcode self-modifcation.
A search for "self mod" in the Exile annotated code is a bit of an eye-opener: http://www.level7.org.uk/miscellany/exi ... sembly.txt
Would it be possible to look at these routines macroscopically, and perform the actions they perform, separating out the incidental write to the code stream from the important writes to memory or peripheral?
Yeah, it's probably a productive direction. If we look at the start of the Galaforce sprite routine at $0B00, it goes wild with self-modifying stores:
(6502db) d b00
[ITRP] 0B00: STA $0B4F
[ITRP] 0B03: STA $0B60
[ITRP] 0B06: STA $0B71
[ITRP] 0B09: STA $0B82
[ITRP] 0B0C: STA $0B93
[ITRP] 0B0F: STA $0BA4
[ITRP] 0B12: STX $0B56
[ITRP] 0B15: STX $0B67
[ITRP] 0B18: STX $0B78
[ITRP] 0B1B: STX $0B89
[ITRP] 0B1E: STX $0B9A
[ITRP] 0B21: STX $0BAB
...
(a bunch more follow too!)

It's most common -- by far -- for self-modifying stores to use the plain abs mode, so the JIT engine has a fully resolved address at compile time. It knows whether x64 code has been compiled at any given address and can act accordingly, including optimizing away the "self-modifcation stamp" for stores to non-code addresses.
At a minimum, coalescing the cache invalidations would be do-able. I do worry that a more complicated scheme would be prone to bugs. For example, if we emit compiled code that contains assumptions about what is or is not at a given store, those assumptions are themselves prone to invalidation. And then those invalidations might cascade across blocks. etc.


Actually, having just written all that, when running Galaforce, beebjit reaches a steady state where there are no compilations occurring. This is because all self-modified operands are spotted and recompiled to just load the latest operand from 6502 memory. So although there's a lot of self-modifying going on, there are no recompiles (or potential ARM cache invalidations) going on. In the short term, I'm going to work on getting even "difficult" cases like Exile running with no recompiles in the steady state.


Cheers
Chris
That is to say, a self-modifying sprite routine could be recognised and special-cased to run at full speed. A self-modifying decryption routine will be emulated conventionally.

User avatar
BigEd
Posts: 2995
Joined: Sun Jan 24, 2010 10:24 am
Location: West
Contact:

Re: New beebjit release: v0.9.1

Post by BigEd » Sun May 17, 2020 10:12 am

> beebjit reaches a steady state where there are no compilations occurring

That's a promising observation!

> coalescing the cache invalidations would be do-able

In the case of ARM, if cache invalidation were too costly, maybe it's possible (and worthwhile) to place the volatile code in uncached memory?

User avatar
Matt Godbolt
Posts: 191
Joined: Mon Jul 31, 2006 11:02 am
Location: Chicago
Contact:

Re: New beebjit release: v0.9.1

Post by Matt Godbolt » Mon May 18, 2020 1:26 pm

BigEd wrote:
Sat May 16, 2020 9:39 pm
Is JS, or web assembly, a useful and feasible target for these techniques?
The core of jsbeeb was tweaked a while back to make it a little more amenable to being "JIT"ted, but the biggest concern is SMC. The clever trick beebjit is able to do to detect writes in a cheap way is not available in JS, so every write to memory has to have a super quick "did that change anything important?" check. I've yet to come up with one that isn't likely to be slower than "just" interpreting everything.

It's something I'd love to get working though :)

User avatar
vanekp
Posts: 772
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: New beebjit release: v0.9.1

Post by vanekp » Wed May 20, 2020 7:21 pm

what is required to build a new version of beebjit for windows using the master from https://github.com/scarybeasts/beebjit.git ?
Peter.

User avatar
scarybeasts
Posts: 319
Joined: Tue Feb 06, 2018 7:44 am
Contact:

Re: New beebjit release: v0.9.1

Post by scarybeasts » Wed May 20, 2020 9:37 pm

Hi,

I currently make the Windows builds by cross-compiling on Linux.
The script is build_win_opt.sh. It depends on the Linux gcc cross-compile package gcc-mingw-w64-x86-64 and not much else.

It should be easy to build on Windows itself. The input to the build process (deliberately) is little more than a list of C files.

I can make an interim build from master if it would be useful to you. v0.9.2 will be released once I've sorted out some HFE issues, including the one you reported.


Cheers
Chris

User avatar
vanekp
Posts: 772
Joined: Thu Nov 30, 2000 7:09 am
Location: The Netherlands
Contact:

Re: New beebjit release: v0.9.1

Post by vanekp » Thu May 21, 2020 9:18 am

managed to do a windows build in Linux Mint without a problem, no idea how to do it in Windows, but it works in Linux so all is good :)
Peter.

Post Reply

Return to “8-bit acorn emulators”