Multi-Threaded TSC (Proof of Concept)

I was originally planning to post this in Modding General Discussion, but it looks like other demonstration-type mods are always in the showcase, so here we go.

On modern computers, you can run multiple applications in parallel.
For example, you could have a text editor, an image editor, and a web browser open simultaneously.

Have you noticed that when you edit TSC scripts, there's normally no way to run multiple TSC events at the same time?
Well - I've created this little assembly hack (a proof of concept hack)
that lets you run up to two TSC events at the same time.

There are many limitations to this prototype hack, so it may not be able to do what you actually want it to do (such as running an event in the background while still allowing you to open doors / talk to NPCs). With a little work, this hack should become more advanced in the future and may be able to support such things.

The implementation is based on three TSC commands. Two of them are newly hacked ones:

<THRXXXX
Creates a new thread that runs event X. In other words, this command will continue to run the current event and will begin running event X at the same time.
If two threads are already running, this command does nothing.

<DCT
Destroy current thread. This will end the currently running thread without affecting any other threads.
If no other threads are running, then <DCT is the same as <END.

<END
Destroys all threads, including the one that is running.

Download the Demonstration Mod
(use the Terminal on the right to run events 103 and 104 at the same time.
use the Computer on the left to run events 105 and 106 at the same time.
Do not start up a new game - just use the existing save file.)

Here is some of the TSC code for the room.
The computer will begin with event #0102 and the terminal will begin with the event #0101.

#0101
<KEY<MSGRunning events 103 and 104
at the same time...<NOD<CLO<EVE0103

#0102
<KEY<MSGRunning events 105 and 106
at the same time...<NOD<CLO<EVE0105

#0103
[color=#FF0000]<THR0104[/COLOR]
<SOU0044<WAI0015
<SOU0044<WAI0015
<SOU0044<WAI0015
<SOU0044<WAI0015
<SOU0044<WAI0015[color=#FF0000]<DCT[/COLOR]

#0104
<SOU0029<WAI0015
<SOU0029<WAI0015
<SOU0029<WAI0015
<SOU0029<WAI0015
<SOU0029<WAI0015[color=#FF0000]<DCT[/COLOR]

#0105
[color=#FF0000]<THR0106[/COLOR]<SOU0012<CMP0101:0078:0000
<WAI0010
<SOU0012<CMP0102:0078:0000
<WAI0010
<SOU0012<CMP0103:0078:0000
<WAI0010
<SOU0012<CMP0101:0079:0000
<WAI0010
<SOU0012<CMP0102:0079:0000
<WAI0010
<SOU0012<CMP0103:0079:0000
<WAI0010[color=#FF0000]<DCT[/COLOR]

#0106
<SNP0087:0101:0078:0001
<WAI0005<WAI0005
<SNP0086:0102:0078:0001
<WAI0005<WAI0005
<SNP0087:0103:0078:0001
<WAI0005<WAI0005
<SNP0086:0101:0079:0001
<WAI0005<WAI0005
<SNP0087:0102:0079:0001
<WAI0005<WAI0005
<SNP0086:0103:0079:0001
<WAI0005<WAI0005
<PRI<MSGEnjoy your hearts & missiles.<NOD<CLO<END

Event 103 will create explosion sounds and event 104 will create teleportation sounds.
Because the events are run in parallel using <THR, you will hear the explosion and teleportation sounds at the same time.

Event 105 will destroy some tiles on the map and will create a sound effect when each tile is removed.
Event 106 will create hearts and missiles at the same coordinates of the destroyed tiles.
These two events are also run in parallel, so when a tile is destroyed, a heart or missile will automatically show up in its place.
Event 105 ends earlier than 106 because it executes a <DCT before 106 can execute its <END.

Download the ASM Files (Doukutsu Assembler format)
Download the installable hex codes (Autohacker format)

WARNING:
--This hack uses the Heavy Press code as if it were free space, so don't use the heavy press in your mod if you want to install this hack without modifying the ASM.
--Do not use flags 7904 through 7969. Actually, I don't remember which set of flags I used for the hack, but they were definitely inside the range 7904 - 7969.
--This hack does not overwrite the second <FAC command, <XX1, or any other existing TSC command. All the normal commands are still there.

This is also my sneaky way of telling you that the Assembler has been updated to version 1.31 because of a progress bar bug in version 1.3.

To all ASM hackers: this is an open-sourcy hack, so go ahead and make modifications to make this hack more effective, etc. I really encourage you to post your findings and/or progress.

 

I have been wanting to give this a whirl but haven't had time to sit down with it yet. However, it looks extremely sexy and I think that with clever implementation it could open up some really exciting new gameplay and design opportunities.

Maybe I will try to implement something like this into my own engine ;]

 

I'm in the same boat as noxid (it makes me horny but I haven't gotten it on my drive yet).
I'll download and look at it once I finish my homework that is due in two hours.

Just an off-the-bat recommendation though, changing <END to <FIN or something and <DCT to <END would make it much more backwards-compatibly friendly.

 

So am I correct in thinking that this simply interleaves the two threads rather than having, uh, "true" multithreading?

I've just started looking at it so tell me if I'm an idiot.

 

So am I correct in thinking that this simply interleaves the two threads rather than having, uh, "true" multithreading?

Yes, those are the mechanics. I'm sorry if that's disappointing, but it's the easiest way to do it.

You could do it the real way with mutexes and such, but that'd be pretty confusing.

A thread is another mechanism for splitting the workload into separate execution streams. A thread is lighter weight than a process. This means it offers less flexibility than a full blown process, but can be initiated faster because there is less for the operating system to set up. What's missing? The separate address space is what is missing. When a program consists of two or more threads, all the threads share a single memory space. If one thread modifies the contents of address 0x800A1234, then all the other threads immediately see a change in the contents of their address 0x800A1234. Furthermore, all the threads share a single heap. If one thread allocates (via malloc or new) all of the memory available in the heap, then attempts at additional allocations by the other threads will fail.

But each thread is given its own stack. This means thread #1 can be calling FunctionWhichComputesALot() at the same time that thread #2 is calling FunctionWhichDrawsOnTheScreen(). Both of these functions were written in the same program. There is only one program.

So yeah, there isn't suddenly a "second stack". But there are two (virtual) program counters (not EIP, but the TSC script position thingy). The C++ / ASM itself is not multithreaded but the TSC parser emulates the functionality of a multithreaded interpreter.

 

Honestly it's what I expected -- It's a very slick implementation of it (probably much better than I could've done), and is a magnificent piece of code. I was simply wondering if you used the harder method, as it would have some pretty deep ramifications.

 

I have been wanting to give this a whirl but haven't had time to sit down with it yet. However, it looks extremely sexy and I think that with clever implementation it could open up some really exciting new gameplay and design opportunities.

Maybe I will try to implement something like this into my own engine ;]

I'd like to see that... where can we learn more about your engine?

 

http://n0xid.wordpress.com

I try to post what I've done each week but sometimes it amounts to little more than faffing about seven days a week. There's also a download link to "try" it that has little more than the base physics functional. It's got more than that now, but I haven't made such things public apart from telling what there is in the weekly blog posts.

Sometimes I forget that some people other than me might actually care about this

 

A new version is out!

Before I go any further, I'd like to thank Noxid for answering one of my ASM questions.

So now... the player is able to create 'second threads' simply by talking to an NPC or interacting with any entity with flag 0x2000 set. This provides much more flexibility for multi-threaded TSC because you can now run events in the background and still have the player do TSC stuff in another thread.

Here's a good example of what you can accomplish: Download Example Mod Version 5

I don't need to tell you what to do because Curly will explain everything once you start the executable. Just make sure you 'Load' game (don't create a new save file).

And the new ASM source code:

Multithreaded TSC Version 5 (Doukutsu Assembler Format)
Multithreaded TSC Version 5 (Autohacker Format)

We also have 2 new commands for special use:

<RTS = Reset thread safety. Normally, we don't want an infinite amount of calls to the same TSC event. What if a player opens a door and he accidentally opens it twice? Normally this cannot happen at all, but with 2 threads it's possible. The same door event will get interleaved with itself, which is usually not what you want. The solution is easy: put an <RTS at the end of the door event, and everything will work out.

<UNK = Very simple command. This will undo the effects of <KEY without needing to call <END or <DCT. Stands for "un-key".

 

Any luck/progress on fixing wai?

Other things:
- Cannot open item menu
- Down doesn't operate entirely as it normally does
- What is the technical stuff behind rts?

 

- Cannot open item menu

Aw... well that's not good. Part of the inventory is TSC-powered so it might be a little annoying to get that to run properly.

- Down doesn't operate entirely as it normally does

Yeah I know. The looking back sprite and the '?' animation need to be added when not interacting with an entity.

- What is the technical stuff behind rts?

Based on the implementation, if thread safety is not used, then whenever you interact with an entity with 0x2000, the entity's TSC event will be run over and over again ad infinitum. Since this is bad, thread safety stores the last event number called inside the 2nd thread into some global var within flagdata. Next time you call a new 2nd thread, the old event # inside flagdata is compared with the new one. If the two match, then no TSC thread is created.

To allow the same event to be called again, you should use <RTS, which simply resets the global variable within flagdata to -1. This lets the same event be called twice inside the 2nd thread, but only when the first call has completely finished (assuming that <RTS is close to <END or <DCT).

Any luck/progress on fixing wai?

I think the idea to making WAI non-atomic is to make it a self-modifying TSC command: <WAI0500 should wait for 1 tick, modify itself to be <WAI0499 (by changing the big TSC string itself), and then not update the script position.

The script position should only increased by 8 when <WAI finally reaches 0000. Each <WAI command will split itself (essentially) into a bunch of 1-tick <WAIs, which means individual <WAIs will not clog up threads.

EDIT: Actually there is an issue with this particular implementation of non-atomic WAI. When an event is called the first time, the WAIs will work fine, but when the event is called again, all its WAIs will look like <WAI0000. To fix this, each WAI must have a method for storing the original number of ticks and resetting itself once 8 is added to the script position. This could be done by reserving a special hex byte, such as 0x00, that will signify the instruction is a non-atomic WAI.

In other words, <WAI0500 will transform into <(0x00)(0x01F4)0499 once executed. The first part is 0x00, which signifies that this instruction is still WAI. The second part holds the original number of ticks, 0x1F4, which is 500 in decimal. The rest are ASCII bytes that are decremented accordingly until they reach 0000. Once that happens, the command is reset to <WAI0500, the parser moves on, and all is well.

 

Clever, that seems to be the best way to do it.
(Unless you can break it into even smaller chunks without killing the ram).
and of course you'd only have to use a slightly modified process for <WAS and whatever other ones there are


Also msg/ms2/ms3 is very atomic.
you could try the same method, but iirc the letter handling is a bit too complicated to allow msg(s[]) = head(s[]) + msg(tail(s[]))