Could we design a TAS-k oriented workflow? (pun)

[TheCuriousHobbyist]

    One of Dolphin's aspects I guess is that it can be used to create a TAS (tool-assisted superplay) movie. However, the tools (pun) that are useful for this are a bit scattered. Example: let's say you start a game, of course one you dumped right?, and then you want to create a TAS. Well, how do you create it? Well, you could record your screen, which might be your first choice, but that can be faulty in that Dolphin can slow down, and if you don't have a frame count from the game recorded in there, like if you just capture the game screen, you have basically a movie that is inaccurate and is not that fun to watch, either. Also, you don't just capture the screen obviously while creating because TAS involves slowdown and frame advancement, so it's not just realtime. That's not how it works. So you'd need some way of recording a video that only contains the game's frames, not the stuff in between that makes it the realtime process time of the super play. So Dolphin does have Dump Frames and Dump Audio, but the thing is that 1) you might not know what you're looking for when you're thinking about that kind of feature, and it's hidden in Dolphin's menu bar, so you might not know where to look either. So this is the first flaw (thing that's not really optimized for a quick TAS creation workflow). Another problem is what if you want to play back your inputs on the exact frame you pressed each one? Dolphin does have input recording, but again it's hidden in the menu bar, and also you need to start the input recording without having the game open and it will start the game, otherwise if you don't know what frame you started recording on in game, which requires Show Stats to be enabled, you have to guess if you even can, otherwise you'll play it back and it starts on Frame 0, when you may have started it on Frame 427 or something, and so your inputs are out of sync by several seconds, which leads to the game getting to states that you didn't get it to in your initial recording, your player going to different places, or running around randomly into walls or dying, which leads to you being annoyed, and of course confused as to why it happens.
Another big thing is Save States. It's not clear whether your input recording is ruined by loading a state, because maybe it's written the things you did since saving and before loading it, and it doesn't delete that stuff, making for a non-continuous desynced input rec. Another thing is random events. It doesn't (at least to my knowledge) record any random numbers in the game so when it's played back those events that were randomized are set so they're not random when you play it back, the same things happen. Maybe it does and that's why it has save states, but it doesn't make it that clear is my point. The problem I have largely with it though, is the 2 things you use frequently while sequencing a TAS run. Slow motion, and frame advancement. Frame advance is hidden in the menu bar. So let's say you're doing a glitch that requires total agility and frame-by-frame advance for 20 frames or something, you now have to go to your menu bar and click Frame Advance 20 times. Or configure it in your hotkey settings. Problem then is that you can't use hotkeys and a controller simultaneously. So if you just hit F to advance a frame, you can't just go to your controller and start pressing stuff and expect it to work, you now have to go to Controllers > Config and then click Refresh, so it recognizes the controller is the input device you want to use, which is obscure. You also need to change the speed for a controller to really be that useful. Problem: It's buried in the Config panel. So it's just a bunch of flicking back and forth on and off between a bunch of places to get to the options that are useful here. Scattered TAS oriented features do not ( and I repeat DO NOT) make for a TAS-k orientated workflow. It's kind of outlandish to me honestly, as I think many using Dolphin want to create runs like Malleo's TTYD speedrun with these tools, they just don't know how or if they do, they find the features annoying to work with by running back and forth between them. So I've actually designed a mockup for a workflow, and I'll update you with the pictures later, as right now I want to post this, I'm kind of anxious to share my thoughts. As you might expect me to say as well, feel free to share yours on this thread.

Update; here are the mockup images.
   
   
   
   

[JMC47]

I can't comment on everything except I'd love to improve the TAS workflow. We don't have anyone currently in the project that is using the TAS tools actively. It may be worth having a separate window that pops up (like graphics settings does) with all the TAS tools in it so it can be moved to a second monitor, etc.

On some of the features.

Dolphin obviously supports input recordings, playing back input recordings, savestates during input recordings. If things don't sync up properly, it means you're doing something wrong or something is broken. I used to do all of my videos through recording inputs ahead of time and then replaying the inputs with framedump on at a super high resolution.

If you're willing to stick around and work with us a bit (and, unfortunately, be patient due to the complexity of things) I'm sure we can sort things out and make it easier.

[TheCuriousHobbyist]

(06-11-2019, 11:38 PM)JMC47 Wrote: I can't comment on everything except I'd love to improve the TAS workflow.  We don't have anyone currently in the project that is using the TAS tools actively.  It may be worth having a separate window that pops up (like graphics settings does) with all the TAS tools in it so it can be moved to a second monitor, etc.

On some of the features.

Dolphin obviously supports input recordings, playing back input recordings, savestates during input recordings.  If things don't sync up properly, it means you're doing something wrong or something is broken.  I used to do all of my videos through recording inputs ahead of time and then replaying the inputs with framedump on at a super high resolution.

If you're willing to stick around and work with us a bit (and, unfortunately, be patient due to the complexity of things) I'm sure we can sort things out and make it easier.

OK, so first, I do appreciate your response. So as for the fact that there's no one in the project using Dolphin's TAS tools actively, there clearly are people that used it to create speedruns. I have made a video about Dolphin recently, check it out here: https://youtu.be/KeXgKmk_d9Y
In the description I included a link to Malleo's TAS video run, which I also used an excerpt from. Not really sure what you mean by people in the project, pretty sure it's people involved in developing Dolphin, people that use the forums, something like that. Doesn't matter. (although I kind of am curious) The reason this might be is something I feel like I've illustrated.
Also, separate window? Cool if you want a change of plan or something, but I didn't depict the workflow in a separate window. I depicted it as an elongation/extension of the main window, whether it's rendered to or not. 
You also wouldn't have to have a second monitor to use it. You could run the iso in part-screen with the tools by your side. If in the future they make it so you can choose to have the workflow in a separate window, that's obviously fine and such.

As for the features and design, I'd like to go through with it by commentating on the images I've exported, attached and inserted into the post. The first image is the interface that would appear when you click "Input Rec Mode" in the top bar, and it's the initial settings.

-- IMAGE #1:

-- Relative frame rate would be the amount of FPS the game would run relative to the speed of everything. So 60 and 30 would make it behave the same. 
-- Record Input obviously determines whether you want to actually record input or if this is just a test run and you just want to try using the workflow. I mean, you can use TAS input without recording, so...
-- Config Override is a choice for the profile of settings you want to use, it would have lightweight and high-end presets designed against desync issues. Also, it might be imperative to make the game ignore the save data/memory slots, so that it doesn't save a progress file and it desynchronizes because the sequence can't erase it once you play it back. And obviously we're talking about a workflow that makes save states an order of magnitude easier to work with, who needs the memory slots?
-- State to Load is the state you want to load in the folder if you have any states already saved there. 1 File from Each State Load creates an individual input sequence file from the point where you load a state.
-- Random numbers obviously records random numbers to the file. This is for any randomized thing in the game. (Edit: No algorithm-generated number is random, it's stuff that's used to seed random numbers, thanks for making that clear, but my point remains the same that it doesn't really give you an answer as to whether it already does this, and even though maybe there's no reason to have control over whether that's saved, that might at least make it clear Dolphin does this)
-- Write to Last Sequence writes the input being recorded from startup to the last sequence file you created. Maybe it could have a folder path choice actually, so you could choose a file to add the input on top of, like if you're saving a state. Maybe it could choose the last modified .dtm sequence in that folder if that's checked.
-- State folder is the folder for keeping your save states. The path to it is to the save states folder in the Dolphin support location, then the name of the folder you enter. Entering a name that's not equal to any in the save states folder creates a new one.
-- Obviously pressing Start starts the game you had selected before opening the TAS workflow. Obviously the game wouldn't be running like it "shows", I just used the same picture. 

-- IMAGE #2 (input sequence recording mode)

Wii remote input is self explanatory except for a couple things. I made it look kind of like a remote because I wanted the design to be easy to grasp, and so unlike the TAS input window for the remote it has, it makes more sense because the layout is the same. The red box is for recording rumble data. Maybe that could be used to add a custom sound sample to the audio dump in play mode (more on this later), or maybe it could be used to record to a separate file that can be used to make haptic engines rumble or phones vibrate or something. I don't know what I'm saying.

-- In gyro settings, Tilt is the rotation of the wii remote forward/backward, or on the up axis. The twist slider compensates for the fact that the knob can't move depth wise. It would also be cool if the knob handle (thing you drag) could change color based on the Twist slider. It's like twisting the remote, which rotates the pointer if it's on the screen. Velocity is the speed at which the remote is moving. This is for accelerometer emulation, as it simulates how fast and what direction it's moving in. Remote position changes the velocity to position of the remote from the sensor, and it converts the position change to velocity instead. Position is inches and velocity is inches/second. Depth is how far the remote is from the sensor in the direction the screen is pointing. Kind of the forward axis position.
-- The warning sign and the list of features to fix is features that get highlighted because they may cause desync. I might be wrong on my depictions, I'm just making this point.

-- So next, Nunchuk settings. the knob under the word Nunchuk is the joystick on the accessory, one of it's main controls. You've also got an accelerometer in there, so there's rotation, velocity, twist, same as the wiimote gyro settings. The switch knob on Velocity switches it to position, and there's Z, and the tiny C button above the Z trigger.

-- "Count any skipped" adds skipped frames to the below frames value. Record W/ Skipped records that same value to the input sequence. w/o Skipped records the frames without skipped. Don't actually really know what skipped frames are or what I mean by them, maybe some of this stuff can be subject to change.

-- Speed is how fast you want the game to run. Record Changes records the speed at any frame in the run. Frame Advance sets it to 0. Change Pitch is if you want to hear it get lower as you decrease the speed. Frame rate * speed is explanatory, maybe it doesn't need to be there.
-- States is followed by 30 squares. The idea is you save the current game state by clicking one of the squares, turning it red. This saves a state with the number of the state in its title to the state folder I mentioned. Clicking it again turns it green again, and deletes the state. If you want it back, the state is loaded, and the game would be paused automatically. Just click it again.

-- IMAGE #3 (Iteration of #2 to show possibilities)
-- Pointer Pos in Gyro settings turns everything except one knob grey, and makes the knob simply control the position of the pointer.
--D-Pad Realism prevents both up and down from being held (ticked, in this case) simultaneously, and same with left and right, simulating the one solid plastic D-Pad instead of allowing full control.

-- IMAGE #4 (Play Input Sequence)

-- Here, you can choose whether it takes skipped frames, speed changes (not altering the video, but the playback, useful for limiting the stuff your computer has to do), and random numbers (random seeds) into account.
-- You also have the previously hidden Dump Frames/Audio options, recording resolution, and whether you want it to capture the main window (if the game is rendering to it), of course that main window can have the workflow on the right side of itself, and if it's not a separate window, it wouldn't capture that. Also you can set if and how much you want to crop in the sides using comma separated numbers.
-- You can also set input sequences to play in order by browsing to them. It'll play them concatenated, in order.
Start Automation will gray out some things like save states, resolution, and stuff that doesn't need to be changed.

Maybe I could also add frame advance and speed sliders to the concept, but that's kind of one reason I'm writing a whole overview of what this stuff could mean and how it could be implemented. I'd like to see this implemented in the next stable build of Dolphin as Dolphin 6. Thanks for your response, I hope this community gets something of this, and I hope someone likes my explanation and finds it interesting.
-TheCuriousHobbyist

[JosJuice]

I don't really think I can comment on your whole proposal (parts of it are a bit hard to understand), but here's a few things...

(06-13-2019, 07:01 PM)TheCuriousHobbyist Wrote: So as for the fact that there's no one in the project using Dolphin's TAS tools actively, there clearly are people that used it to create speedruns. I have made a video about Dolphin recently, check it out here: https://youtu.be/KeXgKmk_d9Y
In the description I included a link to Malleo's TAS video run, which I also used an excerpt from. Not really sure what you mean by people in the project, pretty sure it's people involved in developing Dolphin, people that use the forums, something like that. Doesn't matter. (although I kind of am curious)

JMC most likely means people who are actively involved in the development of Dolphin (either by coding or by contributing in other ways). There are people who make TASes, but those of us who are working on the emulator have relatively little contact with them.

(06-13-2019, 07:01 PM)TheCuriousHobbyist Wrote: Random numbers obviously records random numbers to the file. This is for any randomized thing in the game.

It's not obvious at all what is meant by this. So-called randomness in games is just the output of a PRNG function, which is emulated game code just like any other game code. There's no "randomness hardware" or anything like that that outputs truly random numbers that would need to be saved. What does need to get saved is the initial value of the RTC and other things that could be used to seed PRNGs, but Dolphin already does this.

[JMC47]

You seem passionate about it, but part of being able to communicate is not writing gigantic, unformatted blocks of text that are extremely difficult to read.

If you're passionate about TASing and making it easier to TAS in Dolphin, my suggestion is to breakdown your suggestions in the small, bite-sized pieces, maybe report them on the issue tracker one by one as feature requests or even start them out here.

Maybe it'd be stuff like:

Issue 1: There's no centralized TAS menu - then explain what you'd want there and why and how it makes things better.

issue 2: Some bug that you're running into or behavior that you don't understand.

issue 3: etc.

Keep the descriptions concise and we can try to work from there.

In response to the giant block, you can probably find more info about Dolphin TASing on tasvideos - http://tasvideos.org/forum/viewforum.php?f=34&sid=9a82ffce939301fd2b70b92a0477e70a

[AnyOldName3]

Double line breaks between paragraphs would be a really good start to making your points clearer. Editing those into your existing posts would make them much more readable.

[TheCuriousHobbyist]

(06-14-2019, 01:30 AM)AnyOldName3 Wrote: Double line breaks between paragraphs would be a really good start to making your points clearer. Editing those into your existing posts would make them much more readable.

In response, I've brought out the individual features by bullet-pointing them, and made it more readable.

[TheCuriousHobbyist]

OK, so I just want to address a couple of new things I discovered. If your input recording goes wrong after you've been doing super well at TASing a game, and you just have a bit of extra frames after your issue started, you can save it, but if you play it, you'll have that overhang. There's no way to stop playing back the inputs, and if you choose "Export Recording" in the menu bar, it will just export the recording you played, not just up to that point. All of it. The only thing you can do is to save a state of the game at that point, load it when the broken input recording plays up to it (I'm not even sure if that stops playing an existing sequence), and if it does stop the currently playing one, play the next part. There doesn't seem to be much support for playing back input recordings/switching them out at custom points, writing to them from the current point of the sequence being played, and concatenation of input sequences into one consolidated file. Also, in Dolphin 5, Super Paper Mario crashes at the prologue, so maybe work on fixing that. All in all, I find the whole TAS "workflow" or set of features and the scattered-ness of said features kind of outlandish because they are there, I just think they might not be centralized enough to make many users want to use them, even though they might in the sense they find TAS runs really interesting and cool. What the?

[TheCuriousHobbyist]

(06-13-2019, 08:33 PM)JosJuice Wrote: I don't really think I can comment on your whole proposal (parts of it are a bit hard to understand), but here's a few things...


JMC most likely means people who are actively involved in the development of Dolphin (either by coding or by contributing in other ways). There are people who make TASes, but those of us who are working on the emulator have relatively little contact with them.


It's not obvious at all what is meant by this. So-called randomness in games is just the output of a PRNG function, which is emulated game code just like any other game code. There's no "randomness hardware" or anything like that that outputs truly random numbers that would need to be saved. What does need to get saved is the initial value of the RTC and other things that could be used to seed PRNGs, but Dolphin already does this.

As for the pseudo randomness stuff, thanks for clearing that up. I actually learned how pseudo-random numbers/values work via Khan academy relatively recently. I just wasn't aware of how Dolphin treated or would need to treat (pseudo-)randomness. My point does remain the same though. If you try to create a speedrun or just input recording and think to yourself "What if this random event is different next time I play this back"? And not to mention maybe not knowing the stuff that has to be set on or off. As for the fact that putting that option there has no known purpose, like why would you need an option like that, no it's not something many people would use, but it would be turned on by default, and when they don't need to be recorded, like in some cutscenes, it can be temporarily disabled for conservation purposes.

[JosJuice]

(06-16-2019, 10:15 AM)TheCuriousHobbyist Wrote: As for the fact that putting that option there has no known purpose, like why would you need an option like that, no it's not something many people would use, but it would be turned on by default, and when they don't need to be recorded, like in some cutscenes, it can be temporarily disabled for conservation purposes.

The RTC value that we store is not per frame, it's a single value that is stored for the start of the movie. So "turning it off" during parts of the movie would have no effect on the file size or anything else.