Replies: 91 (Who?), Viewed: 19394 times.
Page 4 of 4
Lab Assistant
#76 Old 2nd Jul 2021 at 12:58 PM
Quote:
Originally Posted by epochtimestamp
I spent some time trying to figure this out. Using ghidra to disassemble and decompile, I was able to find the code that parses the precomp file in TS3W.exe. I also found a call to stat64() just before the parsing. Debugging with IDA Pro, I was able to determine that the path to stat64() is the full path to Shaders_Win32.precomp in the game directory.

So far it isn't looking good for there being a way to override the file with a package in the mods folder. That said, all is not lost. I thought I understood how the game resolved the file location, but stepping through in the debugger I found that didn't actually work how I expected. So perhaps all I'm seeing is a failed check for a mod, resulting in loading the file from disk. I'll keep digging, but I'll have to find an alternative tool, IDA Pro doesn't seem to make looking at live memory easy and I think that's what I need to do if I'm going to figure out whether or not there's potential for overriding the file.

Cheat Engine is a good tool for digging into memory
Virtual gardener
staff: administrator
#77 Old 2nd Jul 2021 at 5:29 PM
Quote:
Originally Posted by SmugTomato
Cheat Engine is a good tool for digging into memory


I'd also recommend Cheat Engine! I've used it with some other games, not the sims though... I actually used this other tool which was a play on Luke skywalker but I can't find it now without getting all this stuff on the actual Luke skywalker But if you came across that, I wouldn't use that on the sims 3, it really wasn't tracking all the things for me. :p
Test Subject
#78 Old 5th Jul 2021 at 3:45 AM
Thanks for the recommendations.

Unfortunately I found Cheat Engine to be a little difficult to use. Searching around in memory for strings I'm pretty sure are there was not yielding any results... likely user error.

Anyway, I've proceeded with some more reverse engineering and identified more resource key lookups in the code. I haven't managed to tie any of those to the shader precomp file, so I'm leaning towards thinking that it cannot be overridden in a package file.

I'm going to focus my efforts on stepping through the code in the IDA debugger and try to understand why the modified precomp files fail to load. At the end of the day, the precomp-in-a-package is a nice to have, but not necessary for modding shaders.
Lab Assistant
#79 Old 9th Aug 2021 at 4:33 PM Last edited by LazyDuchess : 9th Aug 2021 at 5:23 PM.
So I'm a bit out of the loop on this whole research thing, haven't looked at the precomp file too much but I decided to put together a tool that lets you replace and extract the game's pixel shaders in a generic way by hooking into D3D directly. A bit like Reshade except it replaces shaders rather than adding on top.

I do know that the reason the game crashes if the precomp file is modified is because it contains the compiled D3D shaders in bytecode form. These have a checksum embedded in them, and if D3D notices that the data doesn't match it will reject it.

Anyway, here's the tool, 2 mirrors:
SimFileShare
Dropbox

Extract the tool into your Game/Bin, to launch the tool if you're on Steam/Retail just run the game as you normally would, if you're on Origin then use the TS3PatchLauncher.exe

The tool has 2 modes, extraction and replacement. It will prompt you to choose the mode when you boot the game up. It might ask you when you open the launcher as well, in this case it doesn't really matter what you choose and it's harmless as the launcher doesn't use any shaders of course. I already did the extraction step, so you can choose to just play around with replacing shaders.

In the shaders_extract folder the shaders extracted from the game get written to. I already included all of the pixel shaders from TS3 disassembled in this folder so they're in Pixel Shader assembly, not in bytecode format.

In shaders_replace you place your custom shaders. The filenames are the same as the extracted ones so you can just copy those into this folder and they will work. (Although I think it crashes after replacing about 510 shaders, still gotta figure that out)
For the file extensions, if your replacement shader is precompiled then give it the ".cso" format, if it's in Pixel Shader assembly ".msasm" and if it's a source HLSL shader then name it "*.hlsl" and name the entry point function "main".

There's also an "aliases.cfg" file inside shaders_replace, you can use it to map the IDs of the shaders to readable names and document things. It comes with a shader that I aliased "ExteriorFloors" which replaces the shader number 200, seems to be used by some outdoor flooring like asphalt tiles. The cache folder will generate precompiled versions of your replacement shaders when you run the game, it's just there for research purposes.

Anyway... I'm not very good at explaining I know, but any questions feel free to ask. I hope this is useful to someone, we could use it to replace random shaders to see what is what and give them proper names.

Forgot to add, it might be useful to launch the game in Bytecode extraction mode. The shaders extracted this way seem to match perfectly with things that can be found inside of the precomp file.
Lab Assistant
#80 Old 9th Aug 2021 at 5:28 PM
Just realized that I made a little typo. Make sure the assembly shaders end in ".msasm", not ".masm". Updated the files to fix this.
Virtual gardener
staff: administrator
#81 Old 10th Aug 2021 at 10:44 AM
Awesome! Thanks for posting this! Quite like this approach, which might just as well be the best way of doing it for now without having to decode the whole Precomp file (which isn't going tooo well)

I do have a question though, I see it's just replacing shaders that will work, correct? Would be interesting to see if we can intergrate custom shaders as well. Although then the issue you're going to have is S3PE But we can probably come up with a fix for that too!

For now, lemme play around with this, thanks for doing this again!
Lab Assistant
#82 Old 10th Aug 2021 at 3:49 PM Last edited by LazyDuchess : 10th Aug 2021 at 4:23 PM.
Quote:
Originally Posted by Lyralei
Awesome! Thanks for posting this! Quite like this approach, which might just as well be the best way of doing it for now without having to decode the whole Precomp file (which isn't going tooo well)

I do have a question though, I see it's just replacing shaders that will work, correct? Would be interesting to see if we can intergrate custom shaders as well. Although then the issue you're going to have is S3PE But we can probably come up with a fix for that too!

For now, lemme play around with this, thanks for doing this again!


No problem!

And yup, I'm thinking it'd be better and easier to hook the game's code to make a "custom loader" for new shaders that are just dropped in folders or something. This would break the "generic" aspect of this tool a little bit though, but it'd definitely be worth it imo. Maybe this tool could be (re?)purposed as a shader modding framework to install and make custom shaders.

I'm working on runtime in-game debugging features so we don't have to do a lot of guesswork to find shaders since there's so many. Currently implemented a reload shaders feature so you don't have to restart the game when making changes and a couple keys to turn shaders pink while playing so we can figure out which one is which.

Managed to fix the broken black catchlights on pets this way:



I've updated the original downloads with the new version. It will now boot straight into Shader Replacement mode with debugging enabled by default, this can be changed in the .cfg file that comes with it now. The new 0.msasm shader in the replacement folder is the debug shader. Debug controls are Ctrl and Shift to cycle between debugging shaders and the End button to reload all replaced shaders.
Forum Resident
#83 Old 10th Aug 2021 at 7:34 PM
@LazyDuchess Regarding this new tool, I'm curious about something. How exactly is lighting in TS3 controlled at the engine level?

I'm wondering because of the issue of multi-level lighting and off lot lighting being broken since WA. It'd be nice to finally be able to have those fixed, but I wasn't sure if the whole thing surrounding the game's shaders were even pertinent to this. I know next to nothing fundamentally about any of this but I wanted to ask.

You have been chosen. They will come soon.
Lab Assistant
#84 Old 11th Aug 2021 at 1:23 AM
Quote:
Originally Posted by Jathom95
@LazyDuchess Regarding this new tool, I'm curious about something. How exactly is lighting in TS3 controlled at the engine level?

I'm wondering because of the issue of multi-level lighting and off lot lighting being broken since WA. It'd be nice to finally be able to have those fixed, but I wasn't sure if the whole thing surrounding the game's shaders were even pertinent to this. I know next to nothing fundamentally about any of this but I wanted to ask.

Sadly it's not too relevant to shaders I believe. Something as complex as the whole lightmap engine TS3 has for indoors is probably mostly handled in the game's low level executable code, which would require a lot more reverse engineering to figure out.
Forum Resident
#85 Old 11th Aug 2021 at 1:27 AM
Quote:
Originally Posted by LazyDuchess
Sadly it's not too relevant to shaders I believe. Something as complex as the whole lightmap engine TS3 has for indoors is probably mostly handled in the game's low level executable code, which would require a lot more reverse engineering to figure out.


Was kinda figuring that'd be the case, but never hurts to ask.

You have been chosen. They will come soon.
Lab Assistant
#86 Old 11th Aug 2021 at 10:44 AM
I have pretty much zero knowledge on the sims 3’s shaders or how to best utilize this new tool that LazyDuchess has graciously bestowed upon the community, but I wanted to know if it was possible to port shaders over from the sims 3 medieval?
Virtual gardener
staff: administrator
#87 Old 11th Aug 2021 at 11:58 AM
Quote:
I'm working on runtime in-game debugging features so we don't have to do a lot of guesswork to find shaders since there's so many. Currently implemented a reload shaders feature so you don't have to restart the game when making changes and a couple keys to turn shaders pink while playing so we can figure out which one is which.


Ah sweet! Haven't gotten the chance yet to fully look into the shaders yet with the tool, but this would help tremendously!

Also! Small tip, if anyone wants to do any shading coding/fixing/working with them, last time I worked with them I actually did it on a test mesh in Visual studio 2017 (although up is fine too): https://docs.microsoft.com/en-us/vi...er?view=vs-2019

Quote:
Originally Posted by Alunn
I have pretty much zero knowledge on the sims 3’s shaders or how to best utilize this new tool that LazyDuchess has graciously bestowed upon the community, but I wanted to know if it was possible to port shaders over from the sims 3 medieval?


I'm thinking with the nature of the program, this might actually work, especially with the library it's using (and the way it hooks into the game) Of course, LazyDuchess would have to fully confirm, although now that got me curious to try it myself :p

Quote:
Regarding this new tool, I'm curious about something. How exactly is lighting in TS3 controlled at the engine level?


This is indeed not highly done with shaders (and usually it never is, unless we're talking about lightmaps being 'baked'). TS3 uses a lighting technique that TS4 doesn't have. In fact, they upgraded the whole lighting map thing to work more like they do these days. The technique TS3 uses as a light engine is actually done by light probes. I think Unity does a pretty good job of explaining what it is: https://docs.unity3d.com/Manual/LightProbes.html

That's not to say that TS3 doesn't use any lightmap, I think CAS uses different techniques since it clearly is generating a lot of interesting and different textures At least, I remember @LittleDica showing me how CAS just generates these types of textures I've Never ever seen in my life before! TS3 was pretty ahead of it's time technically speaking wise though.
Test Subject
#88 Old 11th Aug 2021 at 1:38 PM
I'm arriving late and maybe they've already asked this question, but does this discovery basically mean that we'll soon be able to get better shaders in-house? Because, basically, this is what is needed for this game to be perfect! :O
Screenshots
Virtual gardener
staff: administrator
#89 Old 11th Aug 2021 at 2:46 PM
Quote:
Originally Posted by boringbones
I'm arriving late and maybe they've already asked this question, but does this discovery basically mean that we'll soon be able to get better shaders in-house? Because, basically, this is what is needed for this game to be perfect! :O


I'd love to have that first image as my game I think that particular one was just from when they made the prototype in Unreal, or it was TS2 shaders > TS3. But I think we first need to figure out what kind of shaders there are in the first place, and what ID belongs to what (and have some really talented shader writers seeing this exact post, HLSL's can be quite hard to program with and learn tbh)

Now the shader names can give us an idea of what to expect, that's at least good! I am just a bit worried of what's part of the light engine (Which would mean cracking open the TS3W.Exe) or what's really done by shaders.
Virtual gardener
staff: administrator
#90 Old 11th Aug 2021 at 2:54 PM
I'm probably going to sound super nooby, but then all I've done with Shaders was all done in babylon.js, which is pretty much GLSL IIRC.

But as I understand it correctly, HLSL is basically the source code, CSO is the compiled version of it (Or well, the file that is being eaten by the GPU :p) And then the msasm in this case is what is 'coming out' of the GPU, (or rather, what's being 'cached') Right?

Just curious about how you managed to change some of the shaders, did you edit the CSO directly? Or the MSASM? I'm assuming the MSASM since that's the Pixel shader data, correct? And if so, I doubt there's any way we can dissasemble the CSO to get the HLSL files out of it again, right?

EDIT: Also! For the technical peeps, I found this useful article! https://interplayoflight.wordpress....hader-assembly/
Virtual gardener
staff: administrator
#91 Old 11th Aug 2021 at 5:21 PM
So through some researching and figuring out if an actual assembly > HLSL is possible, I stumbled across 3D Migoto (https://github.com/bo3b/3Dmigoto/releases ). Tried that first, then was wondering why nothing was being outputted. That, however, was because the program unfortunately only supports DX11 and up. TS3 Is DX9.

Sooo, then I found out that people usually bypass this with a wonderful Directx emulator tool; initially made to support games that run Directx8 and even lower (so old games :p), which will then be emulated to play on Directx11: https://www.pcgamingwiki.com/wiki/DgVoodoo_2

Now that I had that being covered, the great thing about 3DMigoto is that they have an option to extract (or rather an interpretation of the HLSL code by reading the ASM files) all the HLSL files. Now, the game does quit on me when I *actually* want to load the world, but it still seems to extract about the same kind of information as LazyDuchess .

Here are those HLSL files btw, but I'd recommend only using them for mad science-y research :p

(NOTE: all files are in txt files, but that doesn't mean the HLSL code isn't in there. If you open one of the txt files, you'll see where the HLSL is, I just happen to choose the option for it to extract both the previous ASM files, then the compiled ASM and then converting that to HLSL all in that txt file)
Download - please read all instructions before downloading any files!
File Type: zip 3Dmigato extracted shaders.zip (631.2 KB, 8 downloads)
Lab Assistant
#92 Old 11th Aug 2021 at 5:47 PM
Yup, so .CSO is the bytecode as it appears in the Precomp file, that can be directly passed to the GPU and has the checksum stuff so they can't really be modified.
.MSASM is kind of the same thing except in a more readable form, like opening up an .exe in a disassembler or cheat engine, just raw GPU instructions, no checksum.
Then finally HLSL is actual human readable "source" code.

We've done a bit of work, we got the TSM eye shaders to work which has smaller catchlights than vanilla TS3:

For comparison, vanilla TS3:


Fixed the censor being too dark:


And made bloom a lot more bearable:


The problem with porting more complicated TSM shaders is that they need a lot of extra information from the game code that TS3 is obviously not giving us, so a feature I'm looking forward to adding is passing custom data to shaders such as textures, z buffers, etc.
This should give us a ton more possibilities, like ambient occlusion that's done in the actual Post Processing shader from the game so it doesn't get on top of the UI, screen space reflections that actually care about how shiny the surface is supposed to be, tweaking lighting via custom light ramps and so on. Maybe some faked subsurface scattering could be achieved this way, but that would probably require a lot more features and this tool is still a huge WIP.
Page 4 of 4
Back to top