Replies: 91 (Who?), Viewed: 19694 times.
Page 2 of 4
Mad Poster
#26 Old 1st Oct 2019 at 9:50 PM
Well for a start it's good that that value can be increased in TS4. TS3's indoor lighting shaders are one of the game's better qualities, I think, and TS4 doesn't really live up to it in some ways (though the natural light shadows are quite cool).

TS3 could really use some higher values as well. Exterior shadows can easily be modified for higher resolutions, mine are at 8K and that means virtually perfect exterior shadows, but indoors it can be quite blurry.
A short wishlist:

Higher interior shadow level of detail
Higher cubemap resolutions
Extended ambient occlusion functionality
Basic support for global illumination with better radiance from objects

And perhaps entirely unrelated to the shaders: more cloud layers with yet more realistic illumination and perhaps TS4-style world backdrops (notice how clouds are completely different between TS3 and TS4, with both having obvious weaknesses).

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Top Secret Researcher
#27 Old 2nd Oct 2019 at 8:21 PM
The only things I can see changing if we ever crack open the shaders would be the first two from your list. There probably is a lighting/shader property somewhere that allows for lightmap and cube map resolution change.
Rest can probably be achieved with external programs... but they are not in the scope of game's engine.
Mad Poster
Original Poster
#28 Old 3rd Oct 2019 at 5:07 AM
Quote:
Originally Posted by AGuyCalledPi
Well for a start it's good that that value can be increased in TS4. TS3's indoor lighting shaders are one of the game's better qualities, I think, and TS4 doesn't really live up to it in some ways (though the natural light shadows are quite cool).

TS3 could really use some higher values as well. Exterior shadows can easily be modified for higher resolutions, mine are at 8K and that means virtually perfect exterior shadows, but indoors it can be quite blurry.
A short wishlist:

Higher interior shadow level of detail
Higher cubemap resolutions
Extended ambient occlusion functionality
Basic support for global illumination with better radiance from objects


Personally I'd just like extended shadow draw distances and multi-floor lighting to be fixed. Probably the two main things that will have the largest impact on improving the game's appearance. Higher shadow draw distances will keep the worlds from looking flat, while multi-floor lighting just plain makes exterior and interior spaces better looking, and eliminates the need for invisible lights. These are things that should be given for any modern game engine.

On that note- I wonder if if shaders could be edited, an approximation of TS4's natural light shadows could be done by simply making the light cast by windows more crisp? One other thing worth looking into is to see why fences aren't affected by lights at all.

Quote:
And perhaps entirely unrelated to the shaders: more cloud layers with yet more realistic illumination and perhaps TS4-style world backdrops (notice how clouds are completely different between TS3 and TS4, with both having obvious weaknesses).


TS4-style backdrops are really simple- they're just flat planes objects with a scrolling texture. This is entirely possible in TS3 without editing the shaders.
Mad Poster
#29 Old 4th Oct 2019 at 3:00 PM
That's what I said - clouds are essentially a rotating, spherical backdrop in TS4. TS3 only has top-scrolling clouds which means that the horizon can sometimes look awfully plain.

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Virtual gardener
staff: administrator
#30 Old 4th Oct 2019 at 4:08 PM Last edited by Lyralei : 4th Oct 2019 at 4:28 PM.
I feel like whenever we have the talk about "Shaders" people actually don't necessarily understand the difference between "Materials" or "Shaders"

Here's a great video on how shaders are made:


And materials, are mainly texture-based. They both need each other though to create the final results on a mesh
(See: https://forums.cgsociety.org/t/shad...e-same/927326/6)

Now, coming to your question. Shaders are, unfortunately, all engine based. Given how TS3 has been made with HLSL shaders (and I think GLSL for mac copies IIRC), you can't just go in and edit it, no tool can do it. You'll need to have the development build of TS3 itself to adjust the code of shaders. OR Do what this dude did with another DirectX9 game: https://forums.taleworlds.com/index.php?topic=296795.0

EDIT: Correction, TS3 uses OpenGL But does indeed have HLSL Shaders

But as you can see already, it requires 3 things. 1. C++ knowledge. 2. Knowing how the game engine works. 3. Being good at maths and algebra.

However, most people actually don't really know how much EA's shaders can already do by changing values in the MATD file or even the LITE file. For the LITE I always like to give people this example because even though it's a fixed tutorial, it covers so much on the main bits of, well, ts3 lighting http://poccisims3.blogspot.com/2013...-lightings.html
Top Secret Researcher
#31 Old 6th Oct 2019 at 11:09 PM
When I refer to a shader I mean, the actual code that makes the material look like a distinct material in the game. All the material options that show up I consider shaders for example specularity property is probably controlled by a shader. Or fresnel on objects (yes sims 3 has actual fresnel property). Glass 'material' as well is actually just it's own shader. The link you've sent actually does clear it up nicely for people.

Idk if lightmaps and shadowmaps should be considered a shader instance but I'm pretty sure they are engine hardcoded at this point. And ofc if we indeed do not have EA's dev version of the game... there's nothing we can do other than, well, reverse engineer it... which is, from what I can see, not going to happen in long long stream of years to come. A shame really. Sims 3's graphical engine, albeit very old and out of date at this point, is still quite powerful (given by the mere fact that sims 4 is also using it).
Mad Poster
#32 Old 7th Oct 2019 at 12:10 AM
Here's something to put on the wishlist - fix the weird transparency issues that occur when Sim strands of hair are rendered in front of glass. It's weird and ugly, and clearly some shader-related issue. But like you said, big if.

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Top Secret Researcher
#33 Old 14th Oct 2019 at 11:41 PM
I've recently received an information that weird pool water refraction distortion glitch on screen edges when camera is near the water surface happens just because EA has not used normal map normals for the water mesh but because they used mesh normals with normal map applied. That's probably just one line of code, but because it's all inaccessible... we can forget about fixing it.

God EA can fuck up so bad sometimes.
Top Secret Researcher
#34 Old 23rd Dec 2019 at 4:54 PM Last edited by mixa97sr : 23rd Dec 2019 at 5:12 PM.
I've opened the precomp file in a hex editor, I'm looking at it right now and... it might actually be a file with shader definitions. It is a file where shader coding is stored. From what I see it has a lot of shader data going for it and it's structured like this:

- Header with file type and probably also libraries used
- Vertex and Pixel shader data for each shader
- Value Type definition (int, float3...) for each property entry per shader, for all shaders
- Some form of parameter list but I'm not sure if it's actual parameters described previously or something else. From strings it goes like: TECH, PARM, PARM, PASS
- Final list of shaders described in the file

If this is trully the same thing as sims 4's precomp file (which I just compared and they really are 90% the same) then, it should be able to change the existing shaders and add new ones. Because few people managed to actually add new shaders and edit existing ones using the .BT file gurus provided them with.

If I am correct is, how you would do this is you would run binary remplate (.BT) file over shader precomp file and you would get a version that's readable and editable. The thing with sims 4 is that you would be able to have that decompressed file in mods folder work. For sims 3 that is most likely not the case meaning that you'd probably have to compress the file again... which I don't think it's possible.

EDIT: I read Lyra's explanation and I guess she might be right though... that would mean that people who got this to work with sims 4's precomp shader did not actually get it to work? I'm currently in the state of confusion.
I guess I'm hoping too much
Virtual gardener
staff: administrator
#35 Old 23rd Dec 2019 at 9:52 PM
Yeah so basically, the mentions (as far as I've been able to figure out) Are merely like "This shader uses use of Pixel shader data and vertex shader data, etc." Which would define, "Hey I am this shader and I am using this amount of specular, colour at this and that vertice". That is, as far as my knowledge of shaders go. See: https://www.tutorialspoint.com/dire...ixel_shader.htm

But removing the binary bits of the hex code and re-save it after the changes is actually pretty easy to program However, this info needs to be the SAME as the info from the engine.

Quote:
So when you're compiling your code for the game, you obviously want the game to be able to call its textures, shaders, rendering settings, etc. Now I'm not sure if that's common practise perse but in the TS3W.exe as well as the TSLHost.dll, it holds pretty much all the 'directions' of where the game should get its details. They're actually listed like a map structure, which was obviously needed before the game was actually being compiled for release.


When I referred to this, I basically meant that the actual shader settings are in the TS3W.exe and TSLHost.dll (Although the TSLHost.dll is I think just to contact EA's server with the whole sharing gifts in-game with your friends). The Shader data from there uses variables and references that are defined in the Precomp file we're talking about here. I assume this is because of saving loading time and getting easier access to adding more Variables rather than re-compiling the engine over and over again. Hence the mention of "Floats3, etc"

YOu could actually check out how they did it with Minecraft's shaderlab. Minecraft was supposed to be non-crackable regarding shaders, but I think they reversed engineered it in some ways. I haven't actually checked it out though.
Mad Poster
#36 Old 24th Dec 2019 at 1:25 AM
Really wish I knew more about the issues at hand here. Minecraft, running on Java, should probably be easier to reverse engineer than TS3's more proprietary architecture....but the problem with compiling and decompiling software is that there is NOTHING to facilitate it on the end user's end. So to what extent did EA use proprietary software? Maybe I'm just being dumb, it's just regular C+ isn't it?

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Virtual gardener
staff: administrator
#37 Old 27th Dec 2019 at 1:19 PM
So, I took a look into the engine with a hex editor, and it seems pretty straightforward. What I know is that the Engine is made with C++.

Some of you might recognise the mashalAs stuff whenever you're working on things. Here's an example:

Code:
[DllImport("Sims3Common.dll")]
[return: MarshalAs(UnmanagedType.I1)]
public static extern bool World_AddObjectToObjectManagersImpl(ulong scriptHandle, ulong groupId, Vector3 position, int level, Vector3 createFacing, uint hiddenFlags);

Now, marshalAs is primarily used to import C++ dll libraries (IIRC). C++ is also usually the preferred language for making engines, because it's the almost closest language to computer language. TS3's engine is also custom, so it makes sense why they'd use C++.

Now, in the engine, if we were to look at the code I just copied in my post, it would loook like this:

Code:
ScriptCore.World::World_AddObjectToObjectManagersImpl


Which makes a lot of sense!  But if we were to do this with shaders, I think we'd have to make the communication work through editing hex code, so I'd figure making an application that adds the bytes to the .exe file would be better if you were to want end-users (So in our case people who download CC) to use it and safetly install such a thing.

But, of course, currently i'm talking about about functions in the engine And not shader editing :p. But I've got my reasons!

Some useful stuffz:
http://timjones.io/blog/archive/201...-cbuffers-using
https://modthesims.info/wiki.php?ti..._Modding_Basics

Personally, I think i've got myself a lead as to how to add custom materials in a sense like this: https://docs.unity3d.com/Manual/SL-...erExamples.html

But that yet needs to be seen . If I get that working, i'll make a tutorial for it!
Mad Poster
#38 Old 28th Dec 2019 at 5:31 AM
I like where this is going.

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Virtual gardener
staff: administrator
#39 Old 31st Dec 2019 at 10:29 AM
So, @mixa97sr was pretty right! Except for one thing here, which is whether the ts3 precomp file is the same as ts4, and even I was fooled into thinking this, so no worries.

TS4 only has functions calling the shaders. That's it. Ts3 actually has genuine functions as to how the shader should act, be, do, etc. Hence why you'd see references like "Tech" and "Pass". So those are here for a reason Else we'd have been able to find a JSON file or something that TS4 has to be able to customise it without cracking open that file over and over again and recompile the engine every single time, given how they are loading in a list. But that's TS4, not TS3

There are about 604 shaders in the 1.67 patched game. Each have their own "chunk" So to speak. They're string-wise, only mentioned twice. (For example, search for PreviewWallsAndFloors, and you see what I mean). Now, you of course got the FNV32 string version of the shader's name as well and that one is mentioned plenty of times if you were to search it. The first mention of say PreviewWallsAndFloors, in the SSET for the shader, happens to have some garbage bytes and a list of some sort. The end bit of the bytes chunk, however, has the naming of the values (which I presume you can customise to something else? As long as it's done in this file) It contains the default settings of a shader param, but also a "Min" and "Max" value. For textures this is obviously a bit different.

Later on, if we use the FNV32 Hash of PreviewWallsAndFloors, The FNV32 hash is mainly used as an "ID" for the "TECH" Which makes a lot of sense: https://docs.microsoft.com/en-us/wi...ques-and-passes

Just like how we saw that SSET has it's own "Shader" chunk of bytes, so do the functions, which is with a tag called "SBLK" Which stands for "Shader block". See this as a wrapper for the shader functions

Later at the bottom, you've got ever shader + shader data fields, which is probably for initialising everything.... Because, EA :p

So the file is actually simple af, it's just that there are 604 shaders which makes it super big and hard to read. But this is as far as Tash and me  got to reading the file. We're still working on it though!
Mad Poster
#40 Old 31st Dec 2019 at 4:32 PM
Lyra you absolute gem, I have no idea what exactly this all means but it's starting to get dangerously close to a level I can comprehend. Looks like 2020 will be a year of TS3 being completely destroyed in new and exciting ways. HEX editing breaks my brain.....functions and variables and wrappers and functions, I can understand.

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Lab Assistant
#41 Old 2nd Jan 2020 at 2:10 PM Last edited by SmugTomato : 2nd Jan 2020 at 4:29 PM.
I've extracted all the shader names with the data in front of them into a text file in case anyone wants to try and figure out what the data in front of them means.
I've reordered them from [DWORD unknown1, DWORD unknown2, DWORD stringLength, String shaderName] to [shaderName, DWORD unknown1, DWORD unknown2].
unknown2 is 0 in all entries.

All values were read in Little-Endian but were written to the file flipped to big-endian. (Thanks java)
Download - please read all instructions before downloading any files!
File Type: txt shadernames.txt (52.7 KB, 49 downloads)
Description: All values were read in Little Endian
Virtual gardener
staff: administrator
#42 Old 3rd Jan 2020 at 10:13 AM
Ah, nice SmugTomato! :D

I take it this is from the mKeyNameTag bit right? (Aka the bit at the bottom, with alll the strings)? I noticed that has quite a mixture of "Shaders" and "Shader params", given how they're basically strings that are being used for reading purposes.

Anyways, for the programmers out here, Tash and me made a Byte-reading template (Which was actually done quite easily by using ts4's bt file. Hence why it says that at the start still :p) We've edited in such a way that you can now read it for ts3. One note though is that for TS3, the mheapsize is being used for the shaders, whereas for TS4 they are... well, ignored. So while it would still be possible to edit it by reading the offsets, just so that you guys are aware of this.

So far, i'm still trying to figure out the linkage of bits. All I need to figure out now is just what the mplatform entries do, and how the mVertexShaderParams as well as the mPixelShaderParams communicates with the game/package files (Or well, more specifically MLOD/MODLs). So you can use that as a reference.

I used 010 editor to run the template, so make sure you've got a program that can run templates (For 010 editor, see: https://www.sweetscape.com/010editor/manual/Running.htm )

I've also included my research on what I got, which is a "Follow the DiffuseMap through the Phong shader!". mPlatform stuff however isn't included yet.
Download - please read all instructions before downloading any files!
File Type: txt ShaderInfo.txt (6.4 KB, 40 downloads)
Description: My research on the topic, way too long for a post so here you go!
File Type: zip Sims3_Shaders_Win32_Template.precomp.zip (2.9 KB, 24 downloads)
Description: Template that you can run with 010 editor or any bytes editor.
Lab Assistant
#43 Old 3rd Jan 2020 at 10:48 AM Last edited by SmugTomato : 3rd Jan 2020 at 11:20 AM.
Looks like my suspicions might be right about the 0 bytes just being part of a 64bit int with only the lower 32bits being used looking at your txt file, those poor wasted 1.2KB
I was wondering why my generated FNV hashes weren't matching with what I suspected were hashes, turns out my implementation is just faulty, might be because of java's lack of unsigned integers. Should have just checked them it in S3PE to begin with.

*Edit*
Looks like I missed the bold text on the wiki stating that the string should be converted to lowercase first before hashing. Now it makes sense.
Virtual gardener
staff: administrator
#44 Old 3rd Jan 2020 at 11:58 AM
That's allright! Hopefully the template makes it easier to see stuff though. Personally never worked with java before, but I've heard Python can do the job pretty well too regarding doing things on byte level. But yeah when it comes to those things, you really need to be super precise, which is annoying lol!

Currently been working on a small program that can disassemble the "mByteCode" in the VSHD/PSHD section. (see template code) With D3DCDissasembleShader. Mainly because... curiousity and seeing how far things can be edited
Lab Assistant
#45 Old 6th Jan 2020 at 6:22 PM
I'm excited for this! Hope for the best!
Warrior Gryphon
site owner
#46 Old 14th Jan 2020 at 3:21 PM
In case anybody is interested, I did a bit of work to figure out the SSET stuff before we found the TS4 template. :P  So it's fairly easy to link things together:

Code:
SSETRabbitHoleHighDetail
Num QWords:     3
Length of list in dwords:       104
Total size of list is: 416
Num OPTs:       10
        D9FD4034 OPT_SHINY                                      1       0       2
        F2477C6F OPT_SWITCH_SHADOWMAP_TYPE                      2       0       3
        136B6D20 OPT_ALPHATESTENABLED                           6       0       2
        96DE15F6 OPT_SKINNING                                   12      0       2
        F8EF773B OPT_FADE_ENABLED                               24      0       2
        E09BAB91 MAP_NORMAL                                     48      0       2
        8534219B MAP_SPECULAR                                   96      0       2
        9F2A6F85 MAP_EMISSION                                   192     0       2
        7CB9E69F OPT_THICKSNOW                                  384     0       3
        DEF23ADB OPT_FROST                                      1152    0       2
num Shader Fields:      20
RabbitHoleHighDetail
RabbitHole High Detail
Data size length: 432
At position: 1322
diffuse                         0       8       0       100     3       1       1       1
DiffuseUVScale                  20      36      0       100     2       1       1
specular                        44      56      0       100     3       1       1       1
shininess                       68      80      0       100     1       10
SpecStyle                       84      96      0       100     1       2
FresnelOffset                   100     116     0       100     1       0.20000000298023
NormalUVScale                   120     136     0       100     2       1       1
NormalMapScale                  144     160     0       100     1       1
NormalMapUVChannel              164     184     0       100     1       0
DiffuseMapUVChannel             188     208     0       100     1       0
AmbientMapUVChannel             212     232     0       100     1       1
EmissionMapUVChannel            236     260     0       100     1       0
AlphaMaskThreshold              264     284     0       100     1       0
EmissiveLightMultiplier         288     312     0       100     1       1
EmissiveBloomMultiplier         316     340     0       100     1       1
AmbientOcclusionMap             344     364     0       100     7       0
DiffuseMap                      368     380     0       100     7       0
NormalMap                       384     396     0       100     7       0
EmissionMap                     400     412     0       100     7       0
SpecularMap                     416     428     0       100     7       0
Something that the template doesn't do is to link things together, so that's important to do too.

Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
Lab Assistant
#47 Old 18th Jan 2020 at 8:22 AM
So we found out that the ghost state is shaders, fairy wings, the visual effect that is with all the effects in the file is Sims3effects, but what are visual overrides, they are also in the same place as the shaders?
Virtual gardener
staff: administrator
#48 Old 25th Jan 2020 at 10:34 AM
Quote:
Originally Posted by whiteman-Dara
So we found out that the ghost state is shaders, fairy wings, the visual effect that is with all the effects in the file is Sims3effects, but what are visual overrides, they are also in the same place as the shaders?


Visual overrides are actually overrides done on effects . So, a great example could be "no more confetti!". Now, some effects can be changed in colours through creating a script mod out of it. Or, you could use this tool and method: http://www.modthesims.info/showthread.php?t=450194

However, if we take the confetti visual effect as an example (and this goes quite far from what you asked), this does need a mesh and shader of course. Else, the confetti will not have 1. transparency support. 2. the way the light is defined on the confetti. So, how EA fixed this was by adding a "Particle" shader. This is basically the shader for anything confetti-like or whatever "falls" as an effect

Hope this clears it a bit! I'd recommend though if you want to know more about visual effects and editing them to start a new thread. mainly because this is about cracking open the Shader pre-comp file, and not editing visual effect.
Mad Poster
#49 Old 25th Jan 2020 at 1:17 PM
Interesting that you should mention transparency support, because that reminds me of hair.
Sim Hair does not cast shadows. Hair also has transparency issues when rendered against certain other shaders like the glass ones. Could you explain, by the way in which shaders are used to render Sim hair, why these issues occur?

insert signature here
( Join my dumb Discord server if you're into the whole procrastination thing. But like, maybe tomorrow. )
Lab Assistant
#50 Old 25th Jan 2020 at 6:37 PM
Quote:
Originally Posted by AGuyCalledPi
Interesting that you should mention transparency support, because that reminds me of hair.
Sim Hair does not cast shadows. Hair also has transparency issues when rendered against certain other shaders like the glass ones. Could you explain, by the way in which shaders are used to render Sim hair, why these issues occur?


That happens in most rendering engines, alpha calculations and shadow casting are both expensive rendering techniques. There's in general 2 types of alpha, smooth alpha with a gradient from fully opaque to fully transparent, or the vastly cheaper options, visible or not visible with a certain cutoff point in the alpha value.

The second option usually does cast shadows in rendering engines as it's not much more expensive to calculate than a fully opaque object, you either you render it to the shadow map or you don't. With a gradient alpha you'd have to calculate the opacity of the shadow for every pixel too.
Page 2 of 4
Back to top