Hi there! You are currently browsing as a guest. Why not create an account? Then you get less ads, can thank creators, post feedback, keep a list of your favourites, and more!
Replies: 30 (Who?), Viewed: 15079 times.
Page 1 of 2
Quick Reply
Search this Thread
Kuree
Lab Assistant
Original Poster
#1 Old 9th Jul 2014 at 5:19 AM
Default TS4 Compression Algorithm
I'm not sure where to post this discussion, but I want to discuss the algorithm as the new compression bothers me so much.

The new DBPF uses 2.1 version and keeps the similar index format as TS3's. Nevertheless, it introduces a new compression algorithm which I have no clue so far. As far as I know, if the compression tag is 0xFFFF, it uses the previous DBPF compression algorithm(even with 2.1 version number). However, if the compression tag is 0x425A, the compressed file shows now sign of RLE pattern.

What surprises me is that 0x425A is magic number for BZ2 format. Not sure if they use similar algorithm here. Anyway suggestion is appreciated.
Advertisement
lewisb40
Mad Poster
#2 Old 9th Jul 2014 at 6:01 AM
I cannot help you but wanted to let you know how much I appreciate what you are doing. Wishing you much success and hopefully the ones that can help will be here soon. *HUGS* :lovestruc
Resident member of The Receptacle Refugees
Let's help fund mammograms for everyone. If you want to help, Click To Give @ The Breast Cancer Site Your click is free. Thank you.
Bubble Pop
Test Subject
#3 Old 9th Jul 2014 at 7:22 AM
So you're trying to decompress BZ2 but it's showing RLE even though the file is BZ2.

Was now a typo or did you mean no sign of RLE pattern.

Sounds to me like a layer.
Tashiketh
Warrior Gryphon
site owner
#4 Old 9th Jul 2014 at 9:08 AM
Is the compression tag really 0xFFFF? When I was looking at the index, I noticed that the field order is switched, which means there is a switched compressedflag / flags. I suspect that the flags itself might be the compression flags since it's only turned on for compressed files. I can post a screenshot of the index list later on. But yes, when it comes to actually reading the files, my compression algorithm also failed but thats becuase it's expecting the old style header.

I'm going to tackle this today, since, without it we can't get to the internals.
Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
Tashiketh
Warrior Gryphon
site owner
#5 Old 9th Jul 2014 at 9:09 AM
Just as a heads up - lets keep this thread ON TOPIC for discussion of the compression algorithm. Please try and keep gif replies and what not to, well, non existent. Thanks.
Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
Tashiketh
Warrior Gryphon
site owner
#6 Old 9th Jul 2014 at 10:17 AM Last edited by Delphy : 9th Jul 2014 at 11:16 AM.
The first 2 bytes of all the chunks are actually 0xDA78. Where are you seeing 0x425A?

btw in the index there is a flags value, like so:



Seems to be switched on and off for various types. Edit: Okay yes, the 23196 in my list is "BZ". Plus the first 2 bytes of each compressed chunk at DA78 which could be the compression type for Zlib.
Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
Tashiketh
Warrior Gryphon
site owner
#7 Old 9th Jul 2014 at 1:54 PM
So turns out... it's standard C# deflate.

The following will read the compressed chunks: (This is VERY quick and dirty as I have just been experimenting with it)

Code: 
                    if (this.dbpf.smallHeader.MajorVersion == 2 && this.dbpf.smallHeader.MinorVersion == 1)
                    {
                        data = new byte[entry.CompressedSize];
                        this.Stream.Read(data, 0, data.Length);
                        MemoryStream tmpData = new MemoryStream(data);

                        tmpData.ReadByte();
                        tmpData.ReadByte();

                        System.IO.Compression.DeflateStream decompress = new System.IO.Compression.DeflateStream(tmpData, System.IO.Compression.CompressionMode.Decompress, true);

                        MemoryStream decompressed = new MemoryStream();
                        MadScience.StreamHelpers.CopyStream(decompress, decompressed);
                        decompressed.Seek(0, SeekOrigin.Begin);

                        data = new byte[entry.DecompressedSize];
                        data = decompressed.ToArray();

                    }


The key was actually skipping the first 2 bytes of the compressed stream (with their 0xDA78). Then the resulting decompressed size matches what is in the index.

To prove this works, here's a DDS file:

Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
Kuree
Lab Assistant
Original Poster
#8 Old 9th Jul 2014 at 2:11 PM
COOL! I just got up. So we can use standard deflate stream class to decompress it!
Kuree
Lab Assistant
Original Poster
#9 Old 9th Jul 2014 at 3:04 PM Last edited by Kuree : 10th Jul 2014 at 4:15 AM.
For those of you who want to look at the internal files, here is tool I wrote years ago (for fun and with GUI) and I modified it to support TS4 format. It has built-in hex view. It's still a little buggy btw .

Edit:

Update the import method. Though it's useless right now...

I've modified peter's S3PE and s3pi and made them into s4pe.
Here is the github repo
And here is the s4pe demo download link.
More commits
Attached files:
File Type: 7z  PE.7z (41.2 KB, 101 downloads) - View custom content
Description: Update the import function. Though it's useless right now.
Tashiketh
Warrior Gryphon
site owner
#10 Old 9th Jul 2014 at 3:06 PM Last edited by Delphy : 9th Jul 2014 at 3:32 PM.
Known file types so far:

http://simswiki.info/wiki.php?title...PackedFileTypes

List will be updated as we find more.
Story books are full of fairy tales, of Kings and Queens, and the bluest skies.
eskie227
One Minute Ninja'd
#11 Old 9th Jul 2014 at 4:32 PM
Quote: Originally posted by Delphy
Known file types so far:

http://simswiki.info/wiki.php?title...PackedFileTypes

List will be updated as we find more.


Yay! We still have xml files to play with. Emotions be gone...........................................
ellesmelly
Test Subject
#12 Old 10th Jul 2014 at 2:03 PM
wow you are all so smart i didnt understand any of that even though im am doing programming and ICT at school :/ i cant wait for all of the wonderful things that modders will bring to the sims 4 community :D
Volvenom
dodgy builder
#13 Old 10th Jul 2014 at 7:58 PM
it's using dds files too
Boby-LM
Test Subject
#14 Old 13th Jul 2014 at 12:31 AM
I didn't understand any thing of what you just said but I just want to thank every modder who take the time to do tools and mod, because CC is one of my favorite thing in the sims <3 thank y'all <3
DigitalSympathies
Mad Poster
#15 Old 13th Jul 2014 at 1:01 AM
Are the XMLs in the same format as previous, as in they're rather simple to edit (I'd like to take my hand at modding TS4, and TS3 was rather easy after I got the hang of it), or are things a bit swapped around? Would it be possible to translate similar mods between the two games easily (not copypaste of course but similar properties etc)?

Sorry if I sound like a dumbass, I'm running on 2 hours of sleep.

Angie/DS | Baby Sterling - 24/2/2014
This account is mostly used by my sons to download CC now, if you see me active, it's probably just them!
Shimrod101
Mad Poster
#16 Old 13th Jul 2014 at 11:23 AM
@DigitalSympathies

There is no GameplayData file in the CaS Demo, so it isn't possible to really say at this point. There is one standard tuning XML in the Fullbuild file which deals with selection priorities of fonts in the demo, which is written the same way as one would write XML tuning in Sims3. The JAZZ files in the demo are also written in XML, instead of the other language from Sims3; so this is good news.

If the game has similar XML tuning and table-based XML (like the Books list and similar lists) and ITUN, then it will be possible to compare with a Sims3 mod and determine a possible alteration to the relevant Sims4 bit, yes. I did this all the time when making Sims Medieval mods, read through the Sims 3 stuff and applied it to the Medieval XML's I was using.
MedievalMods and Sims3mods: Dive Cave Reset Fix, Resort Revamp, Industrial Oven Revamp, Will O' Wisp fix, UI Sounds Disabled, No Cars, Gnome Family Planner, Townies Out on the Town, No Martial Arts Clothes, Fast Skilling, etc. http://simsasylum.com/tfm/
Fuzzlechan
Test Subject
#17 Old 13th Jul 2014 at 2:47 PM
Well, now I REALLY want to know how all of this works. I'm a software engineering student and none of that made any sense. Does anyone know what language Sims 4 is written in? I keep hearing either C++ or C# for Sims 3.
Inge Jones
One horse disagreer of the Apocalypse
#18 Old 13th Jul 2014 at 3:02 PM
Quote: Originally posted by Fuzzlechan
Well, now I REALLY want to know how all of this works. I'm a software engineering student and none of that made any sense. Does anyone know what language Sims 4 is written in? I keep hearing either C++ or C# for Sims 3.


You are allowed to start new threads for the technical things, rather than try to fit everything under the topic of Compression Algorithm
"You can do refraction by raymarching through the depth buffer" (c. Reddeyfish 2017)
Nysha
Née whiterider
retired moderator
#19 Old 13th Jul 2014 at 8:54 PM
At some point we will make a TS4 modding discussion forum, once we've branched our understanding enough to need multiple threads on different topics. I'll leave it to the coders to give me a nudge to do that, when they start feeling that it'd be beneficial.
What I lack in decorum, I make up for with an absence of tact.
DigitalSympathies
Mad Poster
#20 Old 13th Jul 2014 at 9:46 PM
Quote: Originally posted by Shimrod101
@DigitalSympathies

There is no GameplayData file in the CaS Demo, so it isn't possible to really say at this point. There is one standard tuning XML in the Fullbuild file which deals with selection priorities of fonts in the demo, which is written the same way as one would write XML tuning in Sims3. The JAZZ files in the demo are also written in XML, instead of the other language from Sims3; so this is good news.

If the game has similar XML tuning and table-based XML (like the Books list and similar lists) and ITUN, then it will be possible to compare with a Sims3 mod and determine a possible alteration to the relevant Sims4 bit, yes. I did this all the time when making Sims Medieval mods, read through the Sims 3 stuff and applied it to the Medieval XML's I was using.


This is exciting and rather comforting because that way it'd make a lot of popular XML tuning mods/scripts that much easier to translate. Harder bills, increased accidental death chance, etc, yes? Again, day one modding is a bit of a fantasy of mine.

Angie/DS | Baby Sterling - 24/2/2014
This account is mostly used by my sons to download CC now, if you see me active, it's probably just them!
Srikandi
Eminence Grise
#21 Old 14th Jul 2014 at 3:46 AM
Quote: Originally posted by Fuzzlechan
Well, now I REALLY want to know how all of this works. I'm a software engineering student and none of that made any sense. Does anyone know what language Sims 4 is written in? I keep hearing either C++ or C# for Sims 3.


The Sims 3 core (and presumably 4's too) are written in C++, like almost all commercial software these days. Sims 3 however used C# for scripts (the bits of code, accessible to modders, that control sim behavior). Sims 4, according to an old tweet from SimGuruGraham, uses Python for that, so script modders will have to retune. If you want to bone up on a language for modding, start on Python
Megido
Forum Resident
#22 Old 14th Jul 2014 at 4:54 AM
Bollocks. :{ I know C# but never got the chance to try out scripting with Sims 3.

I guess it's as good a time as any to learn Python.
Kuree
Lab Assistant
Original Poster
#23 Old 14th Jul 2014 at 6:31 PM
So I made this mod in order to test S4PE's functionality. First mod in TS4 ever? LOL
It can remove all the annoying opening intro in the game. Download link can be found at http://www.den.simlogical.com/denfo...hp?topic=3129.0
Boby-LM
Test Subject
#24 Old 14th Jul 2014 at 7:26 PM
Quote: Originally posted by Kuree
So I made this mod in order to test S4PE's functionality. First mod in TS4 ever? LOL
It can remove all the annoying opening intro in the game. Download link can be found at http://www.den.simlogical.com/denfo...hp?topic=3129.0


OMG already a mod <3 thank you :lovestruc
tedw
Field Researcher
#25 Old 15th Jul 2014 at 1:42 PM
Kuree, are you sure the import function is useless? Because I was able to alter a skin file and import it as a replacement.

I just wanted to test it out, so I drew a V shape on the forehead of the blue skin. It's not meant to look good.

Page 1 of 2
Back to top