- Site Map >
- Modding and Creation >
- Sims 4 Creation >
- Modding Discussion >
- E0FF Compression Type?
- Site Map >
- Modding and Creation >
- Sims 4 Creation >
- Modding Discussion >
- E0FF Compression Type?
Replies: 7 (Who?), Viewed: 2420 times.
#1
16th Sep 2018 at 3:22 PM
Last edited by nixylvarie : 16th Sep 2018 at 4:00 PM.
Posts: 9
E0FF Compression Type?
I'm working on a python script which unpackages Sims 4 DBPF files, fixes mod conflicts in XML files, and repackages them.So far the unpackaging part of the script works perfectly, unless one of the files has a E0FF compression type.
Some output from an experiment with ClientDeltaBuild0.package:
Code:
>>> dbpf.debugFIndex(False, False, limitSize=10) [FIELD] [0:TYPE] [1:GRP ] [2:I_HI] [3:I_LO] [4:OFST] [5:SIZE] [6:REAL] [7:CMPR] [0] 101ACA81 00000000 00000000 00000000 11B30C3C 98790280 64620500 425A0100 [1] F3C4206B 00000000 0ED99D43 E91AE499 182B9B32 08400080 34710000 425A0100 [2] 5DE7D001 00000000 53273A2F 3AB5D421 AFD7E913 C8000080 54010000 425A0100 [3] 340FD101 01000100 A3F07916 F8F0423E 00000000 00000080 00000000 E0FF0100 <--- [4] 44504ABC 00000000 F266B4D3 18C65E9F 9AE11D3C 5E010080 F9040000 425A0100 [5] F3C4206B 00000000 F266B4D3 18C65E9F 065AF721 23540080 77A00000 425A0100 [6] 5DE7D001 00000000 5B4E7613 24316E81 86F5E613 F5000080 A8010000 425A0100 [7] 340FD101 00000100 07C1FD5B 2BEB6D31 47B66404 48020080 80040000 425A0100 [8] 7B3E3A05 00000000 9CC4597C 1A7F46A2 DD077214 73010080 C3020000 425A0100 [9] 49185A01 EB790B00 B6B547E6 DFE67BCE 003EFF05 7A600080 5DD40000 425A0100
What is this and what do I do with it?
EDIT: After looking more closely, I realized the file is empty. (*facepalm*) That explains the decompression error, but what does the E0FF mean?
Advertisement
#2
16th Sep 2018 at 6:18 PM
Posts: 2,671
Thanks: 62708 in 190 Posts
Deleted record. Here's the compression types from EA's binary template documentation for the package files:
Code:
string ReadCompressionType(UINT16& t) { switch(t) { case 0x0000: return "Uncompressed"; case 0xfffe: return "Streamable compression"; case 0xffff: return "Internal compression"; case 0xffe0: return "Deleted record"; case 0x5a42: return "ZLIB"; } return "Unknown"; }
#3
16th Sep 2018 at 7:01 PM
Last edited by nixylvarie : 16th Sep 2018 at 8:16 PM.
Posts: 9
Quote: Originally posted by scumbumbo
Deleted record. Here's the compression types from EA's binary template documentation for the package files:
Code:
string ReadCompressionType(UINT16& t) { switch(t) { case 0x0000: return "Uncompressed"; case 0xfffe: return "Streamable compression"; case 0xffff: return "Internal compression"; case 0xffe0: return "Deleted record"; case 0x5a42: return "ZLIB"; } return "Unknown"; } |
Oh, thanks! Streamable would be some kind of RLE, right? What's 0xFFFF?
Are there any other documentation files I can look at?
#4
16th Sep 2018 at 8:48 PM
Posts: 2,671
Thanks: 62708 in 190 Posts
Here's that binary template I quoted from earlier, if that could be of help. There was some stuff on a wiki somewhere (sorry, don't remember where) discussing the various compression types and I used that heavily in decompression for the internal compression. From what I can remember the streamable compression type is never used - don't remember if I implemented it in my programs but I think I skipped it. s4pi likely does handle that as it's more complete in what it's designed to do.
ETA - Regarding the streaming compression - I may also never run into that as I was mostly concerned with decompressing XML types of resources.
ETA - Regarding the streaming compression - I may also never run into that as I was mostly concerned with decompressing XML types of resources.
Attached files:
PackageTemplate.rar (1.4 KB, 22 downloads) - View custom content | ||
----------- --------- -------- ----- -------- ----- -------- ---- ..A.... 4223 1359 32% 04-09-14 14:15 EE8A1C03 PackageTemplate.bt ----------- --------- -------- ----- -------- ----- -------- ---- 4223 1359 32% 1 |
#5
16th Sep 2018 at 9:07 PM
Posts: 2,671
Thanks: 62708 in 190 Posts
Here's the VB code I used to handle the two compression types I handled, in a spoiler as it's fairly long.
Code:
Private Function DecompressData(data As Byte()) As Byte() If _CompressionType = &H5A42US Then ' ZLIB Compression Dim garbage(1) As Byte Dim oStream As New MemoryStream() Dim dStream As DeflateStream Dim iStream As New MemoryStream(data) ' Skip the first two bytes of the stream iStream.Read(garbage, 0, 2) dStream = New DeflateStream(iStream, CompressionMode.Decompress) dStream.CopyTo(oStream) Return oStream.ToArray ElseIf _CompressionType = &HFFFFUS Then ' Internal compression Return InternalDecompression(data) Else ' Unknown compression Throw New Exception("Unable to decompress compression type" & _CompressionType.ToString("X4")) Return Nothing End If End Function Private Function InternalDecompression(data() As Byte) As Byte() Dim udata(_SizeDecompressed - 1) As Byte Dim udata_idx As Integer = 0 Dim data_idx As Integer = 5 ' Skip 2 bytes of flags + 3 indicating length (use the package header for uncompressed size) Dim CompressionFormat As Byte = data(0) Dim ControlCode As Byte Dim Size As Integer Dim CopySize As Integer Dim CopyOffset As Integer ' Skip an extra byte if compression format flag has high-bit set (indicating 4 byte length) If CompressionFormat And &H80 Then data_idx += 1 End If Do ControlCode = data(data_idx) data_idx += 1 If ControlCode <= &H7F Then Size = ControlCode And &H3 CopySize = ((ControlCode And &H1C) / 4) + 3 CopyOffset = ((ControlCode And &H60) * 8) + data(data_idx) data_idx += 1 Array.Copy(data, data_idx, udata, udata_idx, Size) data_idx += Size udata_idx += Size For I = 0 To CopySize - 1 udata(udata_idx + I) = udata((udata_idx + I) - CopyOffset - 1) Next udata_idx += CopySize ElseIf ControlCode <= &HBF Then Size = (data(data_idx) And &HC0) / 64 CopySize = (ControlCode And &H3F) + 4 CopyOffset = ((data(data_idx) And &H3F) * 256) + data(data_idx + 1) data_idx += 2 Array.Copy(data, data_idx, udata, udata_idx, Size) data_idx += Size udata_idx += Size For I = 0 To CopySize - 1 udata(udata_idx + I) = udata((udata_idx + I) - CopyOffset - 1) Next udata_idx += CopySize ElseIf ControlCode <= &HDF Then Size = ControlCode And &H3 CopySize = ((ControlCode And &HC) * 64) + data(data_idx + 2) + 5 CopyOffset = ((ControlCode And &H10) * 4096) + (data(data_idx) * 256) + data(data_idx + 1) data_idx += 3 Array.Copy(data, data_idx, udata, udata_idx, Size) data_idx += Size udata_idx += Size For I = 0 To CopySize - 1 udata(udata_idx + I) = udata((udata_idx + I) - CopyOffset - 1) Next udata_idx += CopySize ElseIf ControlCode <= &HFB Then Size = ((ControlCode And &H1F) * 4) + 4 Array.Copy(data, data_idx, udata, udata_idx, Size) data_idx += Size udata_idx += Size Else Size = ControlCode And &H3 If Size > 0 Then Array.Copy(data, data_idx, udata, udata_idx, Size) data_idx += Size udata_idx += Size End If End If Loop Until ControlCode >= &HFC Return udata End Function
#6
16th Sep 2018 at 9:10 PM
Posts: 2,671
Thanks: 62708 in 190 Posts
And the document I used to figure out that internal decompression was right on this site
http://modthesims.info/wiki.php?tit...BPF/Compression
http://modthesims.info/wiki.php?tit...BPF/Compression
#7
16th Sep 2018 at 10:24 PM
Posts: 9
Quote: Originally posted by scumbumbo
And the document I used to figure out that internal decompression was right on this site http://modthesims.info/wiki.php?tit...BPF/Compression |
What's with the DDS files? I unpackaged and decompressed several but couldn't view them. I saw another thread where they mentioned skipping a two-byte header before decompressing a .DDS...don't remember where that was. It was a very old thread. It's not like I'll need to use them - I'm just using the XML files - but it'd be nice to have a sense of completeness.
Thank you for all your help, by the way. I'm enjoying this project so much I forgot about the whole actually making mods part. Messing around with all these file formats and encodings hurts my head, but I'm enjoying the challenge. I learned a lot about compression in the past 48 hours!
#8
17th Sep 2018 at 5:47 AM
Posts: 2,671
Thanks: 62708 in 190 Posts
Quote: Originally posted by nixylvarie
What's with the DDS files? |
The most I do with DDS files with the projects I've done is to compare hashes of the resource themselves (in the Package Conflict Detector), I never actually decompress them - if the two compressed resources hash to the same result then I'm happy. The only reason I had to work out the compression for the zlib and internal compression stuff was in the process of writing the XML Extractor and read the string tables.
Who Posted
|