121 replies to this topic

[AndalayBay]

A Skyrim plugin is composed of a plugin header which contains a Record with the plugin info and a series of Groups. A Group has some info fields and a set of of sub-Groups or Records. A Record contains some info fields and a set of Fields. A Field is the lowest type of element in the tree, and contains data in a variety of formats depending on context. Both the Group and Record have a set of flags which control their use - the Group flags primarily indicate what type of Group it is while the Record flags are a combination of application indicators and modifiers indicating how the Fields are to be processed. One flag, 0x00040000, indicates that the Field list is compressed. In this case the "data" area contains a four-byte "size of decompressed data" and the compressed stream. We discovered that sometimes the actual decompressed data is longer than the size indicated. The extra data do not appear to be more Fields. We had thought that an undocumented flag - 0x02000000 - was a warning of this but that turned out to be unreliable. TESVGecko will retain the entire decompressed data stream even though we don't know what it is for.

If a plugin utility drops this extra data, it is gone from the plugin. Opening the plugin in the Creation Kit will not necessarily restore this information. Also, if the compression flag is set, then the data must be compressed. Having the compression flag set but providing uncompressed data will cause issues, likely crashing the game or the Creation Kit. We will not be flipping this flag in TESVGecko. If the flag is set, then the data will be recompressed when the plugin is saved.

That's the general overview. We'll have to go through all the records again to identify which ones have the compressed data and in particular the mismatch between stated size and the actual size. Currently TESVGecko doesn't handle different record types any differently - we just list the records in the format they are found in the plugin, so I don't know which records are affected by this issue.

UPDATE: The issues we found earlier were due to using the SharpZipLib port of zLib. This port does not include all of the zLib library. You must use zLib v. 1.2.3 to decompress/compress Elder Scrolls plugin data properly. This applies to all versions of Elder Scrolls and Fallout games up to and including Skyrim.

UPDATE 2: Sharlikran has picked up the development of TESVSnip. He and his team will be changing it so that it uses the proper compression library. You can follow their progress on the Bethesda forums.

Edited by AndalayBay, 01 December 2012 - 11:01 PM.
Added update

[AndalayBay]

Summary of issues:

- TESVSnip uses the stated size of decompressed data and does not process the actual size of the decompressed data, thus truncating the extra data
- TESVSnip does not flip the compression flag when it decompresses the data and saves the plugin. This will cause data corruption because the Creation Kit is being told the data are compressed, but the actual data are not.

Someone put out a fix for TESVSnip which will recompress the data upon save, but the truncation issue has not been fixed.

And these issues are caused by the compression library TESVSnip uses. TESVGecko originally used the same library and we discovered that data were being corrupted when we attempted to re-compress the data upon save. We have now switched to the zLib library.

Edited by AndalayBay, 31 July 2012 - 11:29 PM.

[Ysne58]

Thanks for that information AndalayBay.

[Sharlikran]

Just commenting, I understand what you are saying. When I use Never compress, it saves things to where I can work with it. I recently tried the new 4.3 version and I chose use default compression, Enable auto compression by type. I couldn't even look at the file with TESVSnip after I reloaded it. I had to open it in the CK and edit everything before I could see the trees again.

[AndalayBay]

Welcome to DC, Sharl! Good to see you here.

Ok, so that sounds like TESVSnip isn't compressing records correctly. When it's decompressing the records, it looks at the quoted size and only decompresses that much, truncating the rest. Perhaps when it compresses them, it doesn't record the size properly, so it's unable to decompress them again. Very strange and purely a wild guess on my part. We didn't delve into the code that far.

We were a bit confused about the Zlib documentation, so that's why we looked at TESVSnip to start with. We wanted to see how they implemented it. That's when we found the bug.

Very strange that TESVSnip couldn't open the file again but the CK could. I'm guessing that they use a different library or have implemented it differently. I bet they don't use the quoted size, for one thing.

If you still have that plugin, could you see if you can open it with Gecko? Or perhaps reproduce the scenario and see if Gecko will open it? Just curious.

[Sharlikran]

Yeah I can, I'll let you know what I find. Don't forget that the source to 4.3 is available on Nexus. If it mysteriously went away I have it. If I'm wrong and it's the source to 4.2, then eh I have it anyway.

Edited by Sharlikran, 19 June 2012 - 08:05 AM.

[AndalayBay]

Thanks - I'm looking forward to hearing what you find. I have the source, but it's awfully hard to follow.

[Sharlikran]

The file being mutilated is Blacksmithforge water fix and it says its compatible with several languages.
I have compression on
I edit the file by removing the dependency on Update.esm
The original file size is 102k, the saved file is 40k.
When you reload it in TESVSnip and try to expand the tree nothing appears.
Load it in the CK
I get an error that the ENG strings are wacky
I edit something in the CK, then save it.
Still 40k

I load the original file in the CK, no localization errors
I change the exact same thing, and save it.
Original file 102k, the saved file is 40k

I turn compression off.
I load the original file in TESVSnip
I make no changes, and re save it.
Original file 102k, the saved file is 102k

I load the original file in SkyEdit 0.060 Alpha
I make no changes, and re save it.
Original file 102k, the saved file is 1k
I load resaved file TESVSnip
No data retained by SkyEdit.

Conclusion: Skyedit is not making the file and saving in an uncompressed format. No data to support whether or not the original file was edited with TESVSnip and saved without compression. However, unmodified file saved using TESVsnip is the same size. The CK is compressing the file from 102K to 40K and retaining localization settings. Please note that originally I striped the dependency for update.esm out of the file. I did not test changing something else and then loading it in the CK to test if localization error occurs.

Edited by Sharlikran, 19 June 2012 - 09:22 AM.

[Sharlikran]

The file being mutilated is Blacksmithforge water fix
I have compression off
I edit the file by removing the dependency on Update.esm
The original file size is 102k, the saved file is 102k
TESVGecko displaies expected records.

I have compression on, same settings as above screenshot
I edit the file by removing the dependency on Update.esm
The original file size is 102k, the saved file is 49k (Maybe I looked wrong? 49 and not 40, weird)
TESVGecko displaies expected records.

I have compression on, Unchecked "Enable Auto Compress By Type"
I edit the file by removing the dependency on Update.esm
The original file size is 102k, the saved file is 102k
TESVGecko displaies expected records.

Edited by Sharlikran, 19 June 2012 - 09:52 AM.

[Sharlikran]

I loaded Extended Slider Colors.esp in the CK. Another multi-lingual plugin, without editing it at all and got a localization error.

When loading Extended Slider Colors.esp in TESVGecko, expanding the CLFM Color records it takes a while as expected. TESVGecko crashes when trying to expand the RACE Race / Creature type record while CLFM is expanded. May be a memory or number of records displayed issue. I'd make sure TESVGecko can handle 10240 records. You never know what crazy things people are gonna do with it.

Edited by Sharlikran, 19 June 2012 - 10:55 AM.

[Sharlikran]

I was not able to load Extended Slider Colors.esp into the CK and then save it. I'm probably not going to use v5. If I can't load and save v4 I might discontinue that as well.

[AndalayBay]

I wouldn't pull SkyEdit into this. I have no idea what it does and I've heard of lots of trouble with it. That's another one that probably does more harm than good.

10240 records isn't all that much. I would say that Gecko is crashing because there's some wrong with that plugin. The fact that the CK couldn't handle it either seems to confirm that.

It sounds to me like Gecko is handling the compression properly, which is what I was hoping to confirm.

Thanks for running these tests.

[Thomas Kaira]

I wouldn't pull SkyEdit into this. I have no idea what it does and I've heard of lots of trouble with it.

SkyEdit is a pseudo-editor that was created before the Creation Kit was available. As far as what it does, the Creation Kit made it totally obsolete and no one should be using it anymore.

It's basically a stripped down Creation Kit that isn't capable of handling roughly half of the records Skyrim is using and does not allow for worldspace editing. It's something of a bastard child. It's not low-level like TESVSnip or Gecko, but it's also too simple to be a high-level tool, as well. It's just this "in the middle" utility that was attempting to substitute for the Creation Kit before it arrived, and now that it has, SkyEdit's usefulness has been invalidated. And I believe the author knows this, as it appears all support for SkyEdit has halted.

So do your duty: any mod author who is using SkyEdit, make sure they know that the tool is obsolete now and should not be used (use the Creation Kit instead).

[AndalayBay]

Yeah, Dave Humphrey halted development of it ages ago. Don't use it guys - it has no clue what the new file format for Skyrim is and that has changed a couple of times already.

[Arthmoor]

Maybe Dave ought to have it pulled offline then?

[AndalayBay]

He just has it on GitHub. It's not available on Nexus. Although some guy did do a recent build and that's available on Nexus.

[Thomas Kaira]

He just has it on GitHub. It's not available on Nexus. Although some guy did do a recent build and that's available on Nexus.

It also got taken down recently because he didn't have Humphrey's permission to upload.

And also, TESVSnip's author just threw a temper tantrum and stormed off in a huff because he didn't like the community shouting out that the tool was broken and should not be used. Sorry, man, be we kinda expect your work to... not break ours. We do have minimum standards of quality we expect authors to meet, and you didn't. You have no one to blame but yourself for what happened.

http://forums.nexusm...ost__p__5621149

So, I guess it's time to call Snip Abandonware and toss it into the trash to be forgotten.

Edited by Thomas Kaira, 22 June 2012 - 06:09 AM.

[AndalayBay]

But it's still available for download which means that people will still use it. The "abandoned" version doesn't have Zaldir's warning on it. Anyway, for those who are interested, here's the bad code and the fix.

-- code removed. Issue was due to compression library.

Edited by AndalayBay, 31 July 2012 - 11:31 PM.

[morcroft]

so that's

		 if (AmountRead > Size)
		 {
			 throw new TemperTantrum("Hide head in sand");
		 }

right? Just checking
I'm so glad I gave up trying to write this sort of stuff!

Edited by morcroft, 22 June 2012 - 07:56 AM.

[AndalayBay]

Yup! You got it.