Modding Expeditions (KF)

[Holy_Diver]

Is "MO" a known Sony SDK extension? MO can mean "machine object" like in Gettext parlance. If a portable text format is converted into a binary format.

[TheStolenBattenberg]

MOT is a known format, but it's only human readable. I'm comparing the data I've collected so far with it, which we have more of now. I cracked another part not five minutes ago, and cross referenced it with the others. There is still an unknown int in the header, but we now have some form of animation offset struct and a frame offset struct... If things keep progressing at this rate, we'll have the animations before next week.

New findings are on github.

[TheStolenBattenberg]

Okay, so we have enough information to read the data now; at least the header anyway:


There's a lot of pointers to pointers, it's very annoying.

[Holy_Diver]

New findings are on github.

If you write something like this online you're obliged to include a link

Okay, so we have enough information to read the data now; at least the header anyway:

Okay, so these looks like byte-sized offsets from the top of the file. Is that like any PlayStation SDK format?

It seems strange. I guess 0x14 is not to the first animation really, and the next word (0x24) is, and so the remaining words are the four animation offsets? Maybe 0x14 is an offset beyond an "extra" data block, to be extensible?

Are you positive this is animation? Probably. Good luck figuring it out... that last "animation" is 1 "frame," so it's probably a 0-terminated sequence. MDL uses 0-terminators for its frames sequences, however this is not really that is it. And if these are 0-terminated, maybe I'm seeing things, but it looks like a silly trick is in play that goes like, if the next sequence starts with a 0 then it can double as a 0-terminator, otherwise a terminator is inserted into the byte stream.

EDITED: I suppose it's just as possible the 0x104 offset is understood as the end of the final "frame" but my instinct is 0 termination.

P.S. Saying "Animation [ID:1]" is highly misleading, since there isn't an ID in the file. Say Animation #1. MDL files have animation IDs. But it's nonetheless misleading.

[Holy_Diver]

ALSO: I should warn you that the MDL animations (soft variety) are quite tricky. I don't see their parallels here so far (maybe in the vertex data block... assuming it's deltas.)  Here is what a frame looks like to them:

typedef union _swordofmoonlight_softanimframe
{
   struct{ int8_t lo; uint8_t time; }; uint16_t bits;

#ifdef __cplusplus

   inline operator bool()const{ return bits?true:false; }

#endif

}SWORDOFMOONLIGHT_PACK
swordofmoonlight_softanimframe_t;

I think that "bits" is the 0 terminator. Right now I'm not sure what it does otherwise. I think the animation code I developed uses it, but probably via a pointer instead of this structure/handle.

Strike-through: Oh, it occurred to me it must be written as a union object

The time is unsigned, so that it isn't used to run an animation backward. It always advances.

"lo" is so named because it's hard to peg. It is an index into a channel (I can't say right now what a "channel" is exactly) but if it's negative the channel plays backward (forward in time) but since -0 is impossible, to arrive at the channel's index (subscript) you do abs(lo)-1. It's tricky business.

Just trying to illustrate some of the gymnastics that could be involved... in case it saves you some head scratching. Other things to look out for are large arrays of bits. The MDL animation appears to (looking at notes) put all of the X components in one chunk of sequential bits, and so on for Y & Z. So: XXXXXXYYYYYYYZZZZZZ; Not: XYZXYZXYZXYZXYZ. And I can't recall the encoding scheme exactly, however if a bit is lit then there is a delta elsewhere, and like I say, that is either 1B for a small step, or 2B for a larger step, depending on if its low-order bit is 1 or 0.

P.S. I believe the rationale for separating components is just because 8 bits in a byte is easier to work with than 3+3 bits in a byte, plus overflow? It just packs together better.

[TheStolenBattenberg]

If you write something like this online you're obliged to include a link

I thought you'd of found the file haha, it's all I was updating yesterday: https://github.com/TheStolenBattenberg/Psycpros/blob/master/Documents/File%20Formats/MO.h

Figuring out the details of MO_Frame is absolutely critical right now. The problem is there seems to be a variable length, or something funny going on inside the file, and I can't tell exactly what this data refers too. There's another block of data at the end of this, a list of ints equal to the number of frames which points to vertex data. But that vertex data also appears to have a header of some sort...

Okay, so these looks like byte-sized offsets from the top of the file. Is that like any PlayStation SDK format?

It seems strange. I guess 0x14 is not to the first animation really, and the next word (0x24) is, and so the remaining words are the four animation offsets? Maybe 0x14 is an offset beyond an "extra" data block, to be extensible?

They're unsigned ints, and there is no playstation format that is similar. This is an entirely custom FromSoft format, other than the TMD file that's inside it. The 0x14 is an offset to the start of the animation data; in total their are three offsets. The TMD offset, the Vertex Pointers offset and the Animation Data offset (0x14).

Are you positive this is animation? Probably. Good luck figuring it out... that last "animation" is 1 "frame," so it's probably a 0-terminated sequence. MDL uses 0-terminators for its frames sequences, however this is not really that is it. And if these are 0-terminated, maybe I'm seeing things, but it looks like a silly trick is in play that goes like, if the next sequence starts with a 0 then it can double as a 0-terminator, otherwise a terminator is inserted into the byte stream.

Yeah, this is animation for sure. I found the files in CheatEngines memory viewer and played around with a few of the vertices in the blocks, and it altered the animations.

P.S. Saying "Animation [ID:1]" is highly misleading, since there isn't an ID in the file. Say Animation #1. MDL files have animation IDs. But it's nonetheless misleading.

This is internal naming conventions for how I'm storing the data in Psycpros. I read it out like that because it will make it easier when it comes to exporting it to different file formats, like MD2. Personally I think MM3D (Misfit Model 3D) would be a better call as an export format though, since it supports a more modern vertex morph approach.

Just trying to illustrate some of the gymnastics that could be involved... in case it saves you some head scratching. Other things to look out for are large arrays of bits. The MDL animation appears to (looking at notes) put all of the X components in one chunk of sequential bits, and so on for Y & Z. So: XXXXXXYYYYYYYZZZZZZ; Not: XYZXYZXYZXYZXYZ. And I can't recall the encoding scheme exactly, however if a bit is lit then there is a delta elsewhere, and like I say, that is either 1B for a small step, or 2B for a larger step, depending on if its low-order bit is 1 or 0.

Luckily I don't think this format is this complicated, it seems to store its vertices pretty normally, an X-Y-Z format. It's different to TMDs though, since that would also have a 0 at the end.

Edit:
Just bought a copy of Hex Workshop; I feel as if it will make explaining the format a little easier.

[TheStolenBattenberg]

This is what we've got so far:


KEY:
Mid Blue: Header Info (0x00 - 0x14)
Dark Purple: Animation Locations (0x14 - 0x24)
Dark Green/Dark Blue: Animations, they switch colours at the start of a new animation. (0x24 - 0x68)
Red/Light Purple: Frame Info, switch on new data packet. (0x68 - 0x9C)
Orange: Vertex Data Pointers (0x9C - 0x13C)
Light Blue: Vertex Data Headers? (Not sure on this one.)

Of course I'm not sure exactly why the 'frames' have a variable length, and what tells them to end. I think you might've been onto something with that 0-terminated sequence idea, since that last animation doesn't seem to do anything.

[Holy_Diver]

(BTW: I realized the source of my confusion yesterday/amended my prior posting.)

Figuring out the details of MO_Frame is absolutely critical right now. The problem is there seems to be a variable length, or something funny going on inside the file, and I can't tell exactly what this data refers too. There's another block of data at the end of this, a list of ints equal to the number of frames which points to vertex data. But that vertex data also appears to have a header of some sort...

Something else to look out for is something SOM uses called a CP or control-points file. It takes triangles artists put in the models and converts them into points (from their center) and then has extra data that tracks the points in every animation frame. This is just so the points don't have to be taken from the mesh data/calculated on the fly. They are sorted into categories based on their coloring.

That could be what you are seeing. Or it could be something else. And if it's not, there is probably something similar in some of the other files.

They're unsigned ints, and there is no playstation format that is similar. This is an entirely custom FromSoft format, other than the TMD file that's inside it. The 0x14 is an offset to the start of the animation data; in total their are three offsets. The TMD offset, the Vertex Pointers offset and the Animation Data offset (0x14).

Yeah I suppose. If it's memory mapped it can be a hedge against future additions to the header. Which is just another way of saying it's extensible. 32 bits is a machine word on the PlayStation. In Windows "word" means 16-bit, but that's only because it began on 16-bit machines prior to Windows NT. An "int" doesn't have a size.

Yeah, this is animation for sure. I found the files in CheatEngines memory viewer and played around with a few of the vertices in the blocks, and it altered the animations.

In that case, it shouldn't be hard to figure out, if the memory matches these files. I worked out MDL just by staring at it. Made some revisions while developing a model viewer (http://www.swordofmoonlight.net/canvas.php?shadow=1) that is also inside the website, as part of a cross-referenced database.

This is internal naming conventions for how I'm storing the data in Psycpros. I read it out like that because it will make it easier when it comes to exporting it to different file formats, like MD2. Personally I think MM3D (Misfit Model 3D) would be a better call as an export format though, since it supports a more modern vertex morph approach.

MM3D? Do you use it? It's interesting that you mention it, because I decided I would use that (defunct) project as the beginning of a modeling software geared for SOM developers. Something with vertex-morphing and palettes built into it. That will use COLLADA natively. I have some blog posts about its development on my website. It's called Daedalus, named after the DAE file format. I will start by rewriting the entire thing, but I find that it's better to develop software from a starting point than to just do it from scratch. It helps to get your brain's gears going, and I think it can be more attractive to say that some software had origins elsewhere.... it's less egotistical.

[Holy_Diver]

EDITED:

Of course I'm not sure exactly why the 'frames' have a variable length, and what tells them to end. I think you might've been onto something with that 0-terminated sequence idea, since that last animation doesn't seem to do anything.

I think that if these are indeed frames, then the last one must do something, even if it's counterintuitive. It can only be a pose. But it cannot be a terminator. That would be patently ridiculous. Especially because it has 1 "frame" as part of its metadata.

It is a little mysterious. Something weird about SOM's models, is they are broken into small pieces, or clusters of triangles. This is actually not good on today's hardware, because it increases overhead by way of many unnecessary drawing calls. I don't know if that is strictly CPU overhead, or GPU also. If CPU it's probably negligible. Anyway, it's clearly a vertex caching scheme for SOM. Which was important in the early days, and still is for high polygon models.

On the PlayStation, there may be clustering in order to avoid animating parts of the model that are motionless for any given frame, and possibly cache optimization also. That's my best guess of why the variable length anyway. SOM's are not variable to best of my memory. But it uses a bit array to mask the deltas blocks at the vertex component level.

[Holy_Diver]

EDITED:

This is what we've got so far:


KEY:
Mid Blue: Header Info (0x00 - 0x14)
Dark Purple: Animation Locations (0x14 - 0x24)
Dark Green/Dark Blue: Animations, they switch colours at the start of a new animation. (0x24 - 0x68)
Red/Light Purple: Frame Info, switch on new data packet. (0x68 - 0x9C)
Orange: Vertex Data Pointers (0x9C - 0x13C)
Light Blue: Vertex Data Headers? (Not sure on this one.)

This is what I gathered yesterday. I can say that the white section which I'd not noticed, looks a lot like MDL deltas. You can see, 00, 01, FF, FE, these are byte sized deltas no doubt, small ones in this case, ranging from -2, -1, 0, 1... you know this. But whatever they are these are byte-size cues either to apply deltas forward/backward at a certain rate, or actual deltas, although because they are so small, it seems unlikely their units are millimeters if so. I mean, such micro adjustments would not be visible to the naked eye at PlayStation sample rates.