Modding Expeditions (KF)

[Holy_Diver]

FromSoft are seriously some of the craftiest programmers, their trick for this one was to multiply the vertex indices by 8, so it would point to bogus data in the rest of the file... Caught on when I saw it loading a value of 4096, which is 1.0 in a fixed point 16-bit float. It was loading a normal as a vertex.

Great job! are you sure the data is obfuscated? I kind of got that impression when I worked with Master of Arena's data. But at the same time, I felt like, if their data is purposely obfuscated, that it was so only because of mandate by Sony, or lawyers, in order to demonstrate the data is is not just freely available like a public document. I mean that, it's so simple that it doesn't actually pose a challenge to art historians.

I'm elated by your success... mainly that I don't have to sit down to scrutinize the disc myself! Believe me, I've done nothing but work like this full time for many years, so I'm glad to be able to share the workload for a change. I'm glad you're doing truly beneficial work also.

It's also absolutely crucial to use ePSXe 1.9.25

I see, all the more reason to port this artwork to SOM as fast as our legs can carry us then. I will give you an ISO for the Japanese disc after I have a CD-ROM handy.

On King's Field, I have indeed found the level data, or some form of it. So I spent the night writing a T extractor and TMD viewer so I could be sure... Unfortuently I'm not entirely sure if it's a tileset or some kinda BSP tree system, because FromSoft have done their age old trick of sticking their hands into the Sony SDK and changing the way the files work... They're essentially a TMD file, but they've done something to mess with the vertices:

Yes, this was my experience with the Armored Core disc. If you are having difficulty I can give you the changes to its TMD files, so you can see if they are the same. I programmed a little thing to convert the "files" to standard TMD format.

For what it's worth, SOM's binaries have some strings that mention a "BSP Tree" but I've never found anything in SOM that looks like a BSP tree. But I may yet. I think if it does have anything like that, that it's a very simple maze building system on a 2D grid. Not a true BSP algorithm, but a way to mask walled off dungeon like areas. But it's possible KF is more elaborate than SOM, since a lot of SOM seems to be dumbed down. It uses brute force a lot. You get the impression that it started out like a copy of the PlayStation work From Software must have built up, but that the staff realized that wasn't robust enough for an all purpose level designer, that they felt laypeople shouldn't have to fuss with things. And it's true, that PCs are capable of brute forcing a lot of things by then that PlayStation developers could not have taken for granted.

On the subject of DLL injection... there are a lot of ways that can appear to work, but are not truly reliable. I developed more robust code for SOM over time, as chinks in the armor appeared. So if there is a good candidate for injection, then I can give you hardened code for doing it right. Basically to do it right, you need to use the debugging/breakpoint APIs and suspend the executing thread, insert a breakpoint at the main entrypoint, so that all of the static translation units get initialized before you do the injection, but before the program itself begins. In the case of an emulator, you might have more freedom, since it doesn't launch right into the game, but you may or may not run into conflicts if not in complete control.

Has anyone forked this static version of the PS emulator? It seems pretty dumb to not use static memory for an emulator. Injecting analog controls and things into games is far more valuable than merely running them in my estimation, because that's what keeps them from being truly enjoyable. King's Field 2 is no less beautiful than any game that I know, but the controls can be a stumbling block. I think the glitches are lamentable, which is why I want to see it ported/preserved in my lifetime.


P.S. The second game on my calendar after KF2 is Armored Core: Master of Arena. I think that it will be some time before I can begin working AC features into SOM, but it's definitely something I want to pursue. It will be a big project, but I think it can be a good thing for SOM, because it has a very much a competitive/versus element, which can attract people who are not strictly story/art minded people. I mention it because you clearly have the experience and inclination to make an impact. We may use this (https://www.facebook.com/pg/moratheia/photos/?tab=album&album_id=1391005644250188) raven model from Moratheia as a mascot for a "Project Raven" expansion to SOM, in the coming decades in order to have some cross over among SOM's family of products, which will also exist as databases of artwork for developing new/novel stories/scenarios.

While Master of Arena has a story campaign, it is the last in the PlayStation trilogy (it along with KF were From Software's core in house efforts/labors of love) and so carries over all of the robotic body parts from the original and second volume of its trilogy, and that's why it's an ideal candidate for historical preservation, since the competitive/versus element is my primary objective. Unlike KF, Armored Core is pretty consistent, so that if the story campaign was to be preserved, I think that you'd want to do so for every PlayStation game, starting with the first. But if you are only interested in the "versus" element, then the third/final volume is all that there is to consider.

I can't resist adding that I think that while AC has been a good series up until the last numbered volume (AC's name/streak may be ruined) I always felt like there was something about the PlayStation game that got lost in translation in the PS2 era... mainly because it had a very interesting way of evading the FCS (fire control system) that relied on using the digital d-pad, and it just seemed like a more nimble, engaging experience, that I always wanted to return to, except at the same time, PS era games always feel a little bit suffocating to me, even though it's my favorite era for commercial games, because of the limits of the then technology, and television screens, that can feel like playing games through a keyhole with eyes half closed. I wonder if we can kind remaster these games to make them more comfortable, if they would be markedly superior to the games of today.

[TheStolenBattenberg]

The T extractor actually works properly with T files, and extracts using the header information rather than scanning only for known file types, so I've managed to get every single possible file from King's Field. I think I've found the animation data, but my knowledge of animation technique is limited. We can establish that it will not be skeletal animation, and I doubt it's vertex morph; so we're probably working with some specific values that point to parts of a TMD file, and probably move these parts with code and mathematics, controlled by data in the file. The TMD file that's altered is also within these files, and this one is for the slime, so it's animation should be relatively simple. I will an attach example file to the post (FILE_0.ukn).

EDIT: Looking again, it's vertex morph. There multiple sets of vertices in each file that I didn't see before. This means we are but a step away of having all assets.

Map files come in three parts:
- A tile database, which has instructions for what tiles to use and in what way. It does not contain x nor z coordinates, but does have a height value, so we can establish that this is loaded into some form of array. Below this is data that could be used to control how, but I won't know for sure until I spend more time looking. Presumption would lead me to believe that the data tells the game where in the array each tile belongs, as the total tile count possible is 4000, which is between a 63x63 and 64x64 array, and this is way too small for a King's Field map.

- An unknown database. It could be related to enemies or items, but I haven't spent enough time looking.

- Event data of some form.. All end with the same tag 'CPE', and look like they may control scripts and interactions within a map.

Aside from this, if you want the extractor to get the data yourself, it is complete and functional now. I will attach it to the next post.

-

Great job! are you sure the data is obfuscated?

Yeah it was, the specific files 'RTIM.T' and 'RTMD.T' both contain messed up data. However I think there was a specific reason for this other than protection in the case of the 'RTIM.T' files. RTIM.T contains a load of level texture data, but the way T Files store it make it look like it was set up to be loaded in chunks by the engine, as in some form of texture atlas system.


In this image we can see 9 'big' files, other than one. These line up perfectly with the 9 tile sets and the 9 level files... and the 9 music tracks. The smaller file is most likely Guyra's area, and the back of the Colosseum.

If you are having difficulty I can give you the changes to its TMD files, so you can see if they are the same. I programmed a little thing to convert the "files" to standard TMD format.

I've got it working now, there's a screenshot of a fountain that you can see in the posts above which was loaded from this data. However, information from Armored Core would be interesting, as I can use the information to establish connections between the engines, and hopefully get the T Extractor working for that too, as it doesn't so far.

[Holy_Diver]

The TMD file that's altered is also within these files, and this one is for the slime, so it's animation should be relatively simple. I will an attach example file to the post (FILE_0.ukn).

It sounds like it's probably using an algorithm like Sword of Moonlight's MDL format, which may or may not be the same as Sony's SDK. I'm unsure if its origins have any connection to Id Software's MDL format, or if it began with Id, however game's like Quake used morph animations more or less exclusively, so there may be a connection. I am more than miffed that that vertex animation has become very exotic in modern times. I think it's the superior way to do animation, and I am inclined to gear SOM towards pure vertex animations. SOM also has a rigid/hierarchical "skeletal" animation option, but they are mutually exclusive at the level of the 3D model file.

The MDL format is unsuited to SOM, so my guess is that it was a workhorse for From Software, and SOM's staff were just too pressed for time to develop a more appropriate proprietary format. It uses TIMs and TMD patterns.

EDITED: I should add that the MDL algorithm is pretty complicated. I will look at your attachment, and try to explain it to you one way or another if you like. This (http://svn.swordofmoonlight.net/code/lib/Swordofmoonlight.cpp) file has procedures for animating it, in the "mdl" section.

Map files come in three parts:
- A tile database, which has instructions for what tiles to use and in what way. It does not contain x nor z coordinates, but does have a height value, so we can establish that this is loaded into some form of array. Below this is data that could be used to control how, but I won't know for sure until I spend more time looking. Presumption would lead me to believe that the data tells the game where in the array each tile belongs, as the total tile count possible is 4000, which is between a 63x63 and 64x64 array, and this is way too small for a King's Field map.

Actually that sounds just right. SOM is 100x100 and the KF2 in-game maps (e.g. Pirate's Map) are smaller, by about that much. It points to 2x2 meter tiles, which would be good for mapping to SOM conventions.

- Event data of some form.. All end with the same tag 'CPE', and look like they may control scripts and interactions within a map.

If the events are similar to SOM's it's theoretically possible to generate a PlayStation game from a SOM project file. It would be weird, but would work with any PS emulator, whereas SOM is limited to Windows until I feel like it's identity is solidified, at which point will welcome using it as a basis for a portable media standard.

Aside from this, if you want the extractor to get the data yourself, it is complete and functional now. I will attach it to the next post.

Well, I will be severely disheartened if you disappear before publishing your findings... however there's no real hurry on my part, so if you can refine things further, by all means take your time.

I would like to see an official home for this work. Please take time to think about how you can ensure its future availability and authorship.

I've got it working now, there's a screenshot of a fountain that you can see in the posts above which was loaded from this data. However, information from Armored Core would be interesting, as I can use the information to establish connections between the engines, and hopefully get the T Extractor working for that too, as it doesn't so far.

We have to be certain that nothing is amiss. The fountain screenshot lacked color, etc. And everything must be placed in its context.

For what it's worth, below is the code I wrote to convert some of the Master of Arena graphics (the body parts, etc. as seen in the shop/garage I believe) into standard Sony TMD format. The indices are shifted 2 bits in this case, from what I can tell. sizecheck32 checks asserts pointers are not out-of-bounds.

Code: [Select]

bool Sony(void *in, size_t sz)
{
sizecheck32(10)

unsigned __int32 *p = (unsigned __int32*)in;

assert(p[0]==0x41); //TMD file

if(p[0]!=0x41) return false;

assert(p[1]==0);

if(p[1]!=0) return false;

assert(p[2]>0);

if(p[2]==0) return false;

int nobj = p[2];

assert(nobj<=32); //arbitrary safety check

sizecheck32(3+nobj*7)

for(int i=0;i<nobj;i++)
{
int cobj = 3+i*7; //current object

int psets = p[cobj+5];

//restore primitive count...

p[cobj+5] = 0;

int ptop = p[cobj+4];

assert(ptop%4==0);

if(ptop%4) return false;

ptop = 3+ptop/4;
for(int j=0;j<psets;j++)
{
sizecheck32(ptop+1)

int pmode = 0, pstep;

//these are in the order that they appear in file

//NOTE: unsure what to make of transparent variations,
//except that they are not supposed to be transparent
//in game. So forcing non-transparent. Part models are
//usually either all transparent or not. This suggests
//that possibly different bit configuarations were used
//at one time or another across the development of all
//of the Armored Core games

switch(p[ptop]&0xFFFF)
{
case 0x070F: //transparent textured quadrangle

pmode = 0x2C000709; pstep = 8; break; //0x2E000709;

case 0x070D: //flat shaded textured quadrangle

pmode = 0x2C000709; pstep = 8; break;

case 0x0300: //flat shaded monochrome triangle

pmode = 0x20000304; pstep = 4; break;

case 0x0408: //flat shaded monochrome quadrangle

pmode = 0x28000405; pstep = 5; break;

case 0x0502: //flat shaded textured triangle

pmode = 0x24000507; pstep = 6; break;

case 0x080E: //Gouraud shaded textured quadrangle

pmode = 0x3C00080C; pstep = 9; break;

case 0x0603: //Gouraud shaded textured triangle

pmode = 0x34000609; pstep = 7; break;

case 0x0511: //transparent textured triangle

pmode = 0x24000507; pstep = 6; break; //0x26000507

case 0x040B: //Gouraud shaded transparent monochrome triangle

pmode = 0x30000406; pstep = 5; break; //0x32000406

case 0x060A: //Gouraud shaded transparent textured triangle

pmode = 0x34000609; pstep = 7; break; //0x36000609;

case 0x0810: //Gouraud shaded transparent textured quadrangle

pmode = 0x3C00080C; pstep = 9; break; //0x3E00080C;

case 0x0305: //flat shaded transparent monochrome triangle

pmode = 0x20000304; pstep = 4; break; //0x22000304;

default: assert(0); return false;
}

sizecheck32(ptop+pstep)

int prims = p[ptop]>>16;

sizecheck32(ptop+prims*pstep)

//restore vertex indices

for(int k=0;k<prims;k++)
{
int pcur = ptop+pstep*k;

p[pcur] = pmode; //restore appropriate mode flags

unsigned __int16 *q = (unsigned __int16*)&p[pcur];

switch(pmode)
{
case 0x2E000709: //transparent textured quadrangle
case 0x2C000709: //flat shaded textured quadrangle

q[11]>>=2; q[12]>>=2; q[13]>>=2; q[14]>>=2; break;

case 0x20000304: //flat shaded monochrome triangle
case 0x22000304: //flat shaded transparent monochrome triangle

q[5]>>=2; q[6]>>=2; q[7]>>=2; break;

case 0x28000405: //flat shaded monochrome quadrangle

q[5]>>=2; q[6]>>=2; q[7]>>=2; q[8]>>=2; break;

case 0x24000507: //flat shaded textured triangle
case 0x26000507: //transparent textured triangle

q[9]>>=2; q[10]>>=2; q[11]>>=2; break;

case 0x3C00080C: //Gouraud shaded textured quadrangle
case 0x3E00080C: //Gouraud shaded transparent textured quadrangle

q[11]>>=2; q[13]>>=2; q[15]>>=2; q[17]>>=2; break;

case 0x34000609: //Gouraud shaded textured triangle
case 0x36000609: //Gouraud shaded transparent textured triangle

q[9]>>=2; q[11]>>=2; q[13]>>=2; break;

case 0x30000406: //Gouraud shaded monochrome triangle
case 0x32000406: //Gouraud shaded transparent monochrome triangle

q[5]>>=2; q[7]>>=2; q[9]>>=2; break;

default: assert(0); return false;
}
}

ptop+=prims*pstep;

//restore primitive count

p[cobj+5]+=prims;
}
}

return true;
}


[Holy_Diver]

EDITED: Please tell me the extent about what you know about the FILE_0.ukn attachment.

Just tell me which addresses hold what kind of data as far as you know. There's no sense for me to lose any downtime figuring it out. I've spent many an afternoon staring into a hex editor, but it never becomes like a sixth sense. It usually takes an hour just to get settled in.

[TheStolenBattenberg]

We have to be certain that nothing is amiss. The fountain screenshot lacked color, etc. And everything must be placed in its context.

I didn't bother to load the texture data in yet, since I wanted to get the TMD files working properly, so I just threw a stone texture over it. You can tell it does work however, because alpha values are correct, and the water is set to be semi transparent. I'll have a version soon that also loads texture data, so we'll be able to see for sure.

It sounds like it's probably using an algorithm like Sword of Moonlight's MDL format, which may or may not be the same as Sony's SDK. I'm unsure if its origins have any connection to Id Software's MDL format, or if it began with Id, however game's like Quake used morph animations more or less exclusively, so there may be a connection. I am more than miffed that that vertex animation has become very exotic in modern times. I think it's the superior way to do animation, and I am inclined to gear SOM towards pure vertex animations.

I've looked into the MDL format before, and it isn't very similar. Not to mention, King's Field II was released before Quake, and I don't think Id used MDL before that. From what I can see, this particular file doesn't bother with full frames, and just keeps sets of vertex data. It does however look like a Sony '.MOT' file, but in binary form; which you can find in the playstation documentation, in 'filefmt.pdf' or 'file format 47.pdf', not sure what its official name is.

Actually that sounds just right. SOM is 100x100 and the KF2 in-game maps (e.g. Pirate's Map) are smaller, by about that much. It points to 2x2 meter tiles, which would be good for mapping to SOM conventions.

Awesome, I'll see what happens when I work with that mindset then.

EDITED: Please tell me the extent about what you know about the FILE_0.ukn attachment.

I uploaded a document to the Github link I messaged you, should be MO.T, that has my current findings available.

[TheStolenBattenberg]

Going to post this here in case anyone else comes across this thread, but I've also edited it into the top post.

Here's the GitHub link to the project: https://github.com/TheStolenBattenberg/Psycpros

---

Now, when it comes to the T files and their extraction they're currently pretty confusing to identify when they're extracted, because they don't have names nor true file extensions. I'm considering adding some kinda file format identification to the TExtractor part of Psycpros so you're not randomly trying to open different files hoping that it's the correct type.

[Holy_Diver]

I uploaded a document to the Github link I messaged you, should be MO.T, that has my current findings available.

What is MO.T? (https://github.com/TheStolenBattenberg/Psycpros/search?q=MO.T&unscoped_q=MO.T)

The MDL format does tweening between two frames at a time. There are usually not that many frames (key frames) all told. I can't remember the exact details, except that deltas are compressed, so that if short they are encoded in 1 byte, or if long, in 2 bytes. Each vertex has a bit field that indicates which of any of its components are advanced according to its particular delta. As I recall, short deltas have the lowest order bit set, whereas long deltas do not. So long deltas cannot move an odd number of units, which is millimeters. (edited: and so short deltas can only move an odd number.)

I think how animation works, is you can think of each delta (mesh wide, unless I'm mistaken) as its own entity, that can be applied forward or in reverse. And so the actual animations are timed sequence of these deltas. So you have starting point (the base mesh) and you move it by adding or subtracting a delta. So IOW an animation is just a list of time, delta (index) pairs and the tweening blends between the current two delta/time pairs, until the time moves beyond the future one, and so the past delta is no longer relevant, the future becomes the new past, and the next delta/time pair becomes the new future target.

Since I don't have anything to help me navigate your attachment, maybe you can figure it out yourself, armed with the knowledge of this model. I would be surprised if KF2's animations are substantially different from this. I would be interested to know if every animation works like this, or if there are some rigid animations also. In theory if every animation is "soft" there is always some degree of organic wobbliness in them no matter how mechanical. Some simple movements might be event based instead. SOM has events for rotating and positioning objects.

[TheStolenBattenberg]

Sorry, I meant to put 'MO.h'.. It's in Documents/File Formats in the main repository. There isn't much information there other than a few numbers I've identified and some notes I made on things I noticed in the files.

[Holy_Diver]

Sorry, I meant to put 'MO.h'.. It's in Documents/File Formats in the main repository. There isn't much information there other than a few numbers I've identified and some notes I made on things I noticed in the files.

For what it's worth, the impression I got from your writing is the FILE_0.ukn also includes vertex data, possibly polygons, etc. If so, I wanted you tell me which parts of that file hold what data that you know of. If however the file is only animation data, please say so. And please also say why you believe it is so. Without knowing these things I can't bring myself to make time to pore over the file.

[TheStolenBattenberg]

I haven't found that out yet. I think the 4th int in to the file points to vertex data, but I can't be sure until more information has been done.