Author Topic: Modding Expeditions (KF)  (Read 45949 times)

Offline TheStolenBattenberg

  • Capricorn Crusher
  • **
  • Posts: 192
Modding Expeditions (KF)
« on: July 06, 2018, 10:13:48 pm »
Edit:
This somehow became a project, here's a link to the GitHub so you can enjoy it too: https://github.com/TheStolenBattenberg/Psycpros


Hey,

Long time no see all. How've you all been? I've been spending the last couple of years studying game development in college and fine tuning my programming skills (and becoming substantially less of an idiot).

Recently I've been getting into more advanced game modification, and I've of course made my first victim King's Field... With that I present to you King Kraken, a magnificent creature in a cruel world!




Anyway, I've got some goodies here for anyone who also likes to do a bit of game modification...

1) Attached is Cheat Engine table for NTSC region KF2(1), it's got a bunch of memory values inside, including Player Values, Engine Values, an Item Database, and an Enemy 'Test' thing which you can see used above. There is also a value to adjust the internal screen width, but it causes some funny results...

You'll need to use ePSXe 1.9.25 to take advantage of it, since that's the only PS1 emulator I've found which has static memory, which is absolutely crucial for the table to work.

2) Also attached is a little program I made called 'TScan', it's similar to KFTool but it just extracts Tim data and lets you reimport it, the upside is this program will find literally every FromSoftware style Tim in the game, including non power of 2 images, which allows for total editing.

That's all folks, I hope everyone is doing well,
- TSB
« Last Edit: February 23, 2019, 04:34:54 pm by TheStolenBattenberg »
Ashes to ashes, Dust to dust...
Honor to glory; And iron to rust.
Hate to bloodshed, From rise to fall.
If I never have to die; Am I alive at all?

Offline Holy_Diver

  • Holy Diver
  • Archmage of Light
  • *****
  • Posts: 2280
  • This account won't read/reply to Private Messages
Re: Modding Expeditions (KF)
« Reply #1 on: July 07, 2018, 10:06:51 am »
This is a godsend for me since I'm porting KF2 over to SOM :drool: The more you can help with KF2 the more time I can devote to enhancing SOM. I'm hoping (dreaming) of getting everything together by 2020 since that's their 25th/20th anniversary respectively.

Of course, everyone is encouraged to pitch in. The last thing I want is to do everything by myself!

Offline TheStolenBattenberg

  • Capricorn Crusher
  • **
  • Posts: 192
Re: Modding Expeditions (KF)
« Reply #2 on: July 07, 2018, 01:17:00 pm »
Glad it's helpful to someone, I'm still working on the cheatengine table, but not so much on the TScan. I do have an older version however, which supports direct reading, displaying and even extraction of the other files. I used this version mostly to replace textures with more cleaner versions. I don't know if textures were hand drawn or made from photos (looking at data on KFJ, I'd say the latter), but some of the colour palette choice wasn't great. The texture for the pirate area is a prime example of this, and I'd say this replacement stays true to the original while improving upon it greatly:


My main goal at the moment is to find any missing data that I don't currently have, which is just level data at the minute. Recently I've found how items are stored in memory, and I'm hoping that they are stored the same way in the game files, so I will be able to search them up. Here's that Herb behind the fisherman, altered to boots:


It's interesting to find that the item database in King's Field 2 was not only used for pickups, but certain environmental objects too. Setting the ItemID to 245 will result in this:


But I can also spawn a local skysphere and moon too, which really does give the impression that world data will be close by. It's similar with enemy data, in that certain meshIDs will spawn environmental objects, such as the rocking chair that Leon's mother's ghost sits in.

On this topic, I'd be interested to know if you have information on how SOM stores world items in memory, I'd like to see if their is some correlation (though I fear not, as it seems SOM is more a like to Shadow Tower's engine).

In regards to the 25th aniversary, I'm glad I'm not the only one thinking of that (temporary background art, of course):

Ashes to ashes, Dust to dust...
Honor to glory; And iron to rust.
Hate to bloodshed, From rise to fall.
If I never have to die; Am I alive at all?

Offline Holy_Diver

  • Holy Diver
  • Archmage of Light
  • *****
  • Posts: 2280
  • This account won't read/reply to Private Messages
Re: Modding Expeditions (KF)
« Reply #3 on: July 07, 2018, 11:25:29 pm »
Well, SOM is a better vehicle for preserving KF2, than "romhacking" it. In order to port it over, it's best to have all of the information on the disc on hand. So, whatever we can locate and document is incredibly important. But after the port is complete, the original disc will be the inferior product, since it won't be open source, and the PlayStation is crawling with glitches.

I would like to see everything documented on paper, but if the Cheat Engine map is a better way to create a blueprint all the better. In fact, it might be very useful to map SOM's player and tools in the same vein.

I am interested in preserving the art exactly as it is. And then afterward, it will be an opensource artifact, available for unlimited alteration. I mean, that can happen in parallel also. But when I release it, I intend to release it in the original form. An exact, inviolable replica. Because we are talking about copying a company's product with only tacit approval, I think it's even more important to not stray in official matters.

If we can apply more hands to the task, maybe it will be a cinch to hit the 2020 mark.

If we know where items and monsters, etc. are in 3D coordinates, we can begin to layout the maps in SOM, using their coordinates as reference points, for building the world map around them. I would personally enjoy reconstructing the world map by hand I think, however if we can find its footprint in the games memory banks, all the better.

I have such a work load with SOM, that I would honestly prefer it if someone else were to build the game itself. As long as they can make the work readily available to collaborators. I have a lot to do to make SOM up to the task of hosting the vertical mazes of Melanat, and streaming them in on the fly via their connecting corridors. I wanted to be doing this now, but this year I've been focused more on supporting features for the Moratheia project (that KF2 doesn't actually require) and now VR business. Building out SOM remains priority #1 no matter what.

Just because you asked, below is what an item instance looks like in SOM's memory. But there's also the items embedded in the map's memory, and then the index of parameters, and then their base profiles as well. So there are 4 different places where item information lives in all.

Code: [Select]
struct Item //32B
{
BYTE nonempty; //bool?
BYTE parameters; //566FF0 look-up
BYTE profile_instance; //Up to 64? 556FD0
BYTE layer; //Compared to MPX::x130?
float x,y,z,u,v,w;
int time; //guessing
int unknown; //small
};

P.S. The SOM version will not be identical to the original, because it will be using SOM. It should look the same, free of glitches, ideally, but by "the same" I mainly mean, same 3D coordinates, and same image files. Since it will operate possibly at a higher resolution (although PlayStation VR actually has comparable resolution to the original) it might be worthwhile to upscale the fairly simple/small textures, but if so I would prefer  an SVG strategy to simple upscaling. There will probably be issues with gamma tables, etc. And I'm not 100% committed to preserving the AI, although I'd really like to get at the animation data if possible, and don't want to remake the animations if I can help it. I would like to use off the shelf tools to extract the artwork, but will develop something myself if necessary. I did a short project years ago to get the models off of the third Amored Core game's disc, which used significantly modified version's of Sony's SDK. I've never come across files that use Sony's animation APIs yet (there may have been two, one release at a later date) that are called something like HMD. I collected a lot of PlayStation manuals years ago for this project, that I've not lost track of, but I don't the state today about open knowledge of PlayStation discs, whether it's better represented on the WWW, or worse. I will take the thing apart from scratch if I must.

EDITED: "HMD" looks like head-mounted display, but thinking back, I believe that's the right name for the file-format/library, which I think stands for Hierarchical Model Data.
« Last Edit: July 10, 2018, 01:06:03 am by Holy_Diver »

Offline Holy_Diver

  • Holy Diver
  • Archmage of Light
  • *****
  • Posts: 2280
  • This account won't read/reply to Private Messages
Re: Modding Expeditions (KF)
« Reply #4 on: July 07, 2018, 11:43:43 pm »
ALSO: For what it's worth, somewhere in this forum, there was a link or attachment to a cache of 3D models that someone had successfully extracted from the game. So, if nothing else, the 3D data must be in a standard enough format (something like TMD) on the disc. Though map pieces were not represented. And there were not animations, and they didn't have TIMs either. But I took it upon myself to pull a texture out of VRAM to show that was a way to match TMDs to their TIMs. But you seem to have successfully extracted the TIMs, so it's probably just a matter of putting them together to be able to make profiles for use with SOM.

If someone thinks they know where the animation data is, I will take a look at it myself. I'm sure I poked around the disc a little bit at some point. But I haven't made it into a project at any point in my life.

I would like to form a team, but I don't want this forum/topic to become a gathering place. I will email you if you want to discuss how to organize.

Offline TheStolenBattenberg

  • Capricorn Crusher
  • **
  • Posts: 192
Re: Modding Expeditions (KF)
« Reply #5 on: July 08, 2018, 09:23:36 pm »
Other than finding out how King's Field works, I won't be much use on a SOM project. Despite being a member of the forums here I mostly only signed up to talk about other peoples projects and share some tools and the like that I thought people would find useful; when it comes to SOM I have no clue how to use it, nor the time to learn. I've already got my hands full as the engine programmer for a start up games developer.

What I can do is share information with you and tools though, I do have documents on files with in King's Field, not just cheatengine tables, and they stand true for the American King's Field too, and somewhat for Shadow Tower, so I know they're correct. Not to mention a plethora of tools for conversion, and it wouldn't be too difficult for me to write something to extract texture and model files directly into a format SOM can read, if I can understand the SOM format enough. Though I don't want to share my email publicly here though in fear that a bot will take it, and I note that your don't accept private messages?

Just now I've found the level data for KF2, and extracted it, but it uses some propitiatory file format I have no clue how to read, so that's the next task. This means that I've only to locate Animation data and we'll have everything from the game, but I fear that animation data may be stored inside the EXE like the text, as the way the memory looks would lead me to believe this.
Ashes to ashes, Dust to dust...
Honor to glory; And iron to rust.
Hate to bloodshed, From rise to fall.
If I never have to die; Am I alive at all?

Offline Holy_Diver

  • Holy Diver
  • Archmage of Light
  • *****
  • Posts: 2280
  • This account won't read/reply to Private Messages
Re: Modding Expeditions (KF)
« Reply #6 on: July 08, 2018, 10:51:35 pm »
Though I don't want to share my email publicly here though in fear that a bot will take it, and I note that your don't accept private messages?

You can PM me. I seem to be posting here lately in the hopes that PlayStation VR can generate some attention for SOM. I don't normally, except this place has been completely dead for a while now, and so it's pretty pitiful. That is there to discourage random interactions.

I just now tried to look at the CE arrangement. I couldn't get CE to attach to the ePsxe process. It just says Found 0. Do you know? I tried to "Open" and to "Attach" but I don't know what to do since neither worked. The ePsxe I have is a little annoying because it always hangs for minutes after it is closed. I don't have much modern experience with emulators (when I was but a babe I played everyone and every game but that was a lifetime ago.)

I think something that would be immensely valuable is we can inject a DLL into the process, and use it to drive the camera to implement an analog controller. Then I can envision playing classic KF with VR. And I think combined with a nice looking high resolution rasterization it would be a very solid, modern experience. KF4 might even be playable if the same can be done for it. If you wanted to heavily modify one of the games, that one is a stinker, and could stand to benefit from some revisioning.

My recommendation, is to put core files/documents in a versioning system like GitHub. I prefer Subversion, but GitHub is popular and likely a safer place to keep public files. And I would like to see, alongside this, a project to document everything in a wiki. A wiki is not as easy to copy/back up for safe keeping, but the http://swordofmoonlight.org wiki is very reliable, and that would be my recommendation. I would certainly be more eager to involve myself with editing it, than some commercial wiki.

The SOM port will be based on the original. But it's probably fine to work with the anglicized product up to capturing and publishing the text. Which I have a text transcript of, taken from a Japanese website. So that it's not completely necessary to extract it. (I think it is stored as images in the game.) But if there is a good way to locate the text in the world, that's better than attempting to do so through observation.

In any case, if you help me with connecting CE to the process, I can take a look at it. And if you have any advice for improving the ePsxe experience, I would greatly appreciate it. And if you think there's anything to gain by mapping SOM's process memory, I can certainly provide addresses. I've had a mind to reorganize my code that deals with addresses, so that they can be like compile-time constants and more self-documenting. (The reason they were not always so, is originally I'd attempted to maintain them for the "retail" and "debug" versions of the player, but seeing how counterproductive that was, I long ago standardized on the debug version.)

EDITED: Oh, I think I am not seeing anything because I'm not using the Japan disc. "NTSC region KF2" kind of flew past me. I have the disc, but I don't have a CD-ROM drive... so I will have to wait for a visitor to bring me a laptop with a drive built in in order to take a look. For what it's worth, I prefer to work with the original.

P.S. I tried again after looking at your screenshots. I swear when I tried before that the Value field was absent. But I overlook things all of the time, so I suspect it was always there. I got the impression, from the general lack of any change to the CE display that it was not attaching to the process... offering no visual feedback it had done so, assuming that it has.

EDITED: For what it's worth, I believed you were using the English language version because your visualization prints "Herb" in your screenshot. My thinking is that was a name field.
« Last Edit: July 08, 2018, 11:09:04 pm by Holy_Diver »

Offline TheStolenBattenberg

  • Capricorn Crusher
  • **
  • Posts: 192
Re: Modding Expeditions (KF)
« Reply #7 on: July 09, 2018, 06:14:58 am »
When it comes to CheatEngine, it is the US version of the game, I'm from the UK so I have no idea what you guys call your versions, I always thought NTSC-J was Japanese though. There's nearly no one who has heard nor played King's Field around these parts, so I decided it would be best to base any work off the US version to make it more compatible with the majority King's Field players.

It's also absolutely crucial to use ePSXe 1.9.25 as stated in the first post. The other versions use non-static memory, so the addresses I've marked won't be the correct ones, and it'll just show random values. I don't think plugins effect it since I've changed my gpu plugin a few times and the addresses have remained the same.

On the subject of dll injection, I have an API set up for this and I've implemented a simple but effective mouse control system, so it would be absolutely possible to add analog control. KF4 is an absolute pain, I've looked through the data many times throughout the years but never found a thing, they're using some kind of archive with compression/encryption I'd bet. Unfortunately PCSX2 does not have a version with static memory, so dll injection is near enough impossible unless you want to update the addresses every time you launch it.


On King's Field, I have indeed found the level data, or some form of it. So I spent the night writing a T extractor and TMD viewer so I could be sure... Unfortuently I'm not entirely sure if it's a tileset or some kinda BSP tree system, because FromSoft have done their age old trick of sticking their hands into the Sony SDK and changing the way the files work... They're essentially a TMD file, but they've done something to mess with the vertices:


This is similar to what they did with the Tim files. I don't think it will be anything too complicated, probably just a re-arrangement of the vertex data, but looking at the data in a hex editor is not giving me much hope:


Selected is the vertex data for one of the objects / tiles, the 'F6 FB' you can see is what's worrying me, vertices don't normally look like this.

I'll keep looking and see what I can come out with.
Ashes to ashes, Dust to dust...
Honor to glory; And iron to rust.
Hate to bloodshed, From rise to fall.
If I never have to die; Am I alive at all?

Offline TheStolenBattenberg

  • Capricorn Crusher
  • **
  • Posts: 192
Re: Modding Expeditions (KF)
« Reply #8 on: July 09, 2018, 08:43:17 am »
And... With a good 8 hours work, we now have...


Every single tile from King's Field 2, and King's Field 3. :)

FromSoft are seriously some of the craftiest programmers, their trick for this one was to multiply the vertex indices by 8, so it would point to bogus data in the rest of the file... Caught on when I saw it loading a value of 4096, which is 1.0 in a fixed point 16-bit float. It was loading a normal as a vertex.

Unfortunately I still haven't found the tilemap data, but having the tiles brings us one step closer...

TMD Renderer is pretty good too, supports texture mapping and alpha transparency.
Ashes to ashes, Dust to dust...
Honor to glory; And iron to rust.
Hate to bloodshed, From rise to fall.
If I never have to die; Am I alive at all?

Offline TheStolenBattenberg

  • Capricorn Crusher
  • **
  • Posts: 192
Re: Modding Expeditions (KF)
« Reply #9 on: July 09, 2018, 03:21:16 pm »
And finally...
Ashes to ashes, Dust to dust...
Honor to glory; And iron to rust.
Hate to bloodshed, From rise to fall.
If I never have to die; Am I alive at all?