Modding Expeditions (KF)

[TheStolenBattenberg]

Ah, I forgot to mention the actual findings on the object data.

They've got a grid position in the file, between 0 - 79 like the tiles. But there is also an offset somewhere in the data to give it a more accurate position, and I haven't found that yet, which is why some are miss-aligned. The format is very different to the one stored internally in the engine,

The yellow dot on the bridge is a skeleton underneath it, the four purple dots on the light house are the lamps, and the dots by it are the herbs. Those yellows dots in a row make up the bridge that leads you towards the pirate area, with the traps. Bad spelling on my part, by intractable I meant interactive objects.

I know where the enemy data is located, but I'm having a hard time making a representation of it, the database is very confusing compared to the item one.

[Holy_Diver]

Ah, I forgot to mention the actual findings on the object data.

Seems you forgot again!

Off-topic: I should've said something in response to this before...

Might've looked into wrist braces if I'd of known of them earlier, but my wrists are already far too disfigured now.

I've never heard of "disfigured" wrists, however people who work with keyboard/mouse a lot run into strength/pain issues that are related to the nervous system. I am not a doctor, but I believe the real problem is sleeping with your wrists turned in so that its nerve is pinched through the night, and that computer work is more just a symptom of this problem, which people acquire late in life with or without a lifetime sitting in front of a computer. Doctors tell them to wear wrist braces. I found anyway that wearing a brace in the daytime is not necessary to feel normal, and it is obviously more of a disruption to your life than nighttime wear.

[Holy_Diver]

EDITED/Off-topic: On second attempt I found some BIOS files for Mednafen (http://www.psxdev.net/forum/viewtopic.php?t=56) the US/CA one having a dash in the title threw me for a brief loop.

It seems to have problems with the light or fog on the little plot of land you start, and the waterfall disappears at close range. It might be clipped by the "PGXP" feature. I didn't try it without it. That is interesting that it fixes the perspective issues. I wouldn't have thought that possible. Though it occasionally bugs out. It seems to be why the UI text is squiggly in emulators. I don't understand why the PGXP system cannot discern UI text, even if it had to do so heuristically. I wonder if it's just a KF problem. I don't understand why the menus always have those crossbars in emulators. It's even that way on the PSN game I had. And I think it's on the PS2 too. It wasn't in the original. I'm guessing that there is a standard aspect ratio that will fix the menu, but I didn't see options for that.


EDITED/P.S. FWIW I feel like the perspective correction makes the game feel sterile. Something about those squirmy polygons makes it feel organic/alive. I wonder if that contributes to the overall esthetic, and if so, how will SOM compensate. I hope the extensions I've developed impart some of the same feeling, since they are similarly squirmy/organic, although at much higher resolutions.

[Holy_Diver]

I think the T files are broken up into 2048B units has something to do with this:

   converts 2048bytes/sector images to 2352bytes/sector images

And at the same time I had my CUE file open to this:

FILE "King's Field.bin" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00

This suggests to me that on the CD there is 304 extra bytes for every 2048 bytes. There is also redundancy for every byte on a CD that must be apart from this extra overhead.

[TheStolenBattenberg]

Seems you forgot again!

I'm writing up some documentation for it, as that would be the correct course of action I think. Unfortunately it's harder to test exactly what's going on with these files without using pure trial and error, because FromSoft put checksums at the end of every file. Editing one byte stops it from loading the data and crashes the game. I'm currently working my way though a bunch of MIPS and IDA guides to see if I can stop that, but as it stands I have absolutely no clue what I'm doing when it comes to Assembly, and disassembled code probably isn't the easiest place to start.

I think the T files are broken up into 2048B units has something to do with this.

I remember learning on these forums a few years back that the 2048 byte length sections of data in T files could be to increase the speed at which the file loads, since (apparently) 2048 is the sector size the PS1 would load. Best way to check for the 304 bytes is to look at the files in JPSXDec with raw mode, or the raw CD sectors with CDMage. I don't think there are any though, and honestly I think the 2352 sector thing might be a mistake for the image, it would explain why it's impossible to copy the STR data from the 'OP' directory without using special tools.


On RetroArch/Mednafen:
All emulators are terrible for King's Field, both for the PS1 releases and Ancient City on the PS2. I think this is down to either earlier assumptions of FromSoftware using early Sony PS1 SDKs or FromSoft doing tricky business with the console. It's most likely a mix of both. I think King's Field is mesmerizing, and I don't think fully 3D textured mapped graphics with level streaming had even been a dream when they did it.

RetroArch and that core specifically are what I've found to be the best for it however, since it doesn't suffer from horrible timing issues (probably due to a better R3000A implementation than other emulators).

The grid pattern/crossbars you see on the UI is due to King's Field using 9-Patch sprites for their UI. Interestingly the HUD you see in game is an actual quad with colour data, rather than the sprite, which is why you don't get the grid on that. PGXP (with king's field at least) can't tell the difference between flat, screenspace vertices and the world, so everything gets processed the same. I don't think PGXP cares if the projection changes, it just processes the same way all the time.

I've never heard of "disfigured" wrists, however people who work with keyboard/mouse a lot run into strength/pain issues that are related to the nervous system.

On the disfigured wrists, arthritis runs in my family, and I play a few different instruments as well as program; so my wrists never get a break. I doubt it's purely down to typing.

EDITED/P.S. FWIW I feel like the perspective correction makes the game feel sterile. Something about those squirmy polygons makes it feel organic/alive. I wonder if that contributes to the overall esthetic, and if so, how will SOM compensate. I hope the extensions I've developed impart some of the same feeling, since they are similarly squirmy/organic, although at much higher resolutions.

If your SoM extensions support shaders, it's not too difficult to emulate PS1 graphical artifacts using them at a minimal cost to performance, should you feel it necessary.


I've been reading through 'Everything You Have Always Wanted to Know about the Playstation', and it has a pretty nice section on the texture data around page 29. It seems to be that texture pages are at hard-coded addresses in the VRAM:

Texture pages have a unit size of 256*256 pixels, regardless of color mode. This means that in the frame buffer
they will be 64 pixels wide for 4bit CLUT, 128 pixels wide for 8bit CLUT and 256 pixels wide for 15-bit direct. The
pixels are addressed with coordinates relative to the location of the texture page, not the frame buffer. So the top left
texture coordinate on a texture page is (0,0) and the bottom right one is (255,255). The pages can be located in the
frame buffer on X multiples of 64 and Y multiples of 256. More than one texture page can be set up, but each
primitive can only contain texture from one page.

It seems to me that it should just be possible to mark a texture page bounds in VRAM, then check when data is loaded into that area, convert the data to 32-bit RGBA pixels, and put that into a image buffer instead. It's going to require a little manual offset marking (for where the texture pages are in VRAM), but as I've already discovered they use specific pages anyway, so I know that those offsets won't change... Sort of like reverse texture atlasing.

Of course this is just theory, but I will try it tomorrow and update you with the results. It's certainly better than manually adding the textures onto the tiles again.

[Holy_Diver]

I have to pore over this post later. Just wanted to say that the lighting/waterfall glitches appear in ePSXe also. I don't know how I missed them. I have to check the PlayStation now... or a fat PS2 is as close as I can get. I don't know if that's guaranteed to have PS hardware inside or not.

If your SoM extensions support shaders, it's not too difficult to emulate PS1 graphical artifacts using them at a minimal cost to performance, should you feel it necessary.

It actually has a form of antialiasing that may be unique in the world and superior to everything that exists, with no GPU hit. What this means is geometric lines are usually perfectly straight even across the entire screen. It works on principles that are like the PlayStation's kind of, but its glitches are among the things I most dislike about King's Field II. I want to turn it into a 3D classic, VR poster child. Let's say that right now it is in a very primitive state. However I think the naive criticisms the general population would levy at it  are its strengths. Like a piece of abstract sculpture, its faults are not its rough facets. The level data currently has cracks in it, that look like the PlayStation's. I have to go in and prepare the meshes for this world-beating AA technology. In the meantime, those cracks just have to hearken back to the authentic PS experience.

All emulators are terrible for King's Field, both for the PS1 releases and Ancient City on the PS2. I think this is down to either earlier assumptions of FromSoftware using early Sony PS1 SDKs or FromSoft doing tricky business with the console. It's most likely a mix of both. I think King's Field is mesmerizing, and I don't think fully 3D textured mapped graphics with level streaming had even been a dream when they did it.

Well I don't understand how they work, but I always assumed PS emulators just implemented a virtual machine in terms of the CPU throughput, and anything on top of that was experimental. I also assumed that the rasterizer was functionally 2D (no "transform and lighting") but since I've been searching about "PGXP" I've got the impression that the "GTE" was 3D, and implemented in a coprocessor (firmware?) but I don't really know. How I imagined it is the CPU would convert the data into 2D triangles, and the hardware would rasterize triangles. If it takes 3D data, then it would be very straightforward to implement depth-buffer/perspective-correction.

I'm not super interested in the PlayStation, even though it is the console I most appreciate, and I feel like right now is the time to preserve its artifacts, whether the shadows of its creators are happy with that or not. (EDITED: to see Nintendo lashing out at 8-bit ROM preservation, and emulation in general now is just sad. I think they are killing their brand. I am not certain about the details, but if they are claiming anything but their own first-party titles as Nintendo's birthright they should just be laughed out of town and maybe have the shit kicked out of them.)


EDITED: OH YEAH! I really just wanted to say that the PGXP "Vertex Cache" option is what is poison to the menu textures. I think it's just an optimization. It seems to progressively warp the menus over time to boot.

I think King's Field is mesmerizing, and I don't think fully 3D textured mapped graphics with level streaming had even been a dream when they did it.

EDITED: FWIW I have yet to find a title published before King's Field that I would say is a modern 3D video game. It's arguable if it counts because the PS doesn't have a depth buffer, etc. but I think that's on the hardware more than the essence of the game. Before KF (1994) every game is pseudo-3D. But KF has all of the characteristics of SOM and every game that's come since... since we settled on full-polygon, full-freedom basically. (I'm sure a floodgate of games landed in 1995, and prior to KF2. But there's something to be said for being first.... even if KF2 is the golden child in my book.)

[Holy_Diver]

EDITED/Off-topic: So I tried a PS2. These glitches technically exist there, however the fog is thick enough that the visibility is half of what the emulators are showing.... so IOW either there is a gamma ramp (I assume there is one, because KF's colors are not very natural, and SOM had a custom gamma ramp) that is not being emulated, that would crush black colors so that these glitches are not noticeable, or... however fog works in the PS is simply not set up correctly, or the wrong fog values are in use.

I don't see any options to control this. I tried using 15bit color, but that didn't change the visibility.

EDITED: This suggests that there is something wrong about those tiles in the game's data, but developers never noticed it. I wonder what could make them turn black like that, from a distance?

[Holy_Diver]

I remember learning on these forums a few years back that the 2048 byte length sections of data in T files could be to increase the speed at which the file loads, since (apparently) 2048 is the sector size the PS1 would load. Best way to check for the 304 bytes is to look at the files in JPSXDec with raw mode, or the raw CD sectors with CDMage. I don't think there are any though, and honestly I think the 2352 sector thing might be a mistake for the image, it would explain why it's impossible to copy the STR data from the 'OP' directory without using special tools.

The 304 bytes is for the CD format. It doesn't appear on a mounted disc. It's metadata for seeking around the disc no doubt. Or where the redundant copies are if the disc is scratched.

It seems to me that it should just be possible to mark a texture page bounds in VRAM, then check when data is loaded into that area, convert the data to 32-bit RGBA pixels, and put that into a image buffer instead. It's going to require a little manual offset marking (for where the texture pages are in VRAM), but as I've already discovered they use specific pages anyway, so I know that those offsets won't change... Sort of like reverse texture atlasing.

I'm not sure I follow, but you can use something like OGLE to capture the vram if it's an OpenGL texture. It will take a snapshot whenever the texture is updated. That will give you pictures of the VRAM between frames.

I remember reading about checksums. If you are ever going to modify it you would need to figure out how to generate the checksums. I think just staring at the files will get the best results. They will unravel like a puzzle. I would leave the checksum problem for last... to see if it still feels like it's worth it or not. I don't think it's necessary to figure out what's what.

[TheStolenBattenberg]

I'm not sure I follow, but you can use something like OGLE to capture the vram if it's an OpenGL texture. It will take a snapshot whenever the texture is updated. That will give you pictures of the VRAM between frames.

I think it would be best to just emulate this technology, by creating a buffer the size of the playstation VRAM, then sending all the texture data to there in its native 4BPP format. What I mean by setting texture page bounds is to keep a list of structs that define a rectangle area and a pixel buffer of 256x256 in RGBA format. For King's Field this would be thirty two (y)256x(x)64 areas in VRAM, which would cover the entire (y)1024x(x)512 area. King's Field uses nothing but 4BPP images so it's easy to make this assumption.

When texture data is put into these bounds, I correct it's position to the texture page by subtracting the texture page offset from the image data coordinates,  convert the texture data to 32BPP, and put it in to the texture page's pixel buffer using its corrected position.

I quickly set up an example, using the PS1 vram from an EPSXE savestate, and it turns out it's even easier than I thought. Texture pages don't have their positions set by the developer, they just go from 0 - 1024 in increments of 64, then when they reach the end do the same for the next row. Where the indexes land doing this is exactly where I've been manually loading the data anyway. If you don't mind having a unique set of 32 256x256 images with a bit of junk at times for each map, I'd say this would be fine for SOM. This means we can now effectively get the textures back for items, tiles, ui and everything else that uses textures.

Here's a screenshot, so you have a better idea of what I'm talking about:


EDIT:
Considering how aligned everything is, it's feasible that we could skip the VRAM buffer stage, and just divide the imagex by 64 and imagey by 256 in the RTIM and TIM files to calculate which texture page the data goes on.

[TheStolenBattenberg]

I quickly used the data I'd gathered to for the last time manually add the water texture to 'Page 21' to see if I was correct:



King's Field II looks pretty good with actual depth sorting and a modern transparency method.

Edit:
This also opens up other doors. Since we now know that texture page data is stored in set positions, and where the water texture is, we can check for that Texture page and those UV coordinates when loading the tileset, and change the division for the vertices, to fix the issue you were speaking of before, with overlapping edges. Tiles using specifically that texture can now have their vertices divided by 1024 instead of 1034, to automate even more of the process; Meaning it's no longer a choice between overlapping edges or gaps in the water.