Unproject with VulkanRipper using sphere mesh (RPCS3 updated)

    Final frame (what you see on screen) constracts from multiple render buffers / frame buffers , which is 2d images actually. So your desired mesh will presented in rip multiple times. There are depth pass, shadow pass, reflection, light effects. Depth pass usually do not have color flags and presented in rip with "NONCOLOR" suffix, shadow pass usually use ortographic projection and do not have UVs, presented in rip with "ortho" suffix, so you can simly exclude them deleting or using ripper config switches. Light effects have uvs. For example in RDR2 mesh rendered 3 times, so you should remove 2 of 3 equal meshes manually. Also mesh of final pass (what we usually need) will be almost flat, because of projection to screen.

   VulkanRipper able to find inverse projection matrix (unproject) if sphere object presented in rip. Consider game "Detroit - Become human". Rip at the game begining. To recognize spheres let us unflat the scene using trial unproject parameters in vk_config.xml:

   bUnproject = "true"

   bUserInverseMatrix = "false"

   UnprojectFoV = "45"

   UnprojectNear = "0.1"

   UnprojectFar = "10000"

   UnprojectWidth = ""

   UnprojectHeigth = ""

   bLogWhenRip = "true"

Then among thousands objects we can recognize sphere. Look at the left top corner for filename - "fr2_s46rp0_fb0_id1269", remember number after id (1269), open log file and search for string InverseProjectionMatrix[1269].

Note : More accurate sphere (high tesselated) yields more accurate projection parameters.

Put this matrix in config and rerip the scene.

    bUnproject = "true"

    bUserInverseMatrix = "true"

    inv_proj_matrix ="[[0.686633,0.,0.,0.],[0.,-0.389936,0.,0.],[0.,0.,0.,10.022637],[0.,0.,-1.,-1.000000]]"

    UnprojectFoV = ""

    UnprojectNear = ""

    UnprojectFar = ""

    UnprojectWidth = ""

    UnprojectHeigth = ""

sphere_radius = "0"

Check rip, look at the characters and geometrical objects.

Or skydom (hemisphere) if present.

Note: The render pass (the number after prefix "rp" in filename) should be the same for sphere you found and the scene you trying unproject.

What to do if sphere was not found? - Inject own sphere

There is possibility to inject own sphere if vertex attributes passed to the pipeline in a standart way and vertex binding description provided by application (for RPCS3 see below).

To inject sphere make rip with trial unproject parameters, select mesh index of object you want. In config file fill parameters:

             sphere_index ="1059"

             sphere_radius = "5.0"

             sphere_stride = "0"

where 1059 is mesh index you want, sphere will be injected after it (id will be 1060 for this example), sphere_radius - radius of the sphere and sphere_stride is vertex attributes size in bytes (e.g. for {x,y,z,w} size 16). If sphere_stride = 0 stride will be determined from vertex binding description (if it present) supposed that position will be first in attributes. If  sphere_radius = 0 then all meshes will be interpreted as spheres. Rip scene again -

You will see injected sphere, which is now distoted as all objects. Open log, find tag <SPHERE> ,you will see unproject parametres.

Input unproject matrix in config and rip again. Open rip in blender, move sphere in the origin, rotate/check.

 Now we done (Baldurs Gate 3).

IMPORTANT: mesh after which sphere injected must be static (solid/no weights)

If you have NR2 blender addon you may load mesh using "ProjectionMatrix[id]" of sphere mesh.

Injecting a sphere to gain projection matrix in RPCS3

In VulkanRipper 1.1 added posibility of injecting sphere in RPCS3 game replacing original vertex shader of a mesh. For this purpose you need to rip desired scene and close RPCS3. Open scene in editor (e.g. blender) with some trial unprojection parameters/matrix to unflat the scene or using runtime unprojection via config( e.g. for GLTF ). For example with 45 degree fov and width and height from renderpass info:

bUnproject = "true"

bUserInverseMatrix = "false"

UnprojectFoV = "45"

UnprojectNear = "0.1"

UnprojectFar = "1000000"

UnprojectWidth = ""

UnprojectHeigth = ""

Lets say we interested in scene with main character mesh. Choose unneeded mesh (in opposite case obviously extra rpcs3 launch required) with same renderpass and framebuffer (in this case it have same projection) for replacement by sphere. In this scene skydom background hemisphere(?) has same render pass id 3 and framebuffer id 4 (see mesh naming in manual). From skydom mesh file name we know CRC32 of vertex shader (digits started with 0x), place it in sphere_replace_for_vs_crc parameter in config. Parameters setup for this step in config will be

sphere_index ="1234567"

sphere_radius = "1"

sphere_stride = "16"

sphere_replace_for_vs_crc = "0x6b6936f6"

Save config, launch rpcs3, launch game, rip. After rip open vklog.txt and search for string "<SPHERE>"

For this time you can load your rip with "ProjectionMatrix" value with NR-plugin

or place "InverseProjectionMatrix" value into inv_proj_matrix parameter in config file

bUnproject = "true"

bUserInverseMatrix = "true"

inv_proj_matrix ="[[0.637076,0.,0.,0.],[0.,-0.358355,0.,0.],[0.,0.,0.,-0.099090],[0.,0.,-1.,-1.000000]]"

and reload them via vulkanripper tray menu ("reload volitile config params") while game is running and rip scene again than load in blender as GLTF or NR (with identity matrix)

After application of projection matrix also our injected spheres are fixed (two meshes with shader 0x6b6936f6 was replaced)

If you want repeat this steps game name is Drakenguard 3 (game begining).

More info about projection matrix can be found here:
https://vincent-p.github.io/posts/vulkan_perspective_matrix/
 To learn how final image constructed you may use renderdoc , nsight or other graphic debugger.