Fossilize replay linux что это

Обновлено: 03.07.2024

18 апр в 15:54

Since I switched hardware a few months ago I have been having an issue where my steam client will crash after a few hours, if I leave steam running with no games, it can take a whole day, if I'm playing a demanding game, it can be as quick as 30 minutes, when the client crashes, any running game freezes up and has to be killed.

When running steam from the commandline, the crash appears to be about running out of memory, but RAM usage is never even close to fully allocated:

Installing breakpad exception handler for appid(steam)/version(1618256785) ***** OUT OF MEMORY! attempted allocation size: 4120 **** src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY src/tier0/memstd.cpp (2489) : Fatal assert; application exiting src/tier0/memstd.cpp (2489) : Fatal assert; application exiting Installing breakpad exception handler for appid(steam)/version(1618256785) _ExitOnFatalAssert

I've tried some of the obvious troubleshooting: Reinstalling steam, removing the shader cache, disabling shader caching, reinstalling mesa, full system update.

What else should I try?

18 апр в 16:10 18 апр в 16:29 Yes, please read my entire post, I have plenty of available RAM, I have 64GiB and at least half of that is available when I crash. 18 апр в 17:27

I have seen that behavior with the beta client, but is rather inconsistent, can take up several hours or days or within a few minutes in a just booted up session, or never happens for a few days between power cycles or reboots. Like you, I have seen the OOM message in the console, evwn though Steam itself is using less than 1GiB (as reported by htop) but has several gigs cached and my system is a long shot from being OOM. I suspect a 32-bit process within the Runtime may be running out of address space, but that's just a hunch. That and the inconsistent behavior is rather unnerving.

Then again, I am running the Beta, so. maybe is kind of to be expected? (Instability)

18 апр в 20:27

The rare times I have had this problem is with 'Steam shader caching or pre-caching' feature and the fossilize_replay sub-process, in my real time memory usage applet on the taskbar, in less than 10 seconds, I saw the memory grow to such an extent that the system could no longer support it, and I think I close Steam when this happens.

I don't know what it was but what is certain is that it came from a process called fossilize_replay upon processing shaders before installing or first lime launching games.

Didn't really searched what is causing this since I got no crashes and with 16GB of ram, and full memory in less than 10sec. my only interest is to kill the fu.cking process :D

It only seems to happen once installing a new game, but not always, but now that you have a crash log, I have been able to link this to a possible current opened issue.

18 апр в 23:58 Did you reinstall? Either way, the first look should go to the new hardware. Did you check temperatures? And do run a memtest. 19 апр в 7:16 I have seen that behavior with the beta client, but is rather inconsistent,

I forgot to mention that I was running the beta client too, I've since opted out of the steam client beta but it had bo effect

Did you reinstall? Either way, the first look should go to the new hardware. Did you check temperatures? And do run a memtest.

Temps are fine, I'll run a memtest though, good idea, thanks.

I don't know what it was but what is certain is that it came from a process called fossilize_replay upon processing shaders before installing or first lime launching games.

Didn't really searched what is causing this since I got no crashes and with 16GB of ram, and full memory in less than 10sec. my only interest is to kill the fu.cking process :D

It only seems to happen once installing a new game, but not always, but now that you have a crash log, I have been able to link this to a possible current opened issue.

When I looked into this issue online most of the error messages I see seem to relate to shader caching, but I've disabled precaching, it also happens when I run games I've played many times, with Dota 2 I've tried opengl and vulkan with no difference.

Fossilize is a library and Vulkan layer for serializing various persistent Vulkan objects which typically end up in hashmaps. CreateInfo structs for these Vulkan objects can be recorded and replayed.

VkSampler (immutable samplers in set layouts)
VkDescriptorSetLayout
VkPipelineLayout
VkRenderPass
VkShaderModule
VkPipeline (compute/graphics)

The goal for this project is to cover some main use cases:

For internal engine use. Extend the notion of VkPipelineCache to also include these persistent objects, so they can be automatically created in load time rather than manually declaring everything up front. Ideally, this serialized cache could be shipped, and applications can assume all persistent objects are already created.
Create a Vulkan layer which can capture this cache for repro purposes when errors occur before we can create a conventional capture.
Serialize state in application once, replay on N devices to build up VkPipelineCache objects without having to run application.

GCC 4.8+
Clang
MSVC 2013/2015/2017+

If rapidjson is not already bundled in your project, you need to check out the submodules.

otherwise, you can set FOSSILIZE_RAPIDJSON_INCLUDE_PATH if building this library as part of your project. It is also possible to use FOSSILIZE_VULKAN_INCLUDE_PATH to override Vulkan header include paths.

Normally, the CLI tools will be built. These require SPIRV-Tools and SPIRV-Cross submodules to be initialized, however, if you're only building Fossilize as a library/layer, you can use CMake options -DFOSSILIZE_CLI=OFF and -DFOSSILIZE_TESTS=OFF to disable all those requirements for submodules (assuming you have custom include path for rapidjson). Standalone build:

Link as part of other project:

For Android, you can use the android_build.sh script. It builds the layer for armeabi-v7a and arm64-v8a. See the script for more details.

Overall, a binary database format which contains deflated JSON or deflated varint-encoded SPIR-V (light compression). The database is a bespoke format with extension ".foz". It is designed to be robust in cases where writes to the database are cut off abrubtly due to external instability issues, which can happen when capturing real applications in a layer which applications might not know about. See fossilize_db.cpp for details on the archive format.

The JSON is a simple format which represents the various Vk*CreateInfo structures. When referring to other VK handle types like pImmutableSamplers in VkDescriptorSetLayout , or VkRenderPass in VkPipeline , a hash is used. 0 represents VK_NULL_HANDLE and anything else represents a key. Small data blobs like specialization constant data are encoded in base64. When recording or replaying, a mapping from and to real Vk object handles must be provided by the application so the key-based indexing scheme can be resolved to real handles.

The varint encoding scheme encodes every 32-bit SPIR-V word by encoding 7 bits at a time starting with the LSBs, the MSB bit in an encoded byte is set if another byte needs to be read (7 bit) for the same SPIR-V word. Each SPIR-V word takes from 1 to 5 bytes with this scheme.

Open with Desktop
View raw
Copy raw contents Copy raw contents Loading

Copy raw contents

Fossilize is a library and Vulkan layer for serializing various persistent Vulkan objects which typically end up in hashmaps. CreateInfo structs for these Vulkan objects can be recorded and replayed.

VkSampler (immutable samplers in set layouts)
VkDescriptorSetLayout
VkPipelineLayout
VkRenderPass
VkShaderModule
VkPipeline (compute/graphics)

The goal for this project is to cover some main use cases:

For internal engine use. Extend the notion of VkPipelineCache to also include these persistent objects, so they can be automatically created in load time rather than manually declaring everything up front. Ideally, this serialized cache could be shipped, and applications can assume all persistent objects are already created.
Create a Vulkan layer which can capture this cache for repro purposes when errors occur before we can create a conventional capture.
Serialize state in application once, replay on N devices to build up VkPipelineCache objects without having to run application.

GCC 4.8+
Clang
MSVC 2013/2015/2017+

If rapidjson is not already bundled in your project, you need to check out the submodules.

Link as part of other project:

For Android, you can use the android_build.sh script. It builds the layer for armeabi-v7a and arm64-v8a. See the script for more details.

In the new steam beta after every update to Dota2 (doesn't seem to matter how small of an update), fossilize will rebuild the vulkan shaders when launching Dota2.

When it is rebuilding it eats all of my ram (16gb) and makes my computer swap. Making it unresponsive for around 1-2 minutes while it is building the shader cache.

Perhaps there could be some mechanism in place to make sure fossilize doesn't go overboard with memory usage if it will go over the available amount of memory?

I don't know if this is the correct place to report this. Sorry if it is not.

The text was updated successfully, but these errors were encountered:

Are you able to reproduce it on your end? (With the steam beta)

HansKristian-Work commented May 27, 2020

Haven't triaged it, but I'll look at it.

Ok, I tried replaying locally and I can reproduce rather large memory consumption when replaying on the 5700xt. This does not happen when I run with a dummy device, so the memory consumption must happen in the driver. I suspect it's retaining a lot of memory in the disk caches, and multiply that over lots of processes, it becomes a problem. I've seen this issue before, but to my knowledge it was fixed in Mesa a while back.

Seems like I might need to poke around to see where it's hogging so much memory .

DarkDefender commented May 28, 2020

If it is any help, I am currently using and building mesa from the git master branch.
So I am seeing this with the absolutely bleeding edge drivers.

rLy07 commented Jun 8, 2020

Hey, I didn't know that this has been tracked here before I opened my report at steam for linux. As you can see this effects Shadow of the Tomb Raider as well. Also I'm not sure if the following should be treated as a separate issue (or if is an issue at all) but background processing happens every time I start steam even when there are no updates to the games. On top of that today SotTR run twice(few other games run between the 2) without restarting steam.

Noticed from time to time it starts to release some memory (intended, crash or terminated?) but the consumed memory is by far higher than the released so it will still happen.

How many processes and threads should it be using in normal conditions?

Arch Linux
kernel version: 5.7.0-2-tkg-pds
mesa version: 20.1.0
Graphic card: AMD R7 240
For me it takes like 20 minutes of trying to process shaders and all my 8 GB of ram is eaten up and my whole system can crash. Dota 2 normally takes maximum 3 GB ram when it's been on for some hours.

lewisdiamond commented Jun 13, 2020

I'm also on Arch Linux:

I have installed earlyoom to kill anything that is not steam/dota2 and almost all other processes get killed and processing vulkan shaders takes the full 16GB, barely anything left for the system to run.

I'm having the same issue on KDE neon with an RX570 (Mesa 19.2.8). Doesn't even help to launch the game with -gl, already got two abandons because I didn't manage to restart Dota within five minutes after a crash, even though I had killed all steam processes in htop.
How to fix this mess?

rLy07 commented Jun 14, 2020

I'm having the same issue on KDE neon with an RX570 (Mesa 19.2.8). Doesn't even help to launch the game with -gl, already got two abandons because I didn't manage to restart Dota within five minutes after a crash, even though I had killed all steam processes in htop.
How to fix this mess?

You can probably turn off background shader processing for the time being.

arcsaber commented Jun 14, 2020

You can probably turn off background shader processing for the time being.

I never had it turned on in the first place. I updated to latest oibaf drivers, let the pregame shader processing run through and am going to test this in low priority now.

rLy07 commented Jun 14, 2020

You can probably turn off background shader processing for the time being.

I never had it turned on in the first place. I updated to latest oibaf drivers, let the pregame shader processing run through and am going to test this in low priority now.

I don't have Dota so I don't know it then. Does it have a separate thing for shader processing from the steam wide one that was introduced recently? Or does the shader processing supposed to run at the start of a game? I haven't played SotTR for a while so I don't know if it would run at the start and I haven't really noticed it with other games but those usually only take a few seconds to compile.

Does it have a separate thing for shader processing from the steam wide one that was introduced recently?

No I have never seen this shader processing until recently. I made the following observation, there are two types of launches for me:

I start Dota and the shader processing basically skips itself after a few seconds, I can't see any fossilize process in htop
I start Dota and fossilize tries to do its thing, but after around 40 minutes it's at around 15% and has already filled up my 16gig ram and further is 1gig into swap. I cancelled it.

So I have to suspect that when I skip shader processing, rarely fossilize starts itself in the background and wrecks my system. I'll occasionally check htop while playing and edit this message if I'll witness anything.
Edit: yep, after a fresh restart I open steam, start Dota, shader processing closes itself after about five seconds, but about five minutes later i found that fossilize is already at 4gig ram and I killed it.

Samega7Cattac commented Jun 15, 2020

@arcsaber there's an option on dota settings to disable "compute shaders", try that

arcsaber commented Jun 15, 2020

@arcsaber there's an option on dota settings to disable "compute shaders", try that

Thanks, I tried but after launching Dota fossilize still starts.
On a side note, once I close Dota (e.g. if there's an update), I can't launch it anymore, neither can I stop steam, have to kill the process and restart steam. But that's less of a nuisance.

Plagman commented Jun 15, 2020

When you observe this, do you see the memory usage after Dota has started and when it's shown its window? Are there any contents in the window or just black?

When that happens do you see the memory being consumed by fossilize-replay processes, or Dota itself?

I narrowed down a few things:

When Fossilize does not use its own pipeline cache, Mesa maintains an internal pipeline cache in memory which becomes massive over time.
RADV_DEBUG=nomemorycache is a workaround.

HansKristian-Work commented Jun 23, 2020

Someone reported on Discord that the issue is resolved now, so closing.

DarkDefender commented Jun 23, 2020

@HansKristian-Work Any ETA for when it will be in the steam beta?

The reporter said it was already in Steam, I don't know anything beyond that.

DarkDefender commented Jun 23, 2020

Hmm, I still have the issue then. I just tried 2 hours ago. (Steam says that I'm up to date, and I'm opted into the beta client program)
It seems like it is still slowly eating all my available ram when building the shaders for dota2.

Saroumane commented Jun 28, 2020

Today, 10:40 AM : PC started (cold boot) : Ubuntu 20.04, Radeon RX5700 XT. Steam autostarted.
. : browse web
11h14 : all RAM and all swap are used by "fossilize_replay", has to quit steam.

1st time I see this.

DarkDefender commented Jul 6, 2020

I just wanted to chime in and say it seems to be solved for me now.
It is both a lot faster and only takes around 1 GB of ram or so it seems.

I'll leave this open so others can confirm as well.

The hero selection screen didn't load for me, more specifically only the top row of empty rectangles loaded and I had sound (you may now select your hero), screen was black otherwise. I checked processes and found fossilize_replay. I got an abandon because I couldn't reconnect quickly enough and have to wait an hour. =( Seems like I have to check for fossilize every single time.

Edit: I also got low priority, so because of this mess I have/had to play at least five matches in low priority, that's at least around 2,5 hours of my life wasted in toxic environments for bugs which are out of my control. Come on guys.

I seem to be having this issue with BL3, I have 32GB RAM though, and my machine is so locked up that the keyboard LEDs are not working. I'll give the sched_autogroup_enabled change a try.

Update: made no difference. I have 7 instances of fossilize_replay trying to use 4GB RAM each and 5 of them are waiting on disk.

20 minutes in, nothing has changed. The machine is now seemingly totally locked up. I can't SSH in from anywhere. My only options are to wait, or force a reboot.

Galcian79 commented Dec 9, 2020

Same issue here on Manjaro Linux 5.9.11
Ryzen 5 2600
16 GB RAM
GTX 1060 6GB driver nvidia 455.45

Conky shows a sudden surge in RAM usage with four fossilize_replay processes running at a time. Then nohang starts killing background processes to prevent system hang. Disabling backgroud vulkan shader compiling solved the issue.

VolatileDream commented Dec 9, 2020

@kakra is there a way to know a minimum RAM requirement for any given invocation of fossilize_replay? If so the parallelism could be adjusted to ensure it never eats up more than 80% of free RAM in the system, which would help with oom-killer lockup.

I could probably be more methodical in logging the fossilize_replay executions under prlimit with RAM limits, but I don't think that it's caused me any problems. What would need to happen to validate that fossilize_replay correctly handles ENOMEM?

HelmicNewciv commented Dec 11, 2020

Same issue, I try starting Warframe or Monster Hunter World and a bunch of fossilize_replay processes will eat my RAM and freeze up my system while attempting to compile Vulkan shaders. The process for the game can literally take overnight, and the game won't end up actually launching.

Manjaro Linux: Nibia 20.2
AMD FX(tm)-8350 Eight-Core Processor
Nvidia GTX 1070 8 GB, 455.45
Linux tkg kernel 5.9.13

StarrKiss commented Dec 12, 2020

This is happening here, too, on Manjaro Linux latest with Control. I've been watching the ram charts- instant fozzilize_replay comes up, it rapidly increases until it takes all 16 gigs of ram.

felschr commented Dec 13, 2020

This happens to me with Death Stranding.
My system:
NixOS (nixos-unstable)
GNOME 3.38.2 on X11
GTX 1080 Ti
AMD Ryzen 1700
32 GB RAM

jfdsmit commented Dec 15, 2020

I'm seeing a pattern of nvidia cards in all reports in the last week, and that's also what I have. It has been confirmed earlier that it's the nvidia driver doing the allocations that use all the memory. Does anyone have experience with this issue not occurring with their games earlier? If so, we might be able to pinpoint from which driver release this started happening.

I have submitted a bug report to nvidia a month ago, but no response :(

I'm seeing a pattern of nvidia cards in all reports in the last week, and that's also what I have. It has been confirmed earlier that it's the nvidia driver doing the allocations that use all the memory. Does anyone have experience with this issue not occurring with their games earlier? If so, we might be able to pinpoint from which driver release this started happening.

I can report that I had no such issues until suddenly with Deep Rock Galactic after an update of theirs which changed something in/with the game engine (I believe) in October. I was then running v440 on my Debian system.
I tried upgrading to 450 to fix it, to no avail.

I have submitted a bug report to nvidia a month ago, but no response :(

(y) Nice - let us know how it progresses. Share link if you posted on forum perhaps?

I'm suite sure Valve is working with NVIDIA om this issue. In one of the latest Steam betas, fossilize was disabled entirely for NVIDIA users.

jfdsmit commented Dec 15, 2020

I have submitted a bug report to nvidia a month ago, but no response :(

(y) Nice - let us know how it progresses. Share link if you posted on forum perhaps?

I used e-mail due to my general aversion of creating accounts, I might still do it if you think a forum post would get more traction?

jfdsmit commented Dec 15, 2020

I'm suite sure Valve is working with NVIDIA om this issue. In one of the latest Steam betas, fossilize was disabled entirely for NVIDIA users.

I was already wondering why I saw no "Processing shaders" dialogue yesterday 😅

kakra added a commit to kakra/Fossilize that referenced this issue Dec 20, 2020

I seem to be having this issue with BL3, I have 32GB RAM though, and my machine is so locked up that the keyboard LEDs are not working

This seems to be an effect by coincidence: For me, BL3 is just the first game that is processed - every time. But the issue also occurs with other games.

At least my patch referenced above keeps write-back mostly under control but this is just a primitive idea yet, it needs some more tuning to adapt better to system load changes, maybe it should synchronize between all of the processes somehow. But maybe you could give it a try and share your thoughts?

I got this when booting up monster hunter after deleting my cache and letting steam download them again.

It should theoretically solve shader compilation stutter.

This is the answer. Put it over the speculation.

well I run steam on my integrated gpu (carrizo) and games on my discrete gpu (polaris12) with gamemoderun or DRI_PRIME can it cause any issues?

Can someone ELI5 this for me? Why do Vulkan games need to compile shaders on a per system configuration basis and does DX12 have a similar thing going on?

Right, it’s a load time optimization not a “runtime” optimization. It may affect runtime, but that’s because of the dynamic compile.

I mean, bl3 is poorly optimized

I really wish my non steam games on lutris could take advantage of this

It’s not a runtime performance thing, it’s a load time performance optimization.

Well, to be specific, the shaders built for your hardware are a runtime performance gain, but the cache is to prevent having to generate shaders for your hardware at load time which can be significant.

If you want to see a difference, you need to test the load times. Its hypothetically possible that the difference is moot on a fast enough setup, but that’s the intent.

Any way to force this to occur with Overwatch?

For Nvidia you can specify a shader file name (to identify it) with:

__GL_SHADER_DISK_CACHE_APP_NAME=<app_name> and a shader path with __GL_SHADER_DISK_CACHE_PATH

And the fossilize file path with:

STEAM_FOSSILIZE_DUMP_PATH or ( FOSSILIZE_DUMP_PATH if you compiled fossilize)

Then launch the game. You should get a .foz file in the dump path.

Now you can recreate the shaders with fossilize-replay from $HOME/.steam/root/ubuntu12_64/fossilize_replay (or /usr/bin/fossilize-replay ) with the same __GL_* variables:

Читайте также: