Pragma - A 3D multiplayer sandbox engine with Lua-integration, based on Vulkan and bullet physics

[Lyokanthrope]

Wanted to check this out and tinker around
Crashing outright when loading the engine though, not sure why. RX 480, latest drivers, etc.
[url]https://www.dropbox.com/s/ewdgnj20tmynijv/crashdump_2017-08-01_19-45-47.zip?dl=0[/url]

[Nastardos]

Just wanted to say, keep up the good work! You've done amazing things in short time span that i wish would have been on garrysmod.

[paindoc]

[QUOTE=Lyokanthrope;52530331]Wanted to check this out and tinker around
Crashing outright when loading the engine though, not sure why. RX 480, latest drivers, etc.
[url]https://www.dropbox.com/s/ewdgnj20tmynijv/crashdump_2017-08-01_19-45-47.zip?dl=0[/url][/QUOTE]

Reading the log, it looks like an error getting the max MSAA sample count for a certain image format: checking the Vulkan hardware database it looks like the 480 doesn't have a huge range of supported texture formats, but this should be something Silverlan can patch quickly

if there's one thing that especially hurts about Vulkan, its things like this. Can you disable antialiasing entirely and see if that does anything? The settings initialization clearly didn't complete but if there's some sort of config.ini you might be able to get past it.

[editline]edited[/editline]

okay questions i had for silverlan:

- How do you currently handle memory allocations? I quickly ran into the vkDeviceMemory limit when allocating the primitive way, but found a custom allocator fixed that. I'm not sure how it'll scale beyond tech demos, though, and I'm worried about defragging for memory used with frequently loaded/unloaded assets.
- Have you considered trying dynamic shader editing? Since glslang can be plugged into an application (hell, dolphin-emu does it at runtime), I'm wondering if it'd be possible to allow for that. It would require rebuilding the pipeline though, and I'm not sure if the pipeline cache data would have to be flushed/cleared or not
- How do you handle lighting and fancy things like deferred rendering? Getting the subpass dependencies for this working has me pulling my hair out.
- Is your water simulation using a compute shader?

if you ever open-source this I'm probably going to drool a ridiculous amount, I'd love to see code like yours. I've had to learn on-the-job by going through wretched matlab and C++ code so I need more good code in my life.

if you do ever want help with tracking graphics bugs or something, though, let me know. Even if its just testing new things or trying to break graphics stuff.

[Silverlan]

[QUOTE=Lyokanthrope;52530331]Wanted to check this out and tinker around
Crashing outright when loading the engine though, not sure why. RX 480, latest drivers, etc.
[url]https://www.dropbox.com/s/ewdgnj20tmynijv/crashdump_2017-08-01_19-45-47.zip?dl=0[/url][/QUOTE]
Published a new update which should fix the crash, but it won't fix the underlying issue. I've added you on steam.

[QUOTE=paindoc;52530498]Reading the log, it looks like an error getting the max MSAA sample count for a certain image format: checking the Vulkan hardware database it looks like the 480 doesn't have a huge range of supported texture formats, but this should be something Silverlan can patch quickly[/QUOTE]
The wide-range of unsupported texture formats is completely normal, only a handful are important (R32G32B32A32, BC1, BC3, BC5, ...).
Not quite sure why he's crashing, looking at the dump, everything seems to be fine.

[QUOTE=paindoc;52530498]- How do you currently handle memory allocations? I quickly ran into the vkDeviceMemory limit when allocating the primitive way, but found a custom allocator fixed that. I'm not sure how it'll scale beyond tech demos, though, and I'm worried about defragging for memory used with frequently loaded/unloaded assets.[/QUOTE]
I have several different ways to handle memory allocation, depending on the context. Far more important than memory allocation is handling buffers correctly imo. A few very large buffers are generally better than many small ones, and in many cases you already know how large the data is going to be, so there is never any fragmentation to begin with. I'd suggest you take a look at the stuff from [URL=https://github.com/SaschaWillems/Vulkan]sascha willems[/URL], he has all sorts of examples on how to use Vulkan properly.

[QUOTE=paindoc;52530498]- Have you considered trying dynamic shader editing? Since glslang can be plugged into an application (hell, dolphin-emu does it at runtime), I'm wondering if it'd be possible to allow for that. It would require rebuilding the pipeline though, and I'm not sure if the pipeline cache data would have to be flushed/cleared or not[/QUOTE]
You can already do that.
[video]https://youtu.be/4-nRVDXyWcE[/video]
(Second half of the video.)

You can also write custom shaders in Lua (in combination with GLSL/HLSL code of course).

[QUOTE=paindoc;52530498]- How do you handle lighting and fancy things like deferred rendering? Getting the subpass dependencies for this working has me pulling my hair out.[/QUOTE]
There is no deferred rendering. I've considered using deferred rendering several times, but there are just too many downsides imo, it just isn't worth it.
I'm using forward rendering, which means only about 6 light sources in the view frustum can be used for rendering at a time. For more lights, I'd have to render the scene several times, haven't decided what I'm going to do about that yet.

[QUOTE=paindoc;52530498]- Is your water simulation using a compute shader?[/QUOTE]
Yeah, but it's only really useable for small areas of water right now, since it's incredibly expensive. I'm not sure if this is fixable either, I don't know any games that actually do proper water simulation for larger areas (Pretty sure games like witcher 3 pre-simulate it).
All of the data also needs to be host-readable (=slow), since I need it to calculate buoyancy.

[QUOTE=paindoc;52530498]
if you do ever want help with tracking graphics bugs or something, though, let me know. Even if its just testing new things or trying to break graphics stuff.[/QUOTE]
My main problem is usually with optimizing anything graphics-related, but I'm satisfied with the performance for the time being. My shaders aren't any good though. :v:

[paindoc]

[QUOTE=Silverlan;52531250]The wide-range of unsupported texture formats is completely normal, only a handful are important (R32G32B32A32, BC1, BC3, BC5, ...).
Not quite sure why he's crashing, looking at the dump, everything seems to be fine.[/QUOTE]

I can't claim to know any better, tbh. I've not had any deployed/used software with my renderer so I've not yet figured out good ways to handle that.

[QUOTE=Silverlan;52531250]I have several different ways to handle memory allocation, depending on the context. Far more important than memory allocation is handling buffers correctly imo. A few very large buffers are generally better than many small ones, and in many cases you already know how large the data is going to be, so there is never any fragmentation to begin with. I'd suggest you take a look at the stuff from [URL=https://github.com/SaschaWillems/Vulkan]sascha willems[/URL], he has all sorts of examples on how to use Vulkan properly.[/QUOTE]

Ah, well silly levels of optimization are my thing. Its why I spent so much time stuck on converting the GPU-Open memory allocator to work for me, but it does so quite well now: [URL="https://github.com/fuchstraumer/VulpesRender/blob/master/include/resource/Allocator.h"]header[/URL] / [URL="https://github.com/fuchstraumer/VulpesRender/blob/master/src/resource/Allocator.cpp"]source[/URL]

tldr is I just allocate a couple big buffers that are half of the ideal block size, and then bind resources to sub-regions of that. Its working quite well so far and has honestly abstracted away much of the boilerplate resource/memory management code for me (especially for allocating buffers/images). Only problem is that buffer-image granularity and small resource alignment offsets can slowly fragment the memory, so you end up with lots of little 256-1024 byte unused suballocations sitting around in your VkDeviceMemory objects. This seems to be what you do, based on your answer though. I'm thinking of splitting it again, by dedicated certain pools to certain sizes alongside certain memory types. That way all my small UBO objects go in their own pools, middling buffers go in one, and really big VBOs/EBOs go in their own dedicated pool.

GPU-Open released a defrag routine with their allocator, but I'd rather just make the system more robust tbh.

[QUOTE=Silverlan;52531250]You can already do that.
-vid-
(Second half of the video.)

You can also write custom shaders in Lua (in combination with GLSL/HLSL code of course)[/QUOTE]

I somehow completely missed this tbh, despite thinking the video might hint to such capabilities. I actually just checked your reply taking a break from working on my renderer - where I'm implementing a shader compiler now, haha. Any protips in regards to handling things like pipeline caches? I'm trying to figure out a smart way to handle that. Includes too - I'm thinking of caching used include files at runtime in a hashmap with the "#include" as the key and the actual imported code as the value so I don't reopen and read already used files. Beyond that I'm not far enough in implementation yet to figure things out.

[QUOTE=Silverlan;52531250]Yeah, but it's only really useable for small areas of water right now, since it's incredibly expensive. I'm not sure if this is fixable either, I don't know any games that actually do proper water simulation for larger areas (Pretty sure games like witcher 3 pre-simulate it).
All of the data also needs to be host-readable (=slow), since I need it to calculate buoyancy.
[/QUOTE]

Compute stuff is tricky: if you want help with this in particular, feel free to let me know. I spent a lot of time working with CUDA for a course before I dropped out of school and made a [URL="https://github.com/fuchstraumer/CUDA_Noise"]vaguely awful procedural noise library[/URL] for the course project. The biggest thing was register pressure and tweaking the workgroup/block size. Either I had too many threads for kernels/invocations requiring lots of register space (thus creating tremendous register pressure) or I reached pitifully low occupancy levels running a kernel. The tldr of my CUDA experience is this: don't :v:

[QUOTE=Silverlan;52531250]There is no deferred rendering. I've considered using deferred rendering several times, but there are just too many downsides imo, it just isn't worth it.
I'm using forward rendering, which means only about 6 light sources in the view frustum can be used for rendering at a time. For more lights, I'd have to render the scene several times, haven't decided what I'm going to do about that yet.[/QUOTE]

I don't blame you haha, I'm starting to think its not at all worth it either. It did seem interesting, though, with how the Vulkan Programming guide described the benefits of subpasses and how the GPU can optimize those. I just love exploiting low-level hardware features but there comes a point where doing so requires a degree of masochism and self-loathing that even I fail to muster.

Cheers for the answers! Its not often I get to discuss vulkan with someone in such detail, by any means. I'm still chewing through a number of Sascha's examples: I like some stuff he does a lot, but I also don't like some stuff he does. He's far more experienced than I am, though, and there's still a lot of things I have to implement/experiment with before I consider my renderer even nearly "complete".

[Silverlan]

[QUOTE=paindoc;52531281]I can't claim to know any better, tbh. I've not had any deployed/used software with my renderer so I've not yet figured out good ways to handle that.[/QUOTE]
Might want to watch out for that, some formats you'd expect to be supported by most GPUs are actually very poorly supported, like R32G32B32 (without an alpha channel, not supported on most Nvidia GPUs).

[QUOTE=paindoc;52531281]I somehow completely missed this tbh, despite thinking the video might hint to such capabilities. I actually just checked your reply taking a break from working on my renderer - where I'm implementing a shader compiler now, haha. Any protips in regards to handling things like pipeline caches? I'm trying to figure out a smart way to handle that. Includes too - I'm thinking of caching used include files at runtime in a hashmap with the "#include" as the key and the actual imported code as the value so I don't reopen and read already used files. Beyond that I'm not far enough in implementation yet to figure things out.[/QUOTE]
My #includes are basically just copy & paste from the respective files. The shader code is converted to SPIR-V anyway and any duplicate shader code is ignored because of the include guards, so it hardly matters.
The pipelines are completely recreated and all materials which use that shader are then reloaded (=reloading the descriptor sets).
The only thing you can't do on the fly in my implementation is change existing descriptor set or binding point ids, that can get you into trouble.

[jombo23]

Well this is the coolest shit ive seen all week.

[GeXeH]

Making some damn sweet progress on this, keep up the great work. The launcher crashes for me when I press 'Launch' although the game does launch.

I'm having GFX glitches though;

[vid]https://my.mixtape.moe/fjijdz.webm[/vid]

Radeon 7970 GFX card latest AMD drivers.

[Silverlan]

[QUOTE=GeXeH;52577170]I'm having GFX glitches though;[/QUOTE]
Thanks for the video, that helps a lot. :v:
I've just published an update which should fix all of the issues, can you give that a try?
Not sure about the launcher crash though, haven't seen that one before. Can you try turning of background programs (antivirus?) and see if it still crashes?

[GeXeH]

[QUOTE=Silverlan;52579143]Thanks for the video, that helps a lot. :v:
I've just published an update which should fix all of the issues, can you give that a try?
Not sure about the launcher crash though, haven't seen that one before. Can you try turning of background programs (antivirus?) and see if it still crashes?[/QUOTE]

No problem at all, I'm currently at work but will try it tonight for you and report back!

I'm using Windows Defender so nothing out of the norm, I'll try disabling it and report back on that too!

Hopefully from the video you can see that I have some sort of input lag on the constants map in particular, it's like I'm walking on ice and my keyboard presses are delayed. No problem on the other map though.

[Karmah]

I crash when trying to start a game in any map with any gamemode.
Depending on the map/gamemode combination, I sometimes hard crash without the option of saving a crash report. I've managed to collect 3; [URL="http://s000.tinyupload.com/index.php?file_id=03369897012833129897"]Dump 1[/URL], [URL="http://s000.tinyupload.com/?file_id=00419342038461185489"]Dump 2[/URL], [URL="http://s000.tinyupload.com/?file_id=16713301399713058487"]Dump 3[/URL]
The only changes I've made were changing the graphics preset to the highest, resolution to 1920x1080, window no-border, and FOV to 90.

My specs are:
OS: Windows 10 64 bit
RAM: 16gb
CPU: I7-4770K (3.5GHz)
GPU: GTX 1070 (driver 382.05)

IIRC I have the vulkan drivers installed, because a while ago I tried running some vulkan demo that required them.

[Silverlan]

[QUOTE=Karmah;52579612]
GPU: GTX 1070 (driver 382.05) [/QUOTE]
That version is outdated, latest one is 385.28. There's been a lot of Vulkan related bug-fixes in recent Nvidia driver updates, chances are an update might already fix those issues.
You could also try turning MSAA off. (Let me know if that makes a difference)

If neither work, try validating your files (which will reset your settings), then try with the default settings.

[GeXeH]

As promised, here's my findings;

I disabled Windows Defender and pressed the 'Launch' button but was still presented with the error that the launcher had crashed however, Pragma started as normal afterwards. I ran the launcher as Administrator to rule out permissions but still got the error message, afterwards Pragma launched as normal.

When I run the launcher in compatibility mode for Windows 8 the launcher doesn't crash and Pragma launches as it should - Windows 10 related maybe?

You've fixed the menu text not bleeding, as you will see from my video below. Good job!

I'm still getting input delay which is more noticable on the contraints map and as you can see my player is jumping up and down like it's not happy staying on the floor.

On to the lights map, you've certainly fixed the player shadows, it looks awesome now. There's still a few things not working as expected though;

1) The orange box in the center of the map has some sort of shadow lines through it.
2) The global shadow option on the shader toggle button causes some weird kind of flickering for the whole map.
3) The spotlights appear to be fine until I walk in to where they are shining then they magically disappear and reappear, very strange.

[vid]https://my.mixtape.moe/mrcqak.webm[/vid]

Hopefully this helps - good luck, can't wait to see the result!

[lumaio]

I'm having the exact same issues as GeXeH as far as I can tell even the launcher crashing so probably not a windows 10 issue

OS: Windows 7 64 bit
RAM: 20gb
CPU: i5-3470 (3.4GHz)
GPU: GTX 1060 6GB (driver 385.28)

[vid]https://my.mixtape.moe/tgquvy.mp4[/vid]
the stuttering isn't very visible but it's there as well as the slippery movement

[GeXeH]

[QUOTE=Lumaio;52583029]I'm having the exact same issues as GeXeH as far as I can tell even the launcher crashing so probably not a windows 10 issue

OS: Windows 7 64 bit
RAM: 20gb
CPU: i5-3470 (3.4GHz)
GPU: GTX 1060 6GB (driver 385.28)

the stuttering isn't very visible but it's there as well as the slippery movement[/QUOTE]

I'm happy that it isn't just me having these issues, I'm sure our videos will help Silverlan make an awesome stable product.

[exodusthefirst]

After a few test runs, recording a few bugs I ran into a bit of a problem. On a GTX 1080 Ti I crash whenever attempting to load a map with any kind of anti-aliasing on. Driver version 384.94 on Windows 10 Pro N 64-bit.

I am trying updating my drivers now, we'll see where we get with that.

EDIT: Pragma now crashes when getting to Initializing Server for the first time. It's not a reported crash either, Windows error reporting or whatever is what I get. Validating files does not fix the issue. The latest NVidia driver update (385.41) breaks everything. I cannot seem to get any crash reports from Windows Error Reporting either.

[IMG]http://i.imgur.com/qlmNNPy.png[/IMG]

Looking at the dumps from your crash reporting Silverlan, they seem to be in the kb range while the ones from Windows are 160mb for some god awful reason. I'm not sure if that's just what Windows does, or it's a symptom of the crash.

[Zeh Matt]

I'm curious whats with the inconsistent naming? There is Entity:SimpleFunctionName and there is library.weird_underscore_usage_which_nobody_likes, are you planning to change that?

[Silverlan]

[QUOTE=Zeh Matt;52637430]I'm curious whats with the inconsistent naming? There is Entity:SimpleFunctionName and there is library.weird_underscore_usage_which_nobody_likes, are you planning to change that?[/QUOTE]
It's very consistent actually, lowercase underscore for static class functions and library functions, PascalCase for class methods, to distinguish the two more easily.
I don't plan on changing anything about that.

Anyway, I've been working on some big updates for the lighting and rendering system all week, I'll need until the end of the weekend to finish it up.
I'll try to get the update out on monday and record some videos.

[Tangyboxhead]

Thank you for this project! So far it is very swell and impressive for what you have done.

I plan on making a map or two right now, and maybe even a vehicle.

Will this lighting and rendering system improve performance? that is the one area that hurts a bit here.

Loving the no limit maps and the fact that I can utilize sources tools for work on this though!

Keep up the good work.

[Silverlan]

[QUOTE=Tangyboxhead;52645802]I plan on making a map or two right now, and maybe even a vehicle.[/QUOTE]
Might want to wait on that, I'm currently working on adding [URL=https://facepunch.com/showthread.php?t=1572105&p=52607928&viewfull=1#post52607928]BSP support[/URL] and the vehicle system doesn't work very well at the moment, I'm planning on reworking that some time soon.

[QUOTE=Tangyboxhead;52645802]Will this lighting and rendering system improve performance? that is the one area that hurts a bit here.[/QUOTE]
Depends, where are you getting performance problems? What's your GPU? Drivers up to date?
The new update will massively improve performance in scenes with a lot of light sources and I've also improved water rendering performance quite a bit, as well as general rendering performance.

[Tangyboxhead]

Oh ok.
(Im crazyboxhead on steam btw, I make train mods and hopefully maps, Pragma shows a source like attitude but without its limits (like size, which is a detriment to train maps), which is what I find fascinating!)

I did make a map, but it doesn't work right(lighting ect).
Performance issues were more likely a blunder on my part, restarting the game/engine fixed it.

As for vehicles(the ones i am talking about), they are just entities that slide, so no actual tradition vehicle physics.
As for BSP support I assume that it will hold RAD and baked texture support? Or nah?

Also I noticed you were starting the level editor that was similar to hammer, how's progress?

Thanks.

[conna]

Hey Silverlan, do you have any plans to make it easy for gamemode creators to "package" up a version of Pragma pre-built with their gamemode ready for distribution or something like that? Or is Pragma Engine in itself destined to be its own game / sandbox?

[Gmod4ever]

[QUOTE=Conna;52651784]Hey Silverlan, do you have any plans to make it easy for gamemode creators to "package" up a version of Pragma pre-built with their gamemode ready for distribution or something like that? Or is Pragma Engine in itself destined to be its own game / sandbox?[/QUOTE]

I'm pretty certain Pragma will have the same relation to gamemodes / mods as Garry's Mod does: EG people are expected to already have an installation of Pragma, and then just load the gamemodes / mods onto it. As opposed to having Pragma packaged with gamemodes / mods.

I can't see a situation like Love2D, where there is native support for bundling the engine with the games, simply due to the size.

That being said, if Pragma ends up being a free distribution, then there's nothing stopping people from bundling the engine with their gamemodes / mods anyways.

Not speaking for Silverlan, though. Just my own thoughts on the matter.

[Tangyboxhead]

I have been trying to make Mods on Pragma for a few days. It is extremely similar to GMod, as well as relatively simple to package mods and publish them.

Structures are similar too, folder names being same or similar.
Lua structure being similar, even in syntax.

It isn't an exact replica of GMod however, there are definitely things to learn, but if you have skills in GMod, you will have em in Pragma no time.

For example, here is a Map Mod I am making:
[url]https://mods.pragma-engine.com/index.php?mode=upload&uploadid=92d205e4923e553d6b1f38f360ba1538[/url]

(Not the best example)

Very similar to make as it would be in GMOD.

[exodusthefirst]

It's lua bindings are essentially GMod, just with things like UI being changed to how modern engines do things. I've been working on a gamemode the past few days and my workflow is essentially the same.

[Silverlan]

[QUOTE=Conna;52651784]Hey Silverlan, do you have any plans to make it easy for gamemode creators to "package" up a version of Pragma pre-built with their gamemode ready for distribution or something like that? Or is Pragma Engine in itself destined to be its own game / sandbox?[/QUOTE]
I'm not entirely sure.
The last thing I want is fragmentation, e.g. there being multiple branches of the engine and some addons only end up working with specific versions or something like that.
I'm open to ideas though, I can understand people wanting to ship their gamemode standalone.

[QUOTE=Gmod4ever;52651796]
I can't see a situation like Love2D, where there is native support for bundling the engine with the games, simply due to the size.[/QUOTE]
The size could be reduced a lot by stripping the engine to the bare minimum. The official version contains a bunch of stuff that isn't really needed if you're creating a custom gamemode (e.g. sdk files, test-maps, models, etc.).

[QUOTE=Gmod4ever;52651796]That being said, if Pragma ends up being a free distribution, then there's nothing stopping people from bundling the engine with their gamemodes / mods anyways.[/QUOTE]
In general I don't have a problem with that. I might try to get it onto steam at some point though (it'll still be free of course), so not sure how to handle that then. (I mostly want to get on steam for the steamworks API, there's a bunch of stuff that would really come in handy. (e.g. voice chat))

[EDITLINE]-[/EDITLINE]
Almost done with the new update.
[T]http://sciolyte.com/sharex/pragma_2017-09-05_16-52-11.png[/T] [T]http://sciolyte.com/sharex/pragma_2017-09-05_16-52-50.jpg[/T] [T]http://sciolyte.com/sharex/pragma_2017-09-03_22-06-22.png[/T]

[Azalia]

Would be nice to have a Discord to talk with you and all that and see announcements for updates and etc. Kinda like a centralized page for all the new updates.

[Silverlan]

[QUOTE=Azalia;52686239]Would be nice to have a Discord to talk with you and all that and see announcements for updates and etc. Kinda like a centralized page for all the new updates.[/QUOTE]
I've set one up since I've gotten so many requests for it: [url]https://discord.gg/Ck5BcCz[/url]
I don't have a mic and I've never used Discord before, but we'll see how it goes.

[Azalia]

[QUOTE=Silverlan;52686671]I've set one up since I've gotten so many requests for it: [url]https://discord.gg/Ck5BcCz[/url]
I don't have a mic and I've never used Discord before, but we'll see how it goes.[/QUOTE]

Don't need a mic to have fun with other people! Just a comfortable community to talk to. :3

[pleasure]

How about addons.That idea is alive?