VBLFC event over, more cloud optimizations, new hands, Neos Festa 3 coming soon

Hello everyone and welcome to another weekly update!

This one is  going to be relatively short, as we were busy with the VBLFC over past  week, the first official online convention held on our platform. It has  broken the record number of concurrent users, reaching 432 peak on  Steam.

We also look at some of the optimizations and effects on  the cloud infrastructure with the sudden surge in users and how they  helped to keep the cloud stable and make it more efficient in long term.

The  creator festival Neos Festa is also coming next month for its third  installment, held by the Japanese team. If you'd like to have an  exhibit, you can find more information at festa.neos.com!

VBLFC

During  the last Friday through Sunday, Neos saw its official online  convention, the Virtual Biggest Little Furry Convention. This event  brought hundreds of new users and broke our all time record for number  of concurrent users! We reached 432 peak concurrent users on Steam, with  our metrics going slightly above 500 with all accounts considered  (including headless).



We’d  like to extend a huge thanks to the organizers for choosing Neos,  building beautiful worlds and a number of interacted gadgets and  accessories for the attendants of the event and bringing a number of  guests to serve as entertainment for the program. So again, big thanks  to everyone for their hard work making this event happen and thanks for  everyone participating!



VBLFC  plans to reopen the worlds during the physical convention in October,  so if you missed them, you’ll be able to revisit them again! The VBLFC  suite world is open to the public already and we’ve already seen many  people decorate it and make their own versions of it, so feel free if  you haven’t!

https://youtu.be/NA7tyKAYlUA

Analyzing Cloud Usage & Further Optimizations

As  many people started joining during the VBLFC Opening Ceremony, we  started noticing the cloud response times sharply rising up due to the  sudden increase of users. This has resulted in noticeable slowness of  much of the cloud functionality.

To compensate, we started  scaling up (and out) the infrastructure beyond our default limits, which  brought it down to usable levels, but still far from ideal. Based on  the live metrics, we began optimizing the biggest culprits that resulted  in slow responses and releasing them to the cloud as the event was  ongoing.

Overall  response time during the first night. The sharp increase is when people  join in, the decrease is when the scaling up kicks in. Afterwards the  drops and smoothing out are due to optimizations being deployed.

The  first bits addressed some quick low hanging fruit, like unnecessary  database queries and adding some bits of extra caching. This has  smoothed out some of the response times, but still wasn’t sufficient to  make it snappy.

Following that, we implemented a quick  optimization for the user status update system - the one responsible for  updating status of your contacts - whether they’re online, which world  they’re in and so on. This system was the second biggest contributor in  the cloud load after the messaging system, so it was a prime target for  the optimization.

Once this update was pushed out, to our  surprise the response times have dropped more than we expected them to,  under 50 ms on average, despite about 500 people still being on. This  resulted in better response times than before the sudden increase in  active users, at the cost of some precision - status of new contacts  might take a few minutes to show up.

Impact of the status system optimization

Thanks  to this optimization the remainder of the convention was pretty smooth  and without any cloud slowness or interruptions. Those optimizations  still help even after the event has finished. Based on their  effectiveness, we’ll be moving the status updates to SignalR next,  similarly to the messaging system, which will make it significantly more  responsive and improve the cloud scaling even further.

It is  always a bit scary getting a sharp increase in users, as it doesn’t give  much room for the infrastructure to adjust, but this event has given us  a wealth of information on how it behaves with a quickly rising user  base and which parts become bottlenecks and need to be redesigned.

With  this, we’ll be better prepared for other big events in the future and  just general growth of the Neos community, at least from the cloud  perspective.

Integrating SignalR for messaging system

As  our community keeps growing, we have started noticing the cloud  infrastructure being heavily taxed during the busy days, causing the  cloud service to become less responsive. With some big events coming up,  we decided to prioritize integration of SignalR push notification  service, which was on the roadmap for a while, to improve how well the  system scales and introduce some new capabilities.

The in-game  messaging system has been redesigned to use this service. Instead of  repeated REST (HTTPs) requests, all the communication is done via  WebSockets. This is significantly more efficient for both your  connection, as well as on the cloud service - the previous system was  responsible for using the majority of the system’s throughput!

Thanks  to the new system, the messaging system should be significantly more  reliable and robust in the long term (after fixing some initial bugs and  issues), decreasing the latency of the messages and cases where they  wouldn’t be delivered at all or be slow to respond.

Decreased database usage after optimizations. Shows last 7 days.

The  system also enables some new features, like getting message read  notifications (can be disabled in settings based on community request)  and in the future message writing or voice message recording  notifications.

The service is also planned to be used for more as  well in the future, notably updating the session status, current  profile status and synchronizing cloud variables fully in real time  across different clients. You can check out more at our roadmap on  GitHub.

More cloud optimizations

When  working on the cloud, we found out more underlying causes of issues  that were resulting in some slowness, particular slow message loading,  due to the database queries being highly suboptimal. Those have now been  optimized, resulting in significantly more efficient queries and much  quicker loading of message history as a result.

Server  usage split by endpoint. The orange line is fetching new messages. You  can see the significant decrease after SignalR release, but still some  large spikes. Those were addressed near the end of the grpah.

Some  other parts of the cloud were tweaked and improved as well to be more  robust, improving the rate limiting system with a new solution based on  Redis and adding fallbacks in case of transient connectivity issues.

Overall  we hope those changes will help resolve a number of issues that you  might’ve been experiencing in the past, but we’ll keep monitoring the  system and making more improvements as the community keeps growing.


If you run into any problems, let us know either on Discord or our GitHub!

Security and moderation improvements

Our  moderation tools have gotten some improvements as well based on some  community reports and incidents. Notably we have expanded the ban  evasion detection system to cover more loopholes in the system and used  the SignalR service so any global bans take effect immediately.

We  have also added a configuration option that allows restricting the  functionality of the built-in File Browser so it can only access certain  paths on your system or none at all. Simply setup an array of paths  under pathWhitelist in Config.json file in the Neos installation.  Providing an empty array (“pathWhitelist: [ ]”) will disable the File Browser completely.

This  can be useful for additional security or when running Neos at public  events, schools and other places where you don’t want the visitors/users  to access all the files from within VR.

New Neos Hands

Next  time you start Neos, you'll see the update I just pushed for the new  default avatar. These new hands were designed to match the headset, and  the headset itself was updated with a new Neos logo on the back.

Every  time you apply the default avatar, you are normally assigned a random  color. However, now you can choose to assign yourself a persistent color  that can be used beyond your default avatar using the cloud variable  G-Neos.CustomUserColor!

Please note that this cloud variable's name and function is subject to change.

The whisper bubble's visuals have also been re-imagined as a ring, to be more light and less isolating.

Community Highlights

Hello  everyone, friendly neighborhood Turk here! I hope your week has been  wonderful! It’s been jam packed these past 2 weeks mixed with all the  stuff that’s been going on! So I hope you have been having fun with all  the things going on! This month has all kinds of community events and  moments going on so here’s some of them for you to see!

Firr’s New Homeworld

VBLFC  has come and gone in the community, and there were some fun times! But  for one last hurrah it seems like Firr needs a new homeworld! So  everyone quite literally had come into his world and trashed his room.  Whether he makes this his homeworld is a thing to be seen, but boy is  this headless session/world a treat. There is ALOT of memes.

Neos Festa 3

Neos  Festa 3 is soon on the way! Neos Festa is a creator showcase event,  where folks can submit content and applications to showcase their works  in Neos. Neos Festa has been quite the staple in our community bringing  many games, toys, and avatars to the community enriching the community  in ways we can’t expect. So if you want people to check our work, and  see what people think feel free to submit! Find out more details on the  Discord, or in the submission world. You can also find more stuff at the  website festa.neos.com.

Thank you to everyone who’s collaborating to make this event possible!

What's next

After  the event, we're continuing main focus on BEPUv2 integration. We have  already finalized all collider types, getting mesh colliders and convex  hulls to also work with the asset variant system, precomputing and  caching the data, instead of generating it on the fly every time you  load into the world.

There's still quite more to do. We'll also  be interweaving some more tasks and updates in between. You can always  find the latest progress in the official Discord in the #devlog channel  or check out the Roadmaps on GitHub.

And as always, huge thanks  for everyone's support! Thanks to your passion and creativity, this  place keeps growing and becoming better every day! See you next week!