Alt-F4 #41 - Title loading, please stand by ... 09-07-2021
This week, for issue #41, oof2win2 returns with a somewhat more technical article about what exactly can go wrong when trying to connect to multiplayer games and how to potentially solve these issues. If you’ve ever wondered why Factorio wouldn’t let you join your friends’ multiplayer game, this article might have some explanations for you!
Quick Performance Check oof2win2
If you don’t have a state-of-the-art computer, you probably have seen a message like this saying that you were dropped from the game because your network or computer was not fast enough. Well, what does that actually mean? What can you do about it in the short and long term?
Network Speed Issues
Sometimes, your network may not be the best. It can be due to many things, such as maintenance from your ISP or a power outage of your DNS’s datacenters. There are a few utilities that can however help mitigate these sorts of issues.
To explain how Factorio’s networking works, however, is a longer process. cube has explained this process nicely in FFF#136. One of the most significant things they have done is using UDP but adding back things that are in TCP. UDP has the advantage of being much faster than conventional TCP (which is used for most day-to-day things). UDP achieves this speed by not having a “handshake”, which is two clients (computers) agreeing that they want to communicate together and establishing a connection, which simply means it doesn’t care about what happens with the data, it just sends it. This however also removes the ability to guarantee the delivery of data, ordering the data properly or removing data duplicates (which can sometimes happen).
The fact that UDP removes the guarantee of delivering data however directly contradicts how Factorio must work using fully-deterministic, lockstep algorithms as I explained in an earlier post, as the data must arrive or you would recieve a desync. cube said that Wube have solved this with implementing parts of the TCP protocol into their version of UDP. This means that they still have the speed that they want, but they can also have the reliability of TCP - or close to it, as the Factorio protocol is not publicly available for people to analyze. Hornwitser however has had a go at analyzing the traffic with his dissector, which even helped fix a bug that was in the game since 0.17!
UDP is also very important because it allows NAT punching to work. NAT punching is the process of your client asking an external server (Pingpong server, which is a server hosted by Factorio that serves just this purpose) to tell the Factorio game server to establish a connection with your client, because some routers and firewalls may block “random” connections from IPs that you have not requested anything from, which is good for preventing someone from brute-forcing your computer, but is also an obstacle for the Factorio developers.
Now, let’s try fixing the problem of Factorio dropping you from the game. First, you can try a quick speed test to check whether your network is fast enough using sites such as Ookla’s Speedtest or fast.com (Netflix). Both sites will measure similar data, so pick your poison. There are however three things that matter (referencing Ookla’s Speedtest): download speed, upload speed, and latency.
For Factorio, your download and upload speeds don’t need to be very high, as the map (savefile) is sent over to you when you connect to a server. The speed of the map downloading depends on your download speed and the server’s upload speed; you can’t download something faster than it is being uploaded. Generally, savefiles are within 50MB (if you don’t run with seriously modded games), so on an average connection, a save would be downloaded within 15 seconds. It doesn’t really matter how fast a save downloads though, since it is just a file transfer in the end. The speed of your download does however affect for how many ticks you will be catching up (the part of loading the map after you download it). Generally speaking, anything should be fine, but mileage may vary.
“Catching up” is a process very specific to Factorio. It means that you are catching up to the new actions in the game, because you have an older copy and the server can’t instantaneously send you a completely new copy of the save for you to run from. Instead, the process of joining a multiplayer session looks something like this:
- The server saves the map at a specific tick, let’s say tick 100 and sends the saved map to you. The server also starts sending you all changes from now on (user inputs only, as described in my other article on Factorio multiplayer itself).
- You download the map. However, you don’t have a good internet connection, so the download takes up 90 ticks during which other players have played on the game and have created other things.
- Since you are 90 ticks behind the current state of the map, you cannot join the game. The server has however sent you every action done by players, so your client begins simulating the game and adds in the actions at the correct ticks. When the game is simulated to be at the same tick as the server’s copy, you can finally join the game.
There can however be an issue at step 3. Sometimes, the progress bar called “catching up” will not move, but stay in the same place. The issue with that is that you are simulating the game at the same speed that the server is running it, so you can’t ever catch up those 90 ticks. An analogy for this would be if you would be running on a track at the exact speed of 6km/h, but the track would be extended by 6km every hour, so you would never manage to finish running the track or reach the end of it, because you would be running at the same speed at which the track would be built. An important thing to say is that catching up is not related to the network, but rather to potential hardware bottlenecks, which I will explain later.
Another thing from Ookla’s speedtest is your ping (latency) to their datacenter. That is not always accurate in respect towards a Factorio server, as Factorio servers can be anywhere in the world, whilst Ookla connects you to your closest datacenter.
Reliability
Your connection speed and latency is one thing. However, network reliability can be far more important. An analogy for poor network reliability is if someone sends you 100 messages but only 10 of them actually arrive to you (that would be considered 90% packet loss in networking terms). Network reliability can vary, such as one day, your packets may be travelling properly, whilst another day, a shark may be eating up your data. Network reliability can be measured on the commandline with the ping command or using software such as StarTrinity’s Constant Speed Test, which is much more user-friendly and can display some extra data. For example in the image below, you can see that downloading has failed multiple times and this was due to an unreliable internet connection (turned off internet can simulate that). In reality, your result could be different. In any way, StarTrinity’s speed test is also a good way to find the best spot for WiFi speeds and reliability.
A way to fix an unreliable internet connection is using wired ethernet rather than a wireless connection. Wired is much less prone to being unreliable, as the data is in the form of electricity and travels across a (usually) copper cable, but wireless can be blocked by concrete walls or metal reinforcements which are often in walls. If you really cannot stop using wireless, you should at least consider switching from 5G to 2.4G, as 5G has a much shorter range and is much more prone to being blocked by objects in the path of the airwaves than 2.4G is.
Hardware Bottlenecks
So far, I have focused on the “your network” part of the error message. Another part of said message however is the “or computer” part. This means that it may also be your hardware that is blocking the game from running smoothly. There are various debug settings that you can enable to check for this, such as the show-fps debug option showing your FPS and UPS (FPS are frames per second (graphics), UPS are updates per second - how fast the game is updating, which is dictated by your CPU). To enable a debug setting, you’d press F4 -> select what you wish to -> press F4 again to hide the menu. The setting will be kept on from then on until you disable it.
A very important setting to see what is taking up CPU (UPS) time on your save is show-time-usage. This will show you a list of things that are being calculated in the game, from entity paths, fluid calculations, electric networks and robot paths. If the game is to run smoothly, you want everything to be below 16.66ms (1/60th of a second). If something is taking up a lot of time, you need to think a bit: if it’s robots and you have 50 000 robots flying around somewhere, try disabling that part of your factory and see how the time usage will be after that.
You may think “Why doesn’t the server calculate all of this for me”? Well, that is mostly because it is much easier to send only new parts in the data (as I described in my multiplayer article) rather than sending all of the data every tick. In Factorio, the client runs the same simulation as the server, so your CPU needs to be able to run the same calculations as the CPU of everyone else.
What to do about all of that?
There are a few things you can do to improve the performance of your factory, like switching all power production to solar. But there is only so much you can do without changing the factory itself. So, here is a selection of possible remedies that I thought were the most useful for uncontrollable things, such as pollution.
The easiest solution to a friend not being able to catch up is simply pausing the game. That can be done by pressing shift+space. This is however often only temporary, as the game will still simulate after you unpause the game, so your friend’s computer may not be able to keep up.
If you have a lot of pollution in places and you are further into the game where biters are just an expansion nuisance, you can run this command:
/c for _, surface in pairs(game.surfaces) do
surface.clear_pollution()
end
game.map_settings.pollution.enabled = false
For every surface in the game, it will remove all pollution and after that, it will disable pollution creation, which can improve performance quite a lot in some cases as it won’t have to simulate the pollution anymore.
If you however believe that pollution is not the right thing for you and instead you have large amounts of biters that are just idle, you can run this command instead:
/c game.forces["enemy"].kill_all_units()
It will kill all biters on the "enemy" force (which without any modifications will be biters). It will however only remove mobile units, such as biters and spitters, not spawners or worms. If you would like to remove those too, you can use this command instead:
/c local surface=game.player.surface
for key, entity in pairs(surface.find_entities_filtered({force="enemy"})) do
entity.destroy()
end
If you still feel that biters are just annoying and your base always just deals with them without any input, you can use this command instead:
/c local surface = game.player.surface
local mgs = surface.map_gen_settings
mgs.autoplace_controls["enemy-base"].size = "none"
surface.map_gen_settings = mgs
It will disable biters from generating only on your surface, which by default is Nauvis, however it won’t affect current biters (you will need to use previous commands for that).
If none of these work, the easiest resort to go to is reducing the game speed. Everything will run slower, but you will be able to play with your friends without them getting kicked off:
/c game.speed = 0.8
This command slows down the game by changing its speed (and thereby decreasing UPS). The default is 1, but you can go as low as 0.01, where pretty much nothing is happening. The best idea is to start going down from 1 in increments of 0.1 or 0.05, such as 0.9, 0.8, 0.7 etc. This command will globally reduce the game’s speed, so there will be less updates per second to calculate. The UPS number in the corner of your screen will be multiplied from the base by this constant, in the way $ 1 \cdot 60 = 60 $, $ 0.8 \cdot 60 = 48 $, $ 0.5 \cdot 60 = 30 $.
All of these have been sourced from the Commands wiki page if you want to see more Lua magic. To run any command, simply copy it, open your chatbar and paste it in.
Optimization by Design
You may have also noticed that I have not mentioned a way of how your GPU may be dropping you, even though the message at the top said “your computer”. The reason is that Factorio doesn’t care if your GPU is a 3090 Ti or just integrated graphics, it will just tell it to “render stuff”. If it won’t render it, that’s fine. You can always reduce your resolution or play around with the Factorio graphics settings which can give you an improvement.
Factorio is optimized by design and has always been. Therenas has dissected 1.1 performance upgrades in his post and the Factorio devs have many more posts related to optimizing the game (all of the FFFs have been neatly categorized here by spielsw, so you can easily find optimizations).
Contributing
As always, we’re looking for people that want to contribute to Alt-F4, be it by submitting an article or by helping with translation. If you have something interesting in mind that you want to share with the community in a polished way, this is the place to do it. If you’re not too sure about it we’ll gladly help by discussing content ideas and structure questions. If that sounds like something that’s up your alley, join the Discord to get started!