Alt-F4 #39 - JOSEF 11-06-2021
In this week’s issue of Alt-F4 #39, which is divisible by 13 (quick maffs), Drogiwan Cannobi makes their first contribution to the project by elaborating on the background and on the inner workings of his Rube-Goldberg-esque self-expanding factory that’s lovingly titled JOSEF. It’s a really cool concept that is not even that complicated once you get down to the details.
JOSEF’s Organically Self-Expanding Factory (JOSEF) Drogiwan Cannobi
This is JOSEF, a self-expanding factory. It starts off with a small amount of materials and slowly starts to grow, harvest more resources, make more building materials and grow some more. Assuming there are no mistakes left on my part (and you have infinite computing power), it would slowly grow forever. JOSEF uses the recursive blueprints mod which allows you to automatically deploy blueprints and deconstruction planners using circuit signals. There are already a few examples of self-expanding factories, most notably GreyGoo which was built by NiftyManiac in 2017. Seeing this way back then finally motivated me to give this a shot myself!
This post tries to explain the inner workings of JOSEF. I‘ll try to give a more or less complete explanation of how it all works, although I won‘t be able to go too much into ‘combinator by combinator’ detail.
I will try to convince you that you don’t need to be a programmer or a magician to try this yourself! I didn’t have a terrible amount of experience with advanced circuit stuff before I started this and I only know very basic magic. I promise, if you know how to use a combinator and have the patience to fool around a little bit, you can figure this out, too!
I‘ll first roughly outline how JOSEF works, then dig a little bit into the different parts of it. The general principle goes as follows:
- The base is made of rings of a very simple 2x2 chunk train grid around a central „brain“ cell.
- A new ring gets built once the existing ones have been filled up.
- A train goes through all the cells in a new ring and checks for ore patches.
- Other trains come in to either dedicate the cell to mining, solar or production of building materials.
- Once the existing ring is full, a new ring is started.
- All the cells are connected via global roboport and circuit networks.
Ring mechanism
Once all the existing cells are full, a new ring is built all at once. This is very different from GreyGoo, which built and analyzed cell after cell in a slow-and-steady spiral fashion. I went for complete rings because, honestly, it‘s a lot simpler, it works and I honestly didn‘t see a reason for anything more complex than that. It only needs 10-15 combinators and doesn‘t have the need for complicated memory (aside from storing the ring number you‘re currently at) or anything like that.
The basic ring building mechanism is as follows:
- Every cell is described by its position on an (X,Y) grid. The innermost cell has the coordinates (0,0). The one to the left has (-1,0), the one on top has (0,-1) and so on.
- Variable N defines the ring it’s currently building (the central cell being counted as the N=1 ring). Let‘s take N=3 (i.e. going from a 3x3 to a 5x5 grid) as an example here.
- The ring is split into the horizontal (top/bottom) and vertical rows (left/right). To build the horizontal (blue) rows, you want X to run continuously from the leftmost (-2) to the rightmost (2) coordinate and Y to flip between -2 and 2. Interchange X and Y and you get the vertical (orange) rows.
The last thing required to make this work is some way of increasing the ring number N. I included a combinator in every cell, sending a global Q=1 signal. This way JOSEF will roughly know when the ring is done building - when $ Q=(2N-1)^2 $, to be precise. The “decider” train (that comes to new cells and determines what to do with them) will only go to new cells once this condition is fullfilled. This prevents the train from going to half-built cells and messing them up.
Since I’m not only building an ever-expanding rail grid, there is a second condition for starting the next ring: Every existing cell must be filled with either mining (M), power (E) or one of two production cells (I,J). In other terms, a new ring will be started when $ M+I+J+E=(2N-1)^2 $.
This splits the building process into two distinct phases: Ring building (until enough ‘Q’ signals are received) and cell analysis (until enough M,E,I,J signals are received).
Managing individual cells
Okay, so now there is a bunch of new, empty cells. How to handle those individually?
New cells get built with a station called ‘New Cell’ (creative, isn’t it?) and a block of combinators in them. Those combinators are responsible for outputting construction bot = N in order to deploy the N-th blueprint from a book that is brought to the cell by train. Having the combinators in there from the start proved a lot simpler than constructing them during the analysis part, since half-constructed combinator setups (and also half-deconstructed ones) will inevitably end badly.
When all the cells in a new ring are built and enough materials are available in the logistic network, the builder train starts going through them one by one. It only contains a few materials and a BP book that is unloaded into a deployer chest. There are a few tricks involved for cycling through the blueprint book. The full process is too long to describe here, but let’s look at the start as an example. All the cells start with globally connected roboports but in order to check for ores (as described in the next paragraph) I will need uncoupled roboports, so that’s the first thing to do:
- BP 1 is a deconstruction planner for roboports. The initial roboports (globally connected) in a cell output
U-235=1(because there’s a chest with exactly 1 U-235 in the brain). So initially there is aU-235=4signal from the four roboports in this local circuit network. Once all four roboports are deconstructed (and therebyU-235=0), a combinator set toU-235=0→construction bot = 1triggers, leading to the next blueprint being deployed. - BP 2 builds a roboport in the center of the cell which is uncoupled from the network. The pre-built belt starts feeding 50 construction bots from the train into it. This new roboport is set to ‘read robot statistics’ and a
T=50→construction bot = 1combinator triggers BP 3 once 50 bots were fed into the now uncoupled network. - BP 3 deconstructs the bot belt and immediately triggers BP 4, which will start the “resource finding” process described in the next paragraph.
Generally, for cycling through blueprints you‘ll want to find some entity that can output circuit signals and use that entity being built/deconstructed as an indicator for the next step. In addition, you can use ‘idle construction bots (T=Z)’ or ‘item X available in logistic storage’ to make sure that the building process is actually complete.
As an example for what kinds of problems you can run into if you’re not careful or use time-dependent conditions: As JOSEF started growing larger, it sometimes took a bit of time for bots to come and build the uncoupled roboport. If (for reasons that will appear later) 200 belts were unloaded before that roboport was built, the order of blueprints got messed up, the thing started building belts for mining without deconstructing the bot belt, then dedicated the cell as ‘no mining’. It was an absolute mess and led to the factory getting stuck.
Resource Finder
Now for one of the crucial bits: How to find resources? There is no way of reading ‘ore on ground’ in vanilla, but it’s actually really simple once you have uncoupled roboport networks:
- The builder train unloads a fixed amount of mining drills. If you place a blueprint that includes miners, they will be taken from this chest, so you can precisely say how many miners were placed in that cell. To be on the safe side, the train carries 150 miners and 200 belts.
- JOSEF waits for the miners to be completely unloaded (by waiting for
belts=200) and for the 50 construction bots to be idle (by waiting forT=Zfrom the roboport) after placing the miner blueprint. It checks the number of miners left in the chest and increases the ‘construction bot’ signal by either 1 or 2, depending on whether a certain threshold number of miners has been placed. I picked 10 miners as a minimum to dedicate the whole cell to mining. - This means two different blueprints can be deployed: The “Mining” or the “No Mining” blueprint. There are a few more steps involved, but it then mainly renames the station, deconstructs the chests and rebuilds the roboports to reconnect the cell. Also, this will lead to the blueprint book being put back into the train after everything is done.
To avoid problems with mixed mining, every mining cell contains one station each for iron ore, copper ore, stone and coal. Indidiviual stations will only activate when there is enough ore to fill an entire train. This means most stations will never activate.
Every new cell will be checked in this way. If more than ten miners are placed, it will be dedicated to mining independent of requirements. This is definitely a quick & dirty approach and could be greatly improved.
The initial builder train’s job is done once the cell is designated as “Mining” or “No Mining” and the roboport network is reconnected. Another train will come into the “Mining” cells, properly label the stations, deconstruct the combinators and add an “M” signal to the constant combinator characterizing the type of cell.
My way of finding ores has one big problem: You can never have miners stored in the global logistic network or they will (in part) be placed instead of the ones in the uncoupled network. With my small cells, construction ranges overlap so much that the entire cell can be accessed from the outside. There are a few safeguards in place to prevent miners from becoming available in the global network (while still having them transported to the builder train by logistic bots), but there can be situations where the ore finding gets messed up once the base gets really big. But it’s a rare thing and it doesn’t break anything, so that’s fine with me. In the next version though, I‘ll try to keep miners away from logistic chests completely, except for the ones in cells that are currently being analyzed.
The “No Mining” cells then are handled by two different trains, depending on…
… Power
To make the start a little easier, JOSEF was granted 30 MW of continuous power by an EEI (electric energy interface). Once the power requirements exceed that, JOSEF will build “Power” cells including solar panels and accumulators. One cell produces 7 MW continuously (10 MW peak), so you’ll need a few of those.
JOSEF will determine a need for more power once the charge of an accumulator drops below 70%. This will allow the “Power” train to start. Similarly to the “Mining” train, this guy just goes around (this time to “No Mining” stations), deploys the solar blueprint, deconstructs combinators, adds an “E” signal to the global combinator and gets out of there. There are no further limitations on it because I wanted to make sure that there would always be a way to get more power if JOSEF played itself into a corner.
Due to my way of building complete, ever-increasing rings every now and then, there are huge power spikes. This leads to JOSEF overbuilding power by quite a bit as the first few new cells in every ring usually get dedicated to power. In my larger runs, average draw towards the end was around 500 MW while potential power generation was about 1 GW. This is not the worst in the world (actually, UPS-wise I’m happy with every cell that gets filled with power instead of production) but it would definitely be more efficient if I used a slower, GreyGoo-style expansion mechanism.
Now there‘s only one little detail missing: We want JOSEF to actually make the materials that it‘s using! So all the remaining cells are dedicated to…
… Production
The production of building materials takes place in two different spaghetti cells. The story of how those get built is similar to the ones before: When a station called “No Mining” is available and enough building materials are in the logistic network (and power isn’t running low), a train comes in, deploys a blueprint and adds either “I” or “J” to the global constant combinator, depending on which production cell is built. This guy does one additional thing: It unloads a bit of coal into the logistic network to make sure trains have something to run on, and, for production cell 2, some heavy oil barrels to start coal liquefaction.
The builder train that’s in charge simply alternates between the two different cells. There’s a simple “inserter cycle” that takes the blueprint book out of the train when it gets home, puts in the other one, then waits for the train to leave before activating again.
The first cell produces all the different basic building materials: Belts, assemblers, miners, inserters, rails, trains, stations, signals, power poles – more or less everything that doesn‘t require oil. The basic production cell needs to be built before JOSEF runs out of stuff like belts or rails. This may be a problem if you have too many resource patches in the center or if you don’t give it a big enough stock to start with and it decides to just solar panel the world. But you can always fix this by giving it a few more starting materials.
The second cell produces the more advanced things: Roboports, robots, logistic chests, blueprint deployers, solar panels & accumulators. This one needs to be built before JOSEF runs out of roboports. The builder train brings in ten barrels of heavy oil which get emptied, run through the refinery and get refilled as soon as possible. JOSEF is not producing barrels but instead relying on the initial ones getting recycled infinitely.
The two production cells are far from optimized to proper ratios or anything like that. They slowly and steadily produce all the things necessary but it may take an hour until a certain cell produces, say, their first train station, because it needs to fill up with belts first. However, it turns out that this becomes a non-issue after a bit, but more on that later.
After a while, there will be lots of production cells scattered across the map. This means global construction robots will slowly stop picking up materials from the center (where the initial building materials are located) and start picking up from production cells nearby, which reduces travel time. However, due to lazyness on my part, they still have to bring quite a few materials into the center to load up the builder train, but that’s a different story (and will be handled much more elegantly in JOSEF Mark 2).
The production cells get fed by trains. A special “train deployer” cell in the center generates four new trains (one for every type of ore) once a new “I” or “J” signal is received. Having a dedicated cell for deploying trains (instead of including them in the production cell blueprints) makes sure that blueprinted trains and existing trains never interfere. Without that precaution, there can be instances of blueprint wagons attaching to existing trains and the like, so that has to be avoided.
All trains have a simple ‘ore in until full -> ore out until empty’ schedule and will just pick the nearest station. This means that production cells close to ore patches will be supplied with more resources than those further away which may remain empty for a while. However, this evens out after a while since JOSEF slowly fills up with lots and lots of production materials and many cells just become idle.
The fact that there are four trains for every production cell means that there will be a ridiculous number of trains, most of them inactive. In my >500 cell run JOSEF had 750 trains towards the end. Looking at the tiny, roundabout-heavy train grid it’s a good thing that most of them were idle.
Review & Future Plans
- A few numbers on JOSEF:
- The largest it grew on my (not-so-great) computer was to 25x25 = 625 cells. This took about 34 hours of ingame time. It was still going, but UPS were down to about 40 and things just took an awful lot of time, so I decided to stop it there.
- It had built about 170 power cells, 150 mining cells and about 100 of each production cell.
- A friendly redditor with a more powerful computer ran it up to 29x29 = 841 cells, running at 45 UPS.
- It’s quite interesting to see how JOSEF behaves when it gets a bit larger. Initially, expansion is fairly slow because building materials have to be produced. Looking at the power graph you can see how it initially speeds up from ring to ring, even though the number of cells it has to go through increases with every ring. This is due to the fact that production increases quadratically, while demand only increases linearly (or even stagnates at later stages). At some point around ring 9 or 10, production stops being the bottleneck and the time needed to fill up a ring increases, simply because more cells have to be filled.
- In this version, I relied quite heavily on bots transporting things across the whole base (mainly to stock up the builder train and return unused materials from newly analyzed cells). This was more due to my lazyness than anything else and I‘m definitely planning on improving that for future versions. In general though, my system of spamming production cells everywhere, using an excessive amount of construction bots and giving them access to building materials scattered across the whole base worked a lot better than I had expected – building jobs are given to nearby bots which always take things from the closest location available, as long as you only offer them one type of chest.
- I loved coming up with the system, designing the cells and all that, but I have to be honest: Testing something like this is a bit annoying, especially if you‘re impatient and want to see it running. It‘s like testing a Rube Goldberg machine that runs for many hours – For every tiny change that you make, you have to start over, implement the change and let it run for a couple of hours. Many small fixes you quickly come up with will lead to more unforeseen problems that might only occur in ring 6 or so. You can of course make things easier by e.g. giving it infinite construction materials when you‘re testing the ore finding mechanism, but actual realistic tests are pretty tedious.
This is certainly not the last JOSEF I‘m ever going to build. In fact, I started a little, unprofessional tutorial series in which I‘m building a second version and explaining how and why I do stuff along the way. This is what I’ve got so far:
- Larger cells. I’m currently planning on doing 4x4 chunks instead of 2x2. I’ll have uncoupled bot networks to begin with which will greatly simplify the whole ore finding process.
- I greatly simplified and stabilized the decision tree for going through a blueprint by having a fixed amount of combinators and pasting new settings onto them. This brings along new challenges but it allows for a lot more different cells from a single blueprint book without the need for more combinators.
- Dealing with water, both as a resource and as an obstacle: Done!
- Dealing with biters: Will be difficult to get right. I won‘t tackle deathworlds but small amounts of biters should be okay to deal with.
- Dealing with cliffs: Will only be difficult psychologically. I‘ve always hated them and turned them off right away. Should be very easy to handle.
- Making science: Theoretically done (I just included tiny, 3SPM science builds in the production cells), but it will be a little tricky once running, because it will hugely increase the draw on resources and the number of active trains. Also, I may have to introduce some additional safety measures to make sure that science doesn’t take away from production materials. After all, the factory must grow.
Final Thoughts
A few years ago when I first saw the video of GreyGoo (probably the most popular self-expanding factory to date) I felt really intimidated and I was sure that an absolute genius from out of this world had come up with this. I thought it would simply be impossible for a normal, semi-experienced factorio player to ever achieve something like this.
Now I don‘t want to compare JOSEF to GreyGoo. JOSEF is a really dumbed-down SEF and there are countless things that could (and will!) be improved. But I felt that the many people congratulating me on making this, felt the same way towards JOSEF that I did towards GreyGoo back then. So while I am definitely proud of this and thankful for all the praise, please believe me when I say: This was not that hard! Anybody who has built a clever, circuit-based contraption for sushi belts, balanced train stations, clever Kovarex build, or whatever, can make this as well! Just play around with it a little and you‘ll see that this is actually more than doable. I hope this little glimpse into JOSEF‘s simplistic inner workings has proven that. I‘d really love to see more people taking a stab at this.
Feel free to use my ideas as a starting point, improve on them, toss them out and replace them if you don‘t like them! And of course, I‘d always be happy to discuss, help, debug and brainstorm. A few of JOSEF‘s features only came to life due to discussions with awesome people from this great community.
Contributing
As always, we’re looking for people that want to contribute to Alt-F4, be it by submitting an article or by helping with translation. If you have something interesting in mind that you want to share with the community in a polished way, this is the place to do it. If you’re not too sure about it we’ll gladly help by discussing content ideas and structure questions. If that sounds like something that’s up your alley, join the Discord to get started!