Looking at the htop screenshot, I notice the lack of swap. You may want to enable earlyoom, so your whole server doesn't go down when a service goes bananas. The Linux Kernel OOM killer is often a bit too late to trigger.
You can also enable zram to compress ram, so you can over-provision like the pros'. A lot of long-running software leaks memory that compresses pretty well.
Even better than earlyoom is systemd-oomd[0] or oomd[1].
systemd-oomd and oomd use the kernel's PSI[2] information which makes them more efficient and responsive, while earlyoom is just polling.
earlyoom keeps getting suggested, even though we have PSI now, just because people are used to using it and recommending it from back before the kernel had cgroups v2.
Do you have any insight in to why this isn't included by default in distros like Ubuntu. It's kind of bewildering that the default behavior on Ubuntu is to just lock up the whole system on OOM
Is there any way to get something like the oomd or zram that works on gpu memory? I run into gpu memory leaks more often. Itbseems to be electron usually.
> systemd-oomd periodically polls PSI statistics for the system and those cgroups to decide when to take action.
It's unclear if the docs for systemd-oomd are incorrect or misleading; I do see from the kernel.org link that the recommended usage pattern is to use the `poll` system call, which in this context would mean "not polling", if I understand correctly.
systemd-oomd, oomd, and earlyoom all do poll for when to actually take action on OOM conditions.
What I was trying to say is that the actual information on when there's memory pressure is more accurate for systemd-oomd / oomd because they use PSI, which the kernel itself is updating over time, and they just poll that, while earlyoom is also internally making its own estimates at a lower granularity than the kernel does.
Unrelated to the topic, it seems awfully unintuitive to name a function ‘poll’ if the result is ‘not polling.’ I’m guessing there’s some history and maybe backwards-compatible rewrites?
Specifically, earlyoom’s README says it repeatedly checks (“periodically polls”) the memory pressure, using CPU each time even when there is no change. The “poll” system call waits for the kernel to notify the process that the file has changed, using no CPU until the call resolves. It’s unclear what systemd-oomd does, because it uses the phrase “periodically polls”,
Another option would be to have more memory that required over-engineer and to adjust the oom score per app, adding early kill weight to non critical apps and negative weight to important apps. oom_score_adj is already set to -1000 by OpenSSH for example.
Another useful thing to do is effecively disable over-commit on all staging and production servers (0 ratio instead of 2 memory to fully disable as these do different things, memory 0 still uses formula)
vm.overcommit_memory = 0
vm.overcommit_ratio = 0
Also use a formula to set min_free and reserved memory using a formula from Redhat that I do not have handy based on installed memory. min_free can vary from 512KB to 16GB depending on installed memory.
At least that worked for me in about 50,000 physical servers for over a decade that were not permitted to have swap and installed memory varied from 144GB to 4TB of RAM. OOM would only occur when the people configuring and pushing code would massively over-commit and not account for memory required by the kernel. Not following best practices defined by Java and thats a much longer story.
Another option is to limit memory per application in cgroups but that requires more explaining than I am putting in an HN comment.
Another useful thing is to never OOM kill in the first place on servers that are only doing things in memory and need not commit anything to disk. So don't do this on a disked database. This is for ephemeral nodes that should self heal. Wait 60 seconds so drac/ilo can capture crash message and then earth shattering kaboom...
For a funny side note, those options can also be used as a holy hand grenade to intentionally unsafely reboot NFS diskless farms when failing over to entirely different NFS server clusters. setting panic to 15 mins, triggering OOM panic by setting min_free to 16TB at the command line via Ansible not in sysctl.conf, swapping clusters, arp storm and reconverge.
That's a more complex path I avoided discussing when I referenced CGroups. When I started doing these things kube clusters did not exist. These tips were for people using bare metal that have not decided as a company to go the k3/k8 route. Some of these settings will still apply to k8 physical nodes. The good people of Hetzner would be managing these settings on their bare metal that Kubernetes is running on and would not likely want their k8 nodes getting all broken, sticky and confused after a K8 daemon update results in memory leakage, billions of orphaned processes, etc...
Companies that use k3/k8's they may still have bare metal nodes that are dedicated to a role such as databases, ceph storage nodes, DMZ SFTP servers, PCI hosts that were deemed out of scope for kube clusters and of course any "kittens" such as Linux nodes turned into proprietary appliances after installing some proprietary application that will blow chunks if shimmed into k8's or any other type of abstraction layer.
I mentioned Hetnzer only because the original article mentions it. To be fair, currently it is harder to use than any managed k8s offering because you need to deploy your control plane yourself (but fortunately there are several project that make it as easy as it can be, and this is what I was referring to).
Yeah, no way. As soon as you hit swap, _most_ apps are going to have a bad, bad time. This is well known, so much so that all EC2 instances in AWS disable it by default. Sure, they want to sell you more RAM, but it's also just true that swap doesn't work for today's expectations.
Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot.
This is a wrong belief because a) SSDs make swap almost invisible, so you can have that escape ramp if something goes wrong b) SWAP space is not solely an escape ramp which RAM overflows into anymore.
In the age of microservices and cattle servers, reboot/reinstall might be cheap, but in the long run it is not. A long running server, albeit being cattle, is always a better solution because esp. with some excess RAM, the server "warms up" with all hot data cached and will be a low latency unit in your fleet, given you pay the required attention to your software development and service configuration.
Secondly, Kernel swaps out unused pages to SWAP, relieving pressure from RAM. So, SWAP is often used even if you fill 1% of your RAM. This allows for more hot data to be cached, allowing better resource utilization and performance in the long run.
So, eff it, we ball is never a good system administration strategy. Even if everything is ephemeral and can be rebooted in three seconds.
Sure, some things like Kubernetes forces "no SWAP, period" policies because it kills pods when pressure exceeds some value, but for more traditional setups, it's still valuable.
My work Ubuntu laptop has 40GB of RAM and and a very fast Nvme SSD, if it gets under memory pressure it slows to a crawl and is for all practical purposes frozen while swapping wildly for 15-20 minutes.
So no, my experience with swap isn't that it's invisible with SSD.
I don't know your exact situation, but be sure you're not mixing up "thrashing" with "using swap". Obviously, thrashing implies swap usage, but not the other way around.
I get it that the distinction is real but nobody using the machine cares at this point. It must not happen and if disabling swap removes it then people will disable swap.
I've experimented with no-swap and find the same thing happens. I think the issue is that linux can also evict executable pages (since it can just reload them from disk).
I've had good experience with linux's multi-generation LRU feature, specifically the /sys/kernel/mm/lru_gen/min_ttl_ms feature that triggers OOM-killer when the "working set of the last N ms doesn't fit in memory".
It's seldom invisible, but in my experience how visible it is depends on the size/modularity/performance/etc of what's being swapped and the underlying hardware.
On my 8gb M1 Mac, I can have a ton of tabs open and it'll swap with minimal slowdown. On the other hand, running a 4k external display and a small (4gb) llm is at best horrible and will sometimes require a hard reset.
I've seen similar with different combinations of software/hardware.
Linux being absolute dogshit if it’s under any sort of memory pressure is the reason, not swap or no swap. Modern systems would be much better off tweaking dirty bytes/ratios, but fundamentally the kernel needs to be dragged into the XXI century sometime.
It's kind of solved since kernel 6.1 with MGLRU, see above.
Dirty buffer should also be tuned (limited), absolutely. Default is 20% of RAM, (with 5 second writeback and 30 second expire intervals), which is COMPLETELY insane. I limit it to 64 MB max usually, with 1 second writeback and 3 second expire intervals.
This is not really true of most SSDs. When Linux is really thrashing the swap it’ll be essentially unusable unless the disk is _really_ fast. Fast enough SSDs are available though. Note that when it’s really thrashing the swap the workload is 100% random 4KB reads and writes in equal quantities. Many SSDs have high read speeds and high write speeds but have much worse performance under mixed workloads.
I once used an Intel Optane drive as swap for a job that needed hundreds of gigabytes of ram (in a computer that maxed out at 64 gigs). The latency was so low that even while the task was running the machine was almost perfectly usable; in fact I could almost watch videos without dropping frames at the same time.
Do you know how the le9 patch compares to mg_lru? The latter applies to all memory, not just files as far as I can tell. The former might still be useful in preventing eager OOM while still keeping executable file-backed pages in memory?
That may be the intention, but you shouldn’t rely on it. In practice the average IO size is, or at least was, almost always 4KB.
Here’s a screenshot from atop while the task was running: <https://db48x.net/temp/Screenshot%20from%202019-11-19%2023-4...>. Note the number of page faults, the swin and swout (swap in and swap out) numbers, and the disk activity on nvme0n1. Swap in is 150k, and the number of disk reads was 116k with an average size of 6KB. Swap out was 150k with 150k disk writes of 4KB. It’s also reading from sdh at a fair clip (though not as fast as I wanted!)
<https://db48x.net/temp/Screenshot%20from%202019-12-09%2011-4...> is interesting because it actually shows 24KB average write size. But notice that swout is 47k but there were actually 57k writes. That’s because the program I was testing had to write data out to disk to be useful, and I had it going to a different partition on the same nvme disk. Notice the high queue depth; this was a very large serial write. The swap activity was still all 4KB random IO.
How long is long running? You should be getting the warm caches after at most a few hours.
> Secondly, Kernel swaps out unused pages to SWAP, relieving pressure from RAM. So, SWAP is often used even if you fill 1% of your RAM. This allows for more hot data to be cached, allowing better resource utilization and performance in the long run.
Yes, and you can observe that even in your desktop at home (if you are running something like Linux).
> So, eff it, we ball is never a good system administration strategy. Even if everything is ephemeral and can be rebooted in three seconds.
I wouldn't be so quick. Google ran their servers without swap for ages. (I don't know if they still do it.) They decided that taking the slight inefficiency in memory usage, because they have to keep the 'leaked' pages around in actual RAM, is worth it to get predictability in performance.
For what it's worth, I add generous swap to all my personal machines, mostly so that the kernel can offload cold / leaked pages and keep more disk content cached in RAM. (As a secondary reason: I also like to have a generous amount of /tmp space that's backed by swap, if necessary.)
With swap files, instead of swap partitions, it's fairly easy to shrink and grow your swap space, depending on what your needs for free space on your disk are.
> Yes, and you can observe that even in your desktop...
Yup, that part of my comment was culmination of using Linux desktops for the last two decades. :)
> I wouldn't be so quick. Google ran their servers without swap for ages.
If you're designing this from get go and planning accordingly, it doesn't fit into my definition of eff it, we ball, but let's try this and see whether we can make it work.
> With swap files, instead of swap partitions,...
I'm a graybeard. I eyeball a swap partition size while installing the OS, and just let it be. Being mindful and having good amount of RAM means that SWAP acts as a eviction area for OS first, and as an escape ramp second, in very rare cases.
It doesn't. SSDs came a long way but so did memory dies and buses, and with that the way programs work also changed as more and more they are able to fit their stacks and heaps on memory more often than not.
I have had a problem with shellcheck that for some reason eats up all my ram when I open I believe .zshrc and trust me, it's not invisible. The system crawls to a halt.
If we're talking about SATA SSDs which top at 600MBps, then yes, an aggressive application can make itself known. However, if you have a modern NVMe, esp. a 4x4 one like Samsung 9x0 series or if you're using a Mac, I bet you'll notice the problem much later, if ever. Remember the SSD trashing problem on M1 Macs? People never noticed that system used SWAP that heavily and trashed the SSD on board.
Then, if you're using a server with a couple of SAS or NVMe SSDs, you'll not notice the problem again, esp. if these are backed by RAID (even md counts).
Now that you say, I have a new Lenovo yoga with those SoC ram with crazy parallel channel config (16gb spread across 8 dies of 2gb). It's noticeably faster than my Acer nitro with dual channel 16gb ddr5. I'll check that, but I'd say it's not what the average home user (and even server I'd risk saying) would have.
> it's not invisible. The system crawls to a halt.
I’m gonna guess you’re not old enough to remember computers with memory measured in MB and IDE hard disks? Swapping was absolutely brutal back then. I agree with the other poster, swap hitting an SSD is a barely noticeable in comparison.
I think I've not made myself as clear as I could. Swap is important for efficient system performance way before you hit OOM on main memory. It's not, however, going to save system responsiveness in case of OOM. This is what I mean.
The trade-off depends on how your system is set up.
Eg Google used to (and perhaps still does?) run their servers without swap, because they had built fault tolerance in their fleet anyway, so were happier to deal with the occasional crash than with the occasional slowdown.
For your desktop at home, you'd probably rather deal with a slowdown that gives you a chance to close a few programs, then just crashing your system. After all, if you are standing physically in front of your computer, you can always just manually hit the reset button, if the slowdown is too agonising.
I don't count crawling to a halt as a working machine. Plus it depends. Back in the day I had computers that got blocked for 30-ish seconds which was annoying but gave you the window of opportunity to go kill the offending program. But then you had some that we left, out of curiosity, to work throughout the entire workday and they never recovered.
So most of the time I'd prefer option 3: the OOM killer to reap a few offending programs and let me handle restarting them.
Swap delays the 'fundamental issue', if you have a leak that keeps growing.
If your problem doesn't keep growing, and you just have more data that programs want to keep in memory than you have RAM, but the actual working set of what's accessed frequently still fits in RAM, then swap perfectly solves this.
Think lots of programs open in the background, or lots of open tabs in your browser, but you only ever rapidly switch between at most a handful at a time. Or you are starting a memory hungry game and you don't want to be bothered with closing all the existing memory hungry programs that idle in the background while you play.
I run a chat server on a small instance; when someone uploads a large image to the chat, the 'thumbnail the image' process would cause the OOM-killer to take out random other processes.
Adding a couple of gb of swap means the image resizing is _slow_, but completes without causing issues.
The problem is freezing the system for hours or more to delay the issue is not worth it. I'd rather a program get killed immediately than having my system locked up for hours before a program gets killed.
The fundamental issue here is what the linux fanboys literally think what killing a working process and most of the time the process[0] is a good solution for not solving the fundamental problem of memory allocation in the Linux kernel.
Availability of swap allows you to avoid malloc failure in a rare case your processes request more memory than physically (or 'physically', heh) present in the system. But in the mind of so called linux administrators even if a one byte of the swap would be used then the system would immediately crawl to a stop and never would recover itself. Why it always should be the worst and the most idiotic scenario instead of a sane 'needed 100MB more, got it - while some shit in the memory which wasn't accessed since the boot was swapped out - did the things it needed to do and freed that 100MB' is never explained by them.
[0] imagine a dedicated machine for *SQL server - which process would have the most memory usage on that system?
Also: When those processes that haven't been active since boot (and which may never be active again) are swapped out, more system RAM can become available for disk caching to help performance of things that are actively being used.
And that's... that's actually putting RAM to good use, instead of letting it sit idle. That's good.
(As many are always quick to point out: Swap can't fix a perpetual memory leak. But I don't think I've ever seen anyone claim that it could.)
What if I care more about the performance of things that aren't being used right now than the things that are? I'm sick of switching to my DAW and having to listen to my drive thrash when I try to play a (say) sampler I had loaded.
Detecting things are down is far easier than detecting things are slow.
I'd rather that oom started killing things though than a kernel panic or a slow system. Ideally the thing that is leaking, but if not the process using the most memory (and yes I know that "using" is tricky)
From my understanding, the comment I'm replying to uses EC2 example to portray that swapping is wrong in any and all circumstances, and I just replied with my experience with my system administrator hat.
I'm not an AWS guy. I can see and touch the servers I manage, and in my experience, SWAP works, and works well.
Just for context EC2 typically uses network storage that, for obvious reasons, often has fairly rubbish latency and performance characteristics. Swap works fine if you have local storage, though obviously it burns through your SSD/NVME drive faster and can other side effects on it's performance (usually not particularly noticeable).
Thanks, I'll keep that in mind if I start to use EC2 for workloads.
However, from my experience, normal (eviction based) usage of SWAP doesn't impact the life of an SSD in a measurable manner. My 256GB system SSD (of my desktop system) shows 78% life remaining after 4 years of power on hours, which also served as /home for at least half of its life.
A long running Linux system uses 100% of its RAM. Every byte unused for applications will be used as a disk cache, given you read more data than your total RAM amount.
This cache is evictable, but it'll be there eventually.
Linux used to don't touch unused pages in the RAM in the older days if your RAM was not under pressure, but now it swaps out pages unused for a long time. This allows more cache space in RAM.
> how does caching to swap help?
I think I failed to convey what I tried to say. Let me retry:
Kernel doesn't cache to SSD. It swaps out unused (not accessed) but unevictable pages to SWAP, assuming that these pages will stay stale for a very long time, allowing more RAM to be used as cache.
When I look to my desktop system, in 12 days, Kernel moved 2592MB of my RAM to SWAP despite having ~20GB of free space. ~15GB of this free space is used as disk cache.
So, to have 2.5GB more disk cache, Kernel moved 2592 MB of non-accessed pages to SWAP.
Yes, and if I am writing an API service, for example, I don’t want to suddenly add latency because I hit pages that have been swapped out. I want guarantees about my API call latency variance, at least when the server isn’t overloaded.
I DON’T WANT THE KERNEL PRIORITIZING CACHE OVER NRU PAGES.
You better not write your API in Python, or any language/library that uses amortised algorithms in the standard (like Rust and C++ do). And let's not mention garbage collection.
If you're getting this far into the details of your memory usage, shouldn't you use mlock to actually lock in the parts of memory you need to stay there? Then you get to have three tiers of priority: pages you never want swapped, cache, then pages that haven't been used recently.
If you’re writing services in anything higher level than C you’re leaking something somewhere that you probably have no idea exists and the runtime won’t ever touch again.
That’s a fair question. A page is the smallest allocatable unit of RAM, from the OS/kernel perspective. The size is set by the CPU, traditionally 4kB, but these days 8kB-4MB are also common.
When you call malloc(), it requests a big chunk of memory from the OS, in units of pages. It then uses an allocator to divide it up into smaller, variable length chunks to form each malloc() request.
You may have heard of “heap” memory vs “stack” memory. The stack of course is the execution/call stack, and heap is called that because the “heap allocator” is the algorithm originally used for keeping track of unused chunks of these pages.
(This is beginner CS stuff so sorry if it came off as patronizing—I assume you’re either not a coder or self-taught, which is fine.)
You already maxed it from Kernel's PoV.
8GB of RAM, where 6.8GB is cache. ~700MB is resident and 459 is free because I assume Kernel wants to have some free space to allocate something quite fast.
25MB swap use seems normal for a server which doesn't juggle much tasks, but works on one.
If you are interested in human consumption, there's "free --human" which decided on useful units by itself. The "--human" switch is also available for "du --human" or "df --human" or "ls -l --human". It's often abbreviated as "-h", but not always, since that also often stands for "--help".
Thanks, I generally use free -m since my brain can unconsciously parse it after all these years. ls -lh is one of my learned commands though. I type it in automatically when analyzing things.
ls -lrt, ls -lSh and ls -lShr are also very common in my daily use, depending on what I'm doing.
So that 2M of used swap is completely irrelevant. Same on my laptop
total used free shared buff/cache available
Mem: 31989 11350 4474 2459 16164 19708
Swap: 6047 20 6027
My syslog server on the other hand (which does a ton of stuff on disk) does use swap
Mem: 1919 333 75 0 1511 1403
Swap: 2047 803 1244
With uptime of 235 days.
If I were to increase this to 8G of ram instead of 2G, but for arguments sake had to have no swap as the tradeoff, would that be better or worse. Swap fans say worse.
> So that 2M of used swap is completely irrelevant.
As I noted somewhere, my other system has 2,5GB of SWAP allocated over 13 days. That system is a desktop system and juggles tons of things everyday.
I have another server with tons of RAM, and the Kernel decided not to evict anything to SWAP (yet).
> If I were to increase this to 8G of ram instead of 2G, but for arguments sake had to have no swap as the tradeoff, would that be better or worse. Swap fans say worse.
I'm not a SWAP fan, but I support its use. On the other hand I won't say it'd be worse, but it'd be overkill for that server. Maybe I can try 4, but that doesn't seem to be necessary if these numbers are stable over time.
The OS uses almost all the ram in your system (it just doesn't tell you because then users complain that their OS is too ram heavy). The primary thing it uses it for is caching as much of your storage system as possible. (e.g. all of the filesystem metadata and most of the files anyone on the system has touched recently). As such, if you have RAM that hasn't been touched recently, the OS can page it out and make the rest of the system faster.
At the cost of tanking performance for the less frequently used code path. Sometimes it is more important to optimize in ways that minimize worst case performance rather than a marginal improvement to typical work loads. This is often the case for distributed systems, e.g. SaaS backends.
This is not about belief, but lived experience. Setting up swap to me is a choice between a unresponsive system (with swap) or a responsive system with a few oom kills or downed system.
Swap also works really well for desktop workloads. (I guess that's why Apple uses it so heavily on their Macbooks etc.)
With a good amount of swap, you don't have to worry about closing programs. As long as your 'working set' stays smaller than your RAM, your computer stays fast and responsive, regardless of what's open and idling in the background.
It doesn’t happen often, and I have a multi user system with unpredictable workloads. It’s also not about swap filling up, but giving the pretense the system is operable in a memory exhausted state which means oom killer doesn’t run, but the system is unresponsive and never recovers.
Without swap oom killer runs and things become responsive.
"as soon as you hit swap" is a bad way of looking at things. Looking around at some servers I run, most of them have .5-2GB of swap used despite a bunch of gigabytes of free memory. That data is never or almost never going to be touched, and keeping it in memory would be a waste. On a smaller server that can be a significant waste.
Swap is good to have. The value is limited but real.
Also not having swap doesn't prevent thrashing, it just means that as memory gets completely full you start dropping and re-reading executable code over and over. The solution is the same in both cases, kill programs before performance falls off a cliff. But swap gives you more room before you reach the cliff.
> Running out of memory kills performance. It is better to kill the VM and restart it so that any active VM remains low latency.
Right, you seem to be not understanding what I'm getting at.
Memory exhaustion is bad, regardless of swap or not.
Swap gets you a better performing machine because you can swap out shit to disk and use that ram for vfs cache.
the whole "low latency" and "I want my VM to die quicker" is tacitly saying that you haven't right sized your instances, your programme is shit, and you don't have decent monitoring.
Like if you're hovering on 90% ram used, then your machine is too small, unless you have decent bounds/cgroups to enforce memory limits.
>It's a bit wasteful to provision your computers so that all the cold data lives in expensive RAM.
But that's a job applications are already doing. They put data that's being actively worked on in RAM they leave all the rest in storage. Why would you need swap once you can already fit the entire working set in RAM?
Because then you have more active working memory as infrequently used pages are moved to compressed swap and can be used for more page cache or just normal resident memory.
Swap ram by itself would be stupid but no one doing this isn’t also turning on compression.
> Swap ram by itself would be stupid but no one doing this isn’t also turning on compression.
I'm not sure what you mean here? Swapping out infrequently accesses pages to disk to make space for more disk cache makes sense with our without compression.
Swapping out to RAM without compression is stupid - then you’re just shuffling pages around in memory. Compression is key so that you free up space. Swap to disk is separate.
You mean to tell me most applications you've ever used read the entire file system, loading every file into memory, and rely on the OS to move the unused stuff to swap?
A silly but realistic example: lots of applications leak a bit of memory here and there.
Almost by definition, that leaked memory is never accessed again, so it's very cold. But the applications don't put this on disk by themselves. (If the app's developers knew about which specific bit is leaking, they'd rather fix the leak then write it to disk.)
That's just recognizing that there's a spectrum of hotness to data. But the question remains: if all the data that the application wants to keep in memory does fit in memory, why do you need swap?
Many won't enable swap. For some swap wouldn't help anyways, but others it could help soak up spikes. The latter in some cases will upgrade to a larger instance without even evaluating if swap could help, generating AWS more money.
Either way it's far-fetched to derive intention from the fact.
How programs use ram also changed from the 90s. Back then they were written targeting machines that they knew would have a hard time fitting all their data in memory, so hitting swap wouldn't hurt perceived performance too drastically since many operations were already optimized to balance data load between memory and disk.
Nowadays when a program hits swap it's not going to fallback to a different memory usage profile that prioritises disk access. It's going to use swap as if it were actual ram, so you get to see the program choking the entire system.
If your GC is a moving collector, then absolutely this is something to watch out for.
There are, however, a number of runtimes that will leave memory in place. They are effectively just calling `malloc` for the objects and `free` when the GC algorithm detects an object is dead.
Go, the CLR, Ruby, Python, Swift, and I think node(?) all fit in this category. The JVM has a moving collector.
Every garbage collector has to constantly sift through the entire reference graph of the running program to figure out what objects have become garbage. Generational GC's can trace through the oldest generations less often, but that's about it.
Tracing garbage collectors solve a single problem really really well - managing a complex, possibly cyclical reference graph, which is in fact inherent to some problems where GC is thus irreplaceable - and are just about terrible wrt. any other system-level or performance-related factor of evaluation.
> Every garbage collector has to constantly sift through the entire reference graph of the running program to figure out what objects have become garbage.
There's a lot of "it depends" here.
For example, an RC garbage collector (Like swift and python?) doesn't ever trace through the graph.
The reason I brought up moving collectors is by their nature, they take up a lot more heap space, at least 2x what they need. The advantage of the non-moving collectors is they are much more prompt at returning memory to the OS. The JVM in particular has issues here because it has pretty chunky objects.
> The reason I brought up moving collectors is by their nature, they take up a lot more heap space, at least 2x what they need.
If the implementer cares about memory use it won't. There are ways to compact objects that are a lot less memory-intensive than copying the whole graph from A to B and then deleting A.
It doesn't matter. The GC does not know what heap allocations are in memory vs swap, and since you don't write applications thinking about that, running a VM with a moving GC on swap is a bad idea.
Python’s not a mover but the cycle breaker will walk through every object in the VM.
Also since the refcounts are inline, adding a reference to a cold object will update that object. IIRC Swift has the latter issue as well (unless the heap object’s RC was moved to the side table).
MemBalancer is a relatively new analysis paper that argues having swap allows maximum performance by allowing small excesses, that avoids needing to over-provision ram instead. The kind of gc does not matter since data spends very little time in that state and on the flip side, most of the time the application has twice has access to twice as much memory to use
A moving collector has to move to somewhere and, generally by it's nature, it's constantly moving data all across the heap. That's what makes it end up touching a lot more memory while also requiring more memory. On minor collections I'll move memory between 2 different locations and on major collections it'll end up moving the entire old gen.
It's that "touching" of all the pages controlled by the GC that ultimately wrecks swap performance. But also the fact that moving collector like to hold onto memory as downsizing is pretty hard to do efficiently.
Non-moving collectors are generally ultimately using C allocators which are fairly good at avoiding fragmentation. Not perfect and not as fast as a moving collector, but also fast enough for most use cases.
Java's G1 collector would be the worst example of this. It's constantly moving blocks of memory all over the place.
> It's that "touching" of all the pages controlled by the GC that ultimately wrecks swap performance. But also the fact that moving collector like to hold onto memory as downsizing is pretty hard to do efficiently.
The memory that's now not in use, but still held onto, can be swapped out.
This is really interesting and I've never really heard about this. What is going on with the kernel team then? Are they just going to keep swap as-is for backwards compatibility then everyone else just disables it? Or if this advice just for high performance clusters?
No. I use swap for my home machines. Most people should leave swap enabled. In fact I recommend the setup outlined in the kernel docs for tmpfs: https://docs.kernel.org/filesystems/tmpfs.html which is to have a big swap and use tmpfs for /tmp and /var/tmp.
As someone else said, swap is important not only in the case the system exhaust main memory, but it's used to efficiently use system memory before that (caching, offload page blocks to swap that aren't frequently used etc...)
The beauty of ZRAM is that on any modern-ish CPU it's surprisingly fast. We're talking 2-3 ms instead of 2-3 seconds ;)
I regularly use it on my Snapdragon 870 tablet (not exactly a top of the line CPU) to prevent OOM crashes (it's running an ancient kernel and the Android OOM killer basically crashes the whole thing) when running a load of tabs in Brave and a Linux environment (through Tmux) at the same time.
ZRAM won't save you if you do actually need to store and actively use more than the physical memory but if 60% of your physical memory is not actively used (think background tabs or servers that are running but not taking requests) it absolutely does wonders!
On most (web) app servers I happily leave it enabled to handle temporary spikes, memory leaks or applications that load a whole bunch of resources that they never ever use.
I'm also running it on my Kubernetes cluster. It allows me to set reasonable strict memory limits while still having the certainty that Pods can handle (short) spikes above my limit.
My understanding was that if you're doing random access - ZRAM has near-zero overhead. While data is being fetched from RAM, you have enough cycles to decompress blocks.
My 2cents is that in a lot of cases swap is being used for unimportant stuff leave more RAM for your app. Do a "ps aux" and look at all the RAM used by weird stuff. Good news is those things will be swapped out.
Example on my personal VPS
$ free -m
total used free shared buff/cache available
Mem: 3923 1225 328 217 2369 2185
Swap: 1535 1335 200
Is it possible you misread the comment you're replying to? They aren't recommending adding swap, they're recommending adjusting the memory tunables to make the OOM killer a bit more aggressive so that it starts killing things before the whole server goes to hell.
in either case, what do you do? if you can't reach a box and it's otherwise safe to do so, you just reboot it. so is it just a matter of which situation occurs more often?
The thing is you can survive memory exhaustion if the oom killer can do its job, which it can't many times when there's swap. I guess the topmost response to this thread talks about an earlyoom tool that might alleivate this, but I've never used it, and I don't find swap helpful anyway so there's no need for me to go down this route.
YMMV. Garbage-collected/pointer-chasing languages suffer more from swapping because they touch more of the heap all the time. AWS suffers more from swap because EBS is ridiculously slow and even their instance-attached NVMe is capped compared physical NVMe sticks.
what an ignorant and clueless comment. Guess what? Todays disks are NVMe drives which are orders of magnitude faster than the 5400rpm HDDs of the 90s. Today's swap is 90s RAM.
Does HDD vs SSD matter at all these days? I can think of certain caching use-cases where swapping to an SSD might make sense, if the access patterns were "bursty" to certain keys in the cache
It's still extremely slow and can cause very unpredictable performance. I have swap setup with swappiness=1 on some boxes, but I wouldn't generally recommend it.
> Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot.
My experience is the exact opposite. If anything 2-3 second button clicks are more common than ever today since everything has to make a roundtrip to a server somewhere whereas in the 90s 2-3s button click meant your computer was about to BSOD.
Edit: Apple recently brought "2-3s to open tab" technology to Safari[1].
It's not just 3 seconds for a button click, every time I've run out of RAM on a Linux system, everything locks up and it thrashes. It feels like 100x slowdown. I've had better experiences when my CPU was underclocked to 20% speed. I enable swap and install earlyoom. Let processes die, as long as I can move the mouse and operate a terminal.
Yup, this is a thing. It happens because file-backed program text and read-only data eventually get evicted from RAM (to make room for process memory) so every access to code and/or data beyond the current 4K page can potentially involve a swap-in from disk. It would be nice if we had ways of setting up the system so that pages of code or data that are truly critical for real-time responsiveness (including parts of the UI) could not get evicted from RAM at all (except perhaps to make room for the OOM reaper itself to do its job) - but this is quite hard to do in practice.
I have also seen this in Androids (I tested this on multiple devices - S23U, OnePlus 6,8) , whenever I completely turned off the swap , the phone after a day or two of heavy usage would sometimes hang!
It felt unintuitive since these devices had lot of RAM, and they shouldn't need swap . But turning off swap has always degraded performance for me.
Because some portion of the RAM used by your daemons isn't actually being accessed, and using that RAM to store file cache is actually a better use than storing idle memory. The old rule about "as much swap as main memory" definitely doesn't hold any more, but a few GB to store unneeded wired memory to dedicate more room to file cache is still useful.
As a small example from a default Ubuntu installation, "unattended-upgrades" is holding 22MB of RSS, and will not impact system performance at all if it spends next week swapped out. Bigger examples can be found in monolithic services where you don't use some of the features but still have to wire them into RAM. You can page those inactive sections of the individual process into swap, and never notice.
There is absolutely no point to doing that, which is why file cache is never swapped out. The swapped part is not-recently-used, wired memory from processes, so that there is more room for file cache.
Like a highway brake failure ramp, you have room for handling failures gentler. So services don't just get outright killed. If you monitor your swap usage, any usage of swap gives you early warning that your services require more memory already.
Gives you some time to upgrade, or tune services before it goes ka-boom.
If your memory usage is creeping up, the way you'll find out that you need more memory is by monitoring memory usage via the same mechanisms you'd hypothetically use to monitor your swap usage.
If your memory usage spikes suddenly, a nominal amount of swap isn't stopping anything from getting killed; you're at best buying yourself a few seconds, so unless you spend your time just staring at the server, it'll be dead anyways.
Yeah I had a few servers look up on me without any clear way to recovery because some app was eating up ram. I am ok with the server coming to a crawl as soon as the swap has to be used but at least it won't stop responding all together.
sometimes swap seems to accumulate even though there is plenty of ram. It is too "greedy" by default, probably set for desktops not servers in mind.
Therefore it is better to always tune "vm.swappiness" to 1 in /etc/sysctl.conf
You can also configure your web server / TCP stack buffers / file limits so they never allocate memory over the physical ram available. (eg. in nginx you can setup worker/connection limits and buffer sizes.)
"The practice of System and Network administration" by Tom Limoncelli and Christine Hogan[1] was, together with "Principles of Network and Systems Administration" by Mark Burgess have probably been the books that influenced my approach to sysadmin the most. I still have them. Between them they covered at a high level (at least back when I was sysadmin before devops and Kubernets etc) anything and everything from
- hardware, networks, monitoring, provisioning, server room locations in existing buildings, how to prepare server rooms
- and so on up to hiring and firing sysadmins, salary negotiations[2], vendor negotiations and the first book even had a whole chapter dedicated to "Being happy"
[1] There is a third author as well now, but those two were the ones that are on the cover of my book from 2005 and that I can remember
[2] Has mostly worked well after I more or less left sysadmin behind as well
Depends on the algorithm (and how much CPU is in use); if you have a spare CPU, the faster algorithms can more-or-less keep up with your memory bandwidth, making the overhead negligible.
And of course the overhead is zero when you don't page-out to swap.
Swap to disk involves a relatively small pipe (usually 10x smaller than RAM). So instead of paying the cost to page out to disk immediately, you create compressed pages and store that in a dedicated RAM region for compressed swap.
This has a number of benefits: in practice more “active” space is freed up as unused pages are compressed and often compressible. Often times that can be freed application memory that is reserved within application space but in the free space of the allocator, especially if that allocator zeroes it those pages in the background, but even active application memory (eg if you have a browser a lot of the memory is probably duplicated many times across processes). So for a usually invisible cost you free up more system RAM. Additionally, the overhead of the swap is typically not much more than a memcpy even compressed which means that you get dedup and if you compressed erroneously (data still needed) paging it back in is relatively cheap.
It also plays really well with disk swap since the least frequently used pages of that compressed swap can be flushed to disk leaving more space in the compressed RAM region for additional pages. And since you’re flushing retrieving compressed pages from disk you’re reducing writes on an SSD (longevity) and reducing read/write volume (less overhead than naiive direct swap to disk).
Basically if you think of it as tiered memory, you’ve got registers, l1 cache, l2 cache, l3 cache, normal RAM, compressed swap RAM, disk swap - it’s an extra interim tier that makes the system more efficient.
> zram, formerly called compcache, is a Linux kernel module for creating a compressed block device in RAM, i.e. a RAM disk with on-the-fly disk compression. The block device created with zram can then be used for swap or as a general-purpose RAM disk
To clarify OP's represention of the tool, it compresses swap space not resident ram. Outside of niche use-cases, compressing swap has overall little utility.
Incorrect, with zram you swap ram to compressed ram.
It has the benefit of absorbing memory leaks (which for whatever reason compress really well) and compressing stale memory pages.
Under actual memory pressure performance will degrade. But in many circumstances where your powerful CPU is not fully utilized you can 2x or even 3x your effective RAM (you can opt for zstd compression). zram also enables you to make the trade-off of picking a more powerful CPU for the express purpose of multiplying your RAM if the workload is compatible with the idea.
PS: On laptops/workstations, zram will not interfere with an SSD swap partition if you need it for hibernation. Though it will almost never be used for anything else if you configure your zram to be 2x your system memory.
> Incorrect, with zram you swap ram to compressed ram.
That reads like what they said? You reserve part of the RAM as a swap device, and memory is swapped from resident RAM to the swap ramdisk, as long as there’s space on there. And AFAIK linux will not move pages between swap devices because it doesn’t understand them beyond priority.
Zswap actually seems strictly better in many cases (especially interactive computers / dev machines) as it can more flexibly grow / shrink, and can move pages between the compressed RAM cache and the disk swap.
Some workloads may do better with zswap. Cache is compressed, and pages evicted to disk based swap on an LRU basis.
The case of swap thrashing sounds like a misbehaving program, which can maybe be tamed by oomd.
System responsiveness though needs a complete resource control regime in place, that preserves minimum resources for certain critical processes. This is done with cgroupsv2. By establishing minimum resources, the kernel will limit resources for other processes. Sure, they will suffer. That’s the idea.
Of course swap should be enabled. But oom killer has always allowed access to an otherwise unreachable system. The pause is there so you can impress your junior padawan who rushed to you in a hurry.
To enable a swap file in Linux, first create the swap file using a command like sudo dd if=/dev/zero of=/swapfile bs=1G count=1 for a 1GB file. Then, set it up with sudo mkswap /swapfile and activate it using sudo swapon /swapfile. To make it permanent, add /swapfile swap swap defaults 0 0 to your /etc/fstab file.
Works really well with no problems that I've seen. Really helps give a bit more of a buffer before applications get killed. Like others have said, with SSD the performance hit isn't too bad.
Get better VPS then. Openvz and other kernel paravirtualization have limits, go for Xen or KVM instead (Xen has paravirtualization as well, but I'm not sure how much it's actually used). Full virtualization (implemented by Xen and KVM) do not allow you to prevent swap from being used.
well once you "need" that swap, it will be writing pages across the network due to the storage being external to the physical server, so the latency is terrible
They both offer virtualized guests under a hypervisor host. EC2 does have more offload specialization hardware but for the most part they are functionally equivalent, unless I'm missing something...
Reminds me of my current customer.
We (another freelancer and me) built an application that replaced an Excel sheet, which was the foundation of the business until then. So the usual so far.
We have a policy that our customers are responsible for all their business-related input, but we make the decisions about the technical implementation.
Every technical decision that the customer wants to make basically costs extra.
In this case we built a rather simple multi-tenancy B2B app using Laravel, with one database per tenant.
They planned to start with a single customer/tenant, scaling up to maybe a few dozen within the next years, with less than a hundred concurrent users over the first five years.
There were some processes with a little load, but they were few, running less that a minute each and already built up to run asynchronous.
We planned a single Hetzner instance and to scale up as soon as we would see it reaching its limits.
So less than 100 €/month.
The customer told us that they have a cooperation with their local hosting provider (with "special conditions!") and that they wanted to use them instead.
My colleague did all the setup, because he is more experienced in that, but instead of our usual five-minute-setup in Forge (one of the advantages of the Laravel ecosystem), it took several weeks with the hosting provider, where my colleague had to invest almost full time just for the deployment.
The hosting provider "consulted" out customer to invest in a more complex setup with a load balancer in front, to be able to scale right away.
They also took very long for each step, like providing IP addresses or to handle the SSL certificates.
We are very proud of our very fast development process and having to work with that hosting provider cost us about one third of our first development phase for the initial product.
It's been around two years since then.
While the software still works as intended, the customer could not grow as expected. They are still running with only one single tenant (basically themselves) and the system barely had to handle more than two concurrent users.
The customer recently accidentally mentioned that they pay almost 1000€/month for the hosting alone.
But it scales!
I can only sympathize here because I have those exact issues with some of our customers
They don't want our hosting solutions but insist on using their own hosting partners
The result are similar:
- its at least five times more expensive on pure hosting costs
- we lose a considerable amount of time dealing with the hosting partner (which we bill to the customer)
- it's always a security nightmare, either because they put so much "safety protections" in place that it's unusable (think about the customer wanting an Internet-facing website, but the servers are private...) or because they don't put any safety settings in place so the servers are regularly taken down through awfully simple exploits (think about SSH root access with "passw0rd" as password...)
- customer keep complaining about performances to us, but what can you do when the servers are sharing a 100Mbps connection, or the filesystem is on an NFS with <20Mbps bandwidth
This smells a lot like IONOS. They can put their certifications where the light doesn't shine. 10x the cost, really baaad provisioning API and bottlenecks, broken OS images, useless support...
Super interesting, and truly unfortunate when that happens! Just thinking about having to wait for SSL certificates like in the old days (versus Let's Encrypt) would frustrate me to no end.
Forge seems like a great integrated solution (I subscribe to their newsletter and their product updates seem quite frequent and useful). What's been your experience with them? Any particular things you like, or dislike about them?
I'm also curious when you talk about scaling up Forge - is that something you've done, and is that generally easy to do?
Local hosting can make sense. Being able to drive to your provider and talk to them in person is quite valuable, and if you want to get the highest support tier from a large cloud provider you will often pay several times more compared to the same service with no support, assuming you are a large enough customer that they are willing to sell it. Cooperation with local businesses can also result in some fair amount of additional sales (sending customer to each other, buying services from each other, word of mouth, ectra), so the product cost may not represent the complete picture.
Local hosting can also be comparing apple with oranges. A local data center that provide a physical machine is very different from a cloud provider, especially if that cloud is located in a different continent and under different jurisdictions. Given that they were providing SSL certificates, was this a local php webshop? Data centers should be a bit more proficient with things like IP addresses and setting up any cast, but less so in providing help with php or certificates, and if they sell that it may not be their area of expertise.
Just saw Nate Berkopec who does a lot of rails performance stuff posting about the same idea yesterday saying Heroku is 25-50x price for performance which is so insane. They clearly have zero interest in competing on price.
It's a shame they don't just license all their software stack at a reasonable price with a similar model like Sidekiq and let you sort out actually decent hardware. It's insane to consider Heroku if anything has gotten more expensive and worse compared to a decade ago yet in comparison similar priced server hardware has gotten WAY better of a decade. $50 for a dyno with 1 GB of ram in 2025 is robbery. It's even worse considering running a standard rails app hasn't changed dramatically from a resources perspective and if anything has become more efficient. It's comical to consider how many developers are shipping apps on Heroku for hundreds of dollars a month on machines with worse performance/resources than the macbook they are developing it on.
It's the standard playback that damn near everything in society is going for though just jacking prices and targeting the wealthiest least price sensitive percentiles instead of making good products at fair prices for the masses.
Jacked up prices isn't what is happening here. There is a psychological effect that Heroku and other cloud vendors are (wittingly or unwittingly) the beneficiary of. Customer expectations are anchored in the price they pay when they start using the service, and without deliberate effort, those expectations change in _linear_ fashion. Humans think in linear terms, while actual compute hardware improvements are exponential.
Heroku's pricing has _remained the same_ for at least seven years, while hardware has improved exponentially. So when you look at their pricing and see a scam, what you're actually doing is comparing a 2025 anchor to a mid-2010s price that exists to retain revenue. At the big cloud vendors, they differentiate customers by adding obstacles to unlocking new hardware performance in the form of reservations and updated SKUs. There's deliberate customer action that needs to take place. Heroku doesn't appear to have much competition, so they keep their prices locked and we get to read an article like this whenever a new engineer discovers just how capable modern hardware is.
I mean Heroku is also offering all of the ancillary stuff around their product. It's not literally "just" hosting. It's pretty nice to not have to manage a kube cluster, to get stuff like ephemeral QA envs and the like, etc....
Heroku has obviously stagnated now but their stack is _very cool_ for if you have a fairly simple system but still want all the nice parts of a mode developed ops system. It almost lets you get away with not having an ops team for quite a while. I don't know any other provider that is low-effort "decent" ops (Fly seems to directionally want to be new Heroku but is still missing a _lot_ in my book, though it also has a lot)
Well said. I’ve been expecting an obvious spiritual successor for a long time. They have a surprising number of features compared to most platforms. Their databases/redis and features like forking were quite good (as long as you were super big), logplex/log shipping, auto scale, add-on ecosystem, promotion pipelines, container support if needed (good build packs/git support if you don’t), good CLI or API, OS/patch management, hobby plans and enterprise plans, and more. And on top of all of that, the user/projects system is something mortals can wrap their heads around. They found the sweet spot between raw servers and the complexity quagmire of the mega-clouds a surprisingly long time ago.
There are some folks with good offerings (Fly, Railway, etc), but the feature set of Heroku is deeper, and more important for production apps, than most people realize. They aren’t a good place for hobbyists anymore though. I agree with that.
I think it’s easy to forget how much you get with a modern setup like this, and how much work it is to maintain it. If you’re at a big corp, the team who maintains this stuff is larger than most mid corp’s engineering departments. For a solo person, it’s fine. But if you have 10-30 engineers, it’s a lot of work, and paying heroku $1000/mo is significantly cheaper than having even a junior engineer spend 40% of their time on keeping up.
To be fair, AWS quite proudly talk about all the times they've lowered prices on existing services, or have introduced new generations that are cheaper (e.g. their Graviton EC2 instances).
> It's a shame they don't just license all their software stack at a reasonable price with a similar model like Sidekiq and let you sort out actually decent hardware
We built and open sourced https://canine.sh for exactly that reason. There’s no reason PaaS providers should be charging such a giant markup over already marked up cloud providers.
Heroku is pricing for “# of FTE headcount that can be terminated for switching to Heroku”; in that sense, this article’s $3000/mo bill is well below 1.0 FTE/month at U.S. pricing, so it’s not interesting to Heroku to address. I’m not defending this pricing lens, but it’s why their pricing is so high: if you aren’t switching to Heroku to layoff at least 1-2 FTE of salary per billing period, or using Heroku to replace a competitor’s equivalent replacement thereof, Heroku’s value assigned to you as a customer is net negative and they’d rather you went elsewhere. They can’t slam the door shut on the small fry, or else the unicorns would start up elsewhere, but they can set the pricing in FTE-terms and VCs will pay it for their moonshots without breaking a sweat.
This looks decent for what it is. I feel like there are umpteen solutions for easy self-hosted compute (and tbh even a plain Linux VM isn't too bad to manage). The main reason to use a PAAS provider is a managed database with built-in backups.
Its the flexibility and power of Kubernetes that I think is incredible. Scaling to multiple nodes is trivial, if your entire data plane is blown away, the recovery is trivial.
You can also self host almost any open source service without any fuss, and perform internal networking with telepresence. (For example, if you want to run an internal metabase that is not available on public internet, you can just run `telepresence connect`, and then visit the private instance at metabase.svc.cluster.local).
Canine tries to leverage all the best practices and pre-existing tools that are already out there.
But agreed, business critical databases probably shouldn't belong on Kubernetes.
Fully agreed - our recommendation is to /not/ run your prod Postgres db yourself, but use one of the many great dedicated options out there - Crunchy Data, Neon, Supabase, or AWS RDS..!
It really depends upon how much data you have. If its enough to just dump then go crazy. If it isn't its a bit more trouble.
Regardless, you're going to have a much easier time developing your app if your datastore access latency is submillisecond rather than tens of milliseconds.
You're running at a pretty small scale if running your database locally for sub-milisecond latency is practical. The database solution provided by the DBA team in a data center is going to have about the same latency as RDS or equivalent. Typical intra-datacenter network latency alone is going to be 1-3ms.
They were talking about using things like Supabase, not just RDS.
Also, "small scale" means different things to different people. Given the full topic at hand, I would call it "nano scale". Depending upon your specific schema, you can serve tens of thousands of queries per second with a single server on modern hardware, which is way more than enough for the vast majority of workloads.
> $50 for a dyno with 1 GB of ram in 2025 is robbery
AWS isn't much better honestly.. $50/month gets you an m7a.medium which is 1 vCPU (not core) and 4GB of RAM. Yes that's more memory but any wonder why AWS is making money hand-over-fist..
AWS certainly also does daylight robbery. In the AWS model the normal virtual servers are overpriced, but not super overpriced.
Where they get you is all the ancillary shit, you buy some database/backup/storage/managed service/whatever, and it is priced in dollars per boogaloo, you also have to pay water tax on top, and of course if you use more than the provisioned amount of hafnias the excess ones cost 10x as much.
Most customers have no idea how little compute they are actually buying with those services.
Not sure if it's an apples-to-apples comparison with Heroku's $50 Standard-2X dyno, but an Amazon Lightsail instance with 1GB of RAM and 2 vCPUs is $7/month.
That is assuming you need that 1 core 24/7, you can get 2 core / 8gb for $43, this will most likely fit 90% of workloads (steady traffic with spikes, or 9-5 cadence).
If you reserve that instance you can get it for 40% cheaper, or get 4 cores instead.
Yes it's more expensive than OVH but you also get everything AWS to offer.
This, plus as a backup plan going from Heroku to AWS wouldn't necessarily solve the problem, at least with our infra. When us-east-1 went down this week so did Heroku for us.
Heroku is the Vercel of Rails: people will pay a fortune for it simply because it works. This has always been their business model, so it’s not really a new development. There’s little competition since the demand isn’t explosive and the margin is thin, so you end up with stagnation
You'd be surprised. There are very few because it takes a lot more work to build reliable systems across mid-market cloud providers (flakey APIs, missing functionality, etc). Plus you need to know the idiosyncrasies of all the various frameworks + build systems.
That said, they are emerging. I'm actually working on a drop-in Vercel competitor at https://www.sherpa.sh. We're 70% lower cost by running on EU based CDN and dedicated servers (Hetzner, etc). But we had to build the relationships to solve all the above challenges first.
I am not sure what's there to license. The hard and expensive part is in the labor to keep everything running. You are paying to make DevSecOps Somebody Else's Problem. You are paying for A Solution. You are not paying for software. There are plenty of Heroku clones mentioned in this thread.
I know you mean this sarcastically but I actually 100% agree with this particular on the steak point. Especially with beef prices at all time record highs and restaurant inflation being out of control post pandemic. It takes so much of the enjoyment out of things for me if I feel i'm being ripped off left and right.
What you're missing here is that companies happily pay the premium to Heroku because it lets them focus on building the product and generating business rather than wasting precious engineering time managing infra.
By the time the product is a success and reaches a scale where it becomes cost prohibitive, they have enough resources to expand or migrate away anyway.
I suppose for solo devs it might be cheaper to setup a box for fun, but even then, I would argue that not everyone enjoys doing devops and prefers spending their time elsewhere.
Maybe what bothers people so much is more of the fact that when Heroku first came out, it was much harder to do what that platform does. In the past 20 years or so, there has been a ton of improvement in the tools available. What could’ve taken you three full-time employees can probably be done with 20% of someone’s time after the initial set up which also isn’t that hard. So, it seems like instead of charging like 50X the cost of the servers themselves, maybe Heroku could be charging 10X. But it seems like salesforce probably just bought Heroku as a cash-generating machine. They probably figure they have a lot more to lose in cutting the bills of their old customers who don’t want to migrate anything, then they could gain from attracting new customers who aren’t already locked in.
Honestly, reading these threads it sounds to me like a lot of people are still launching new projects on Heroku. I wouldn’t have guessed that was true before reading this.
Not the best comment but I agree with the sentiment. I fear far too often, people complain about price when there are competitors/other cheaper options that could be used with a little more effort. If people cared so much then they should just use the alternative.
No one gets hurt if someone else chooses to waste their money on Heroku so why are people complaining? Of course it applies in cases where there aren't a lot of competitors but there are literally hundreds of different of different options for deploying applications and at least a dozen of them are just as reliable and cheaper than Heroku.
I'm hurt because a service I'm using is based on Heroku. I'm on the "unlimited" plan but they have backtracked on that and now say I'm too big for them...
The problem with Heroku's pricing is that it's set high enough that I no longer use it and neither does anyone else I know. I suspect they either pivoted to a different target market than me, which would be inconvenient but I'd be okay with it, or killed off their own growth potential by trying to extract revenue, which I would find sad.
I’m pretty sure their target market is people who have already built something kind of complex on there and don’t have the time/money budget to do a big migration. In that way, they know their customers are stuck but can afford the current prices, so keeping pricing static or gradually increasing makes sense.
This argument doesn't work with such commoditized software. It's more like comparing an oil change for $100 plus an hour of research and a short drive against a convenient oil change right next door for $2,500.
Nobody is forced to go to the expensive one. If they are still in business then enough people apparently consider it a reasonable deal. You might not, but others do. Whether I'm being downvoted or not.
It's just trendy to bash cloud and praise on-premises in 2025. In a few years that will turn around. Then in another few years it will turn around again.
The cloud has made people forget how far you can get with a single machine.
Hosting staging envs in pricey cloud envs seems crazy to me but I understand why you would want to because modern clouds can have a lot of moving parts.
Teaching a whole bunch of developers some cloud basics and having a few cloud people around is relatively cheap for quite a while. Plus, having test/staging/prod on similar configurations will help catch mistakes earlier. None of that "localstack runs just fine but it turns out Amazon SES isn't available in region antartica-east-1". Then, eventually, you pay a couple people's wages extra in cloud bills, and leaving the cloud becomes profitable.
Cloud isn't worth it until suddenly it is because you can't deploy your own servers fast enough, and then it's worth it until it exceeds the price of a solid infrastructure team and hardware. There's a curve to how much you're saving by throwing everything in the cloud.
Deploying to your private cloud requires basically the same skills. Containers, k8s or whatnot, S3, etc. Operating a large DB on bare metal is different from using a managed DB like Aurora, bit for developers, the difference is hardly visible.
RDS/managed database is extremely nice I will admit, otherwise I agree. Similarly s3, if you're going to do object storage, then running minio or whatever locally is probably not cheaper overall than R2 or similar.
The cloud has made people afraid of linux servers. The markup is essentially just the price business has to pay because of developer insecurity. The irony is that self hosting is relatively simple, and alot of fun. Personally never got the appeal of Heroku, Vercel and similar, because theres nothing better than spinning up a server and setting it up from scratch. Every developer should try it.
> The irony is that self hosting is relatively simple, and alot of fun. Personally never got the appeal of Heroku, Vercel and similar, because theres nothing better than spinning up a server and setting it up from scratch.
It's fun the first time, but becomes an annoying faff when it has to be repeated constantly.
In Heroku, Vercel and similar you git push and you're running. On a linux server you set up the OS, the server authentication, the application itself, the systemctl jobs, the reverse proxy, the code deployment, the ssl key management, the monitoring etc etc.
I still do prefer a linux server due to the flexibility, but the UX could be a lot better.
> It's fun the first time, but becomes an annoying faff when it has to be repeated constantly.
I have to ask - do scripts not work for you?
When I had to do this back in 2005 it was automated with 3 main steps:
1. A preseed (IIRC) debian installation disc (all the packages I needed where installed at install time), and
2. Which included a first-boot bash script that retrieved pre-compiled binaries from our internal ftp site, and
3. A final script that applied changes to the default config files and ran a small test to ensure everything started.
Zero human interaction after powering a machine on with the disc in the drive.
These days I would do it even better (system-d configs, Nix perhaps, text files (such as systemd units) can be retrieved automagically after boot, etc).
I use NixOS and a lot of it is in a single file. I just saw some ansible coming by here, and although I have no experience with it, it looked a lot simpler than Nix (for someone from the old Linux world, like me… eventhough Nix is, looking through your eyelashes, just a pile of key/value pairs).
Ansible, Salt and Puppet are mostly industry standard. Those tools are commonly referred to as configuration management (systems).
Ansible basically automates the workflow of: log in to X, do step X (if Y is not present). It has broad support for distros and OSes. It's mostly imperative and can be used like a glorified task runner.
Salt let's you mostly declaratively describe the state of a system. It comes with a agent/central host system for distributing this configuration from the central host to the minions (push).
Puppet is also declarative and also comes with an agent/central host system but uses a pull based approach.
Specialized/ exotic options are also available, like mgmt or NixOS.
Salt and Puppet are useful for managing a fleet of servers running various applications, especially when you need to scale those applications horizontally or want geo-distribution.
Ansible can also do that, on top of literally anything else you could want - network configuration, infrastructure automation, deployment pipelines, migrations, anything. As always, that flexibility can be a blessing or a curse, but I think Ansible manages it well because it's so KISS.
RedHat's commercial Ansible Automation Platform gives you more power for when you need it, but you don't need it starting out.
The other commenter already answered the usecase question, for self-hosting you will likely find ansible the easiest entrypoint.
It is in general the simplest of these systems to get started with and you should be able to incrementally adopt it. There is also a plethora of free online resources available for it.
A combination of HashiCorp Packer and Ansible means I can "publish" a VM ready-to-rock image to a public cloud provider gallery and use it to run a VM in said cloud.
Ansible-Lockdown is another excellent example of how Ansible can be used to harden servers via automation.
Takes less than a day, because most of the stuff is scriptable. And for a simple compute node setup at Hetzner (I.e. no bare metal, but just a VM) it takes me less than half an hour.
I dunno, the cloud has mostly made me afraid of the cloud. You can bury yourself in towering complexity so easily on AWS. (The highly managed stuff like Vercel I don't have much experience with, so maybe it's different.)
I will recommend to try GCP or Azure, the complexity is lower there! AWS is great for big corporate that needs a lot of lego pieces to do their own custom setup. At the contrario, GCP and Azure solutions are often more bundled.
"Self hosting" may actually be referring not to hosting your own on-prem hardware, but to renting bare metal in which case the concerns around power usage, networking, etc. are offloaded to the provider.
my take is that it's fun up until there's just enough brittleness and chaos.. too many instance of the same thing but with too many env variables setup by hand and then fuzzy bug starts to pile up
Honestly I think it's the database that makes devs insecure. The stakes are high and you usually want PITR and regular backups even for low traffic apps. Having a "simple" turnkey service for this that can run in any environment (dedicated, VPS, colo, etc.) would be huge.
I think this is partly responsible for the increased popularity of sqlite as a backend. It's super simple and lightstream for recovery isn't that complicated.
Most apps don't need 5 9s, but they do care about losing data. Eliminate the possibility of losing data, without paying tons of $ to also eliminate potential outages, and you'll get a lot of customers.
Never got the appeal of having someone else do something for you, and giving them money, in exchange for goods and services? Vercel is easy. You pay them to make it easy. When you're just getting started, you start on easy mode before you jump into the deep end of the pool. Everybody's got a different cup of tea, and some like it hot and others like it cold.
Flour, salt, and water are exceedingly cheap. I have to imagine the loaf of bread I buy from my baker reflects considerably more than a 50x markup compared to baking my own.
It’s a lot cheaper than me learning to bake as well as he does—not to mention dedicating the time every day to get my daily bread—and I’ll never need bread on the kind of scale that would make it worth my time to do so.
Bread is a great example! You can buy a loaf for $3-4. It is not a 50x markup. Like growing your own veggies, baking bread is for fun, not for economics.
But the cloud is different. None of the financial scale benefits are passed on to you. You save serious money running it in-house. The arguments around scale have no validity for the vast, vast majority of use cases.
Vercel isn't selling bread: they're selling a fancy steak dinner, and yes, you can make steak at home for much less, and if you eat fancy steak dinners at fancy restaurants every night you're going to go broke.
So the key is to understand whether your vendors are selling you bread, or a fancy steak dinner, and to not make the mistake of getting the two confused.
That’s a tremendously clarifying framework, and it makes a lot of sense to me. Thank you.
I wonder, though—at the risk of overextending the metaphor—what if I don’t have a kitchen, but I need the lunch meeting to be fed? Wouldn’t (relatively expensive) catering routinely make sense? And isn’t the difference between having steak catered and having sandwiches catered relatively small compared to the alternative of building out a kitchen?
What if my business is not meaningfully technical: I’ll set up applications to support our primary function, and they might even be essential to the meat of our work. But essential in the same way water and power are: we only notice it when it’s screwed up. Day-to-day, our operational competency is in dispatching vehicles or making sandwiches or something. If we hired somebody with the expertise to maintain things, they’d sit idle—or need a retainer commensurate with what the Vercels and Herokus of the world are charging. We only need to think about the IT stuff when it breaks—and maybe to the extent that, when we expect a spike, we can click one button to have twice as much “application.”
In that case, isn’t it conceivable that it could be worth the premium to buy our way out of managing some portion of the lower levels of the stack?
In that case, you don't want cloud; you want an MSP, whose core competence is running those IT services. They, in turn, have the skills to colo a rack at a DC or to manage rented servers, amortized across a number of clients.
In practice, there are two situations where cloud makes sense:
1. You infrequently need to handle traffic that unpredictably bursts to a large multiple of your baseline. (Consider: you can over provision your baseline infrastructure by an order of magnitude before you reach cloud costs)
2. Your organization is dysfunctional in a way that makes provisioning resources extremely difficult but cloud can provide an end run around that dysfunction.
Note that both situations are quite rare. most industries that handle that sort of large burst are very predictable: event management know when a client will be large and provision ticket sales infra accordingly, e-commerce knows when the big sale days will be, and so on. In the second case, whatever organizational dysfunction caused the cloud to be appealing will likely wrap itself around the cloud initiative as well.
Please do yourself a flavour and check the price of flour.
Water is cheap, yes. Salt isn't all that cheap, but you only need a little bit.
> [...] and I’ll never need bread on the kind of scale that would make it worth my time to do so.
If you need bread by hand, it's a very small scale affair. Your physique and time couldn't afford you large scale bread making. You'd a big special mixer and a big special oven etc for that. And you'd probably want a temperature and moisture controlled room just for letting your dough rise.
I blush to admit that I do from time to time pay $21 for a single sourdough loaf. It’s exquisite, it’s vastly superior to anything I could make myself (or anything I’ve found others doing). So I’m happy to pay the extreme premium to keep the guy in business and maintain my reliable access to it.
It weighs a couple of pounds, though I’m not clear how the water weight factors in to the final weight of a loaf. And I’m sure that flour is fancier than this one. I take your point—I don’t belong in the bread industry :)
Well, in your case, you are mostly paying for the guy's labour, I presume.
(Similarly to how you pay Amazon or Google etc not just for the raw cloud resources, but for the system they provide.)
I grew up in Germany, but now live in Singapore. What's sold as 'good' sourdough bread here would make you fail your baker's training in Germany: huge holes in the dough and other defects. How am I supposed to spread butter over this? And Mischbrot, a mixture of rye and wheat, is almost impossible to find.
So we make our own. The goal is mostly to replicate the everyday bread you can buy in Germany for cheap, not to hit any artisanal highs. (Though they are massively better IMHO than anything sold as artisanal here.)
Interestingly, the German breads we are talking about are mostly factory made. Factory bread can be good, if that's what customers demand.
Going on a slight tangent: with tropical heat and humidity, non-sourdough bread goes stale and moldy almost immediately. Sourdough bread can last for several days or even a week without going moldy in a paper bag on the kitchen counter outside the fridge, depending on how sour you go. If you are willing to toast your bread, going stale during that time isn't much of an issue either.
(Going dry is not much of an issue with any bread here--- sourdough or not, because it's so humid.)
Wait, what? Salt is literally one of the cheapest of all materials per kilogram that exists in all contexts, including non-food contexts. The cost is almost purely transportation from the point of production. High quality salt is well under a dollar a pound. I am currently using salt that I bought 500g for 0.29 euro. You can get similar in the US (slightly more expensive).
This was a meme among chemical engineers. Some people complain in reviews on Amazon that the salt they buy is cut with other chemicals that make it less salty. The reality is that there is literally nothing you could cut it with that is cheaper than salt.
One way or another, salt is not a major driver of cost in bread, because there's relatively little salt in bread. (If there's 1kg of flour, you might have 20g of salt.)
Yeah, but then we're just haggling. If you know how to change the belt on your car and already have the tools, it's different from when you're stranded with no tools and no garage and no belt.
If you're a mechanic you're supposed to know how to change the belt on your car. It would be insane if you write code and work with computers for a living but you dont know how to set up a web server.
I am pretty sure I know much more about code than you do, and at the same time you probably know much more about web servers and sysadmin than I do. I don't mind if it stays like that. And I am saying this having programmed my own web server in Java about 25 years ago.
Hum... Writing a game engine is a high-difficulty task that should be available to any reasonably good software developer with a few months to study for it. Making it in assembly is a sure way to take 10 times the time of another low level language like C, but shouldn't be an impossibility either.
Configuring a web server is a low-difficulty task that should be available for any good software developer with 3 days to study for it. It's absurd for a developer to need to configure a web server, but insist on paying a large rent and cede control to some 3rd party instead of just doing it.
Fucking every web-dev assumes web-dev is all of programming. I have always been a game dev, never done any internety stuff, I was never interested in it. I would defo find the game engine task a lot easier. I already know what I would do. I wouldnt know where to start with the server + I dont know what the "gotchas" are. If I was forced to do that I would schedule a really long time for it.
I don’t assume that (and am not a dev - DBRE / SRE) at all. I have massive respect for game devs, since you’re one of the few subsets that seems to give a shit about performance.
I bet you could figure out `apt install nginx` and a basic config pretty quickly, definitely faster than a web dev could learn game programming. “What do you mean, I have to finish each loop in 16 msec?”
The cloud was a good deal in 2006 when the smallest aws machine was about the size of a ok dev desktop and took over two years of renting to justify buying the physical machine outright.
Today the smallest, and even large, aws machines are a joke, comparable to a mobile phone from 15 years ago to a terrible laptop today, and take about three to six months to in rent as buying the hardware outright.
If you're on the cloud without getting 75% discount you will save money and headcount by doing everything on prem.
I think OP is using these less as staging and more as dev environments for individual developers. That seems like a great use of a single server to me.
I'd still like a staging + prod, but keeping the dev environments on a separate beefy server seems smart.
I've been using a development server for about 9 years and the best thing I ever did was move to a machine with a low-power Xeon D for a time. It made development painful enough that I quickly fixed the performance issues I was able to overlook on more powerful hardware. I recommend it, even just as an exercise.
For similar reasons, in the Google office I worked in you had the option to connect to a really intentionally crappy wifi that was simulating a 2G connection.
The "platform" software runs on is just other software. If your prod environment is managed kubernetes then you don't lose much if your staging environment is self-hosted kubernetes.
Load balancers, IAM roles, kubernetes upgrades, postgres upgrades, security settings, DNS records, http routes... there's a lot that can go wrong and makes it useful to have a staging environment.
This could be the premise for a fun project based infra learning site.
You get X resources in the cloud and know that a certain request/load profile will run against it. You have to configure things to handle that load, and are scored against other people.
also how far you can get with a single machine has changed massively in the past 15 years. 15 years ago a (really beefy) single machine meant 8 cores with 256GB ram and a couple TB of storage. Now a single machine can be 256 cores on 8TB of ram and a PB of storage.
Exactly, and the performance of consumer tech is wildly faster. Eg, a Ryzen 5825U mini pc with 16GB memory is ~$250USD with 512GB nvme. That thing will outperform of 14 core Xeon from ~2016 on multicore workloads and absolutely thrash it in single thread. Yes lack of ECC is not good for any serious workload, but great for lower environments/testing/prototyping, and it sips power at ~50W full tilt.
I saw a twitter thread recently where someone tried to make this point to someone shilling AWS spaghetti architectures. They were subsequently dog-piled into oblivion but the mental gymnastics people can do around this subject is a sight to behold.
Simplicity is uncomfortable to a lot of people when they're used to doing things the hard way.
The best part is when you start with a $3000/month cloud bill during development and finally realize that hosting the production instance this way would actually cost $300k/month, but now it's too late to change it quickly.
You put your staging env in the same (kind of) place you put your prod system because you need to replicate your prod environment as faithfully as possible. You also then get to re-use your deployment code.
Do you plan on keeping it in your home? At that point I'd be worried about ISP networking or power guarantees unless you plan on upgrading to business rates for both. If you mean colo, well, if you're sure you'll be using it in X years, it's worth it, but the flexibility of month-to-month might be preferable.
And unless you live in the US and is willing to actually shot someone (with all the paperwork that entails, as well as physical and legal risks), the fact is that you can't actually stop a burglary.
We've had a similar experience at Hack Club, the nonprofit I run that helps high schoolers get into coding and electronics.
We used to be on Heroku and the cost wasn't just the high monthly bill - it was asking "is this little utility app I just wrote really worth paying $15/month to host?" before working on it.
This year we moved to a self-hosted setup on Coolify and have about 300 services running on a single server for $300/month on Hetzner. For the most part, it's been great and let us ship a lot more code!
My biggest realization is that for an organization like us, we really only need 99% uptime on most of our services (not 99.99%). Most developer tools are around helping you reach 99.99% uptime. When you realize you only need 99%, the world opens up.
Disco looks really cool and I'm excited to check it out!
Cheers, let me know if you do / hop onto our Discord for any questions.
We know of two similar cases: a bootcamp/dev school in Puerto Rico that lets its students deploy all of their final projects to a single VPS, and a Raspberry Pi that we've set up at the Recurse Center [0] which is used to host (double checking now) ~75 web projects. On a single Pi!
Just to be aware when you say "Even with all 6 environments and other projects running, the server's resource usage remained low. The average CPU load stayed under 10%, and memory usage sat at just ~14 GB of the available 32 GB."
The load average in htop is actually per CPU core. So if you have 8 CPU cores like in your screenshot, a load average of 0.1 is actually 1.25% (10% / 8) of total CPU capacity - even better :).
Cool blog! I've been having so much success with this type of pattern!
what does this service offer over an established tool like Coolify? currently hosting most of my services on a cheap Hetzner VPS so i'm interested what Disco has to offer
Coolify and other self-hosting options such as Kamal are great. We're all in the same boat!
I'd say the main differences is that we 1) we offer a more streamlined CLI and UI rather than offering extensive app/installation options 2) have an api-key based system that lets team members collaborate without having to manage ssh access/keys.
Generally speaking, I'd say our approach and tooling/UX tends to be more functional/pragmatic (like Heroku) than one with every possible option.
https://devpu.sh/ is another alternative, it has a nice UI built with Hypermedia (HTMX).
I am building https://github.com/openrundev/openrun/. Main difference is that OpenRun has a declarative interface, no need for manual CLI commands or UI operations to manage apps. Another difference is that OpenRun is implemented as a proxy, it does not depend on Traefik/Nginx etc. This allows OpenRun to implement features like scaling down to zero, RBAC access control for app access, audit logs etc.
Downside with OpenRun is that is does not plan to support deploying pre-packaged apps, no Docker compose support. Streamlit/Gradio/FastHTML/Shiny/NiceGUI apps for teams are the target use case. Coolify has the best support and catalog of pre-packaged apps.
It’s all fine and dandy, but I wonder why so little discussion around this (mainly high-level comments “DBs are hard”?
> disco provides a "good enough" Postgres addon.
> This addon is a great way to quickly setup a database when Postgres is not mission critical to your system. If you need any non-basic features, like replication, automatic failover, monitoring, automatic backups and restore, etc. you should consider using a managed Postgres provider, such as Neon or Supabase.
How come automatic backups is considered an “advanced” feature?
Also I can’t think of a single application since 2012 that I have worked on that did not have a secondary/follower instance deployed. Also suggesting Neon and friends is fine, but I wonder what is your average latency, Hetzner does not have direct connection to the DCs these databases are hosted.
Backups are only advanced in the context of our Postgres being "Good" enough (maybe our built-in Posgres could be called "Barely enough" but that sounds a bit lame) :-)
I fully agree with you though, it's table stakes (unintended pun!) for any prod deployment, just as read-only followers, etc. Our biggest, most important point, is that folks should be using real dbs hosted by people who know what they're doing. The risk/reward ratio is out of whack in terms of doing it yourself.
And finally, re Hetzner and cross-DC latency, that's unfortunately a very good issue that we had to plan for in another case - specifically, a customer using Supabase (which is AWS-based). The solution was to simply use an EC2 machine in the same region. Thankfully, some db providers end up being explicit about which AWS region they run in - and obviously, using AWS RDS is also an option! It's definitely a consideration.
The article's title seems inaccurate - as far as I understood there never was a $3000/mo bill; there was a $500/(mo,instance) staging setup that has been rightly optimized to $55/mo before running six instances.
> Critically, all staging environments would share a single "good enough" Postgres instance directly on the server, eliminating the need for expensive managed database add-ons that, on Heroku, often cost more than the dynos themselves.
Heroku also has cheaper managed database add-ons, why not use something like that for staging? The move to self hosting might still make sense, my point is that perhaps the original staging costs of $500/mo could have been lower from the start.
I answered elsewhere with the list of dynos, but the short version is that between the list of services that each deployment required, and the size of the database, it truly (and unfortunately) did end up costing $500 per staging.
Heroku's pricing is wild. About a decade ago I just about fell out of my chair when I found out the startup I was at was burning upwards of $10k/mo just to generate QR codes (made out of html tables so that they would reliably display in emails). It worked out to something like $0.15/code
The lead who wrote it had never even profiled code before, after some changes we cut it down to ~$0.01/per, but that's still insane.
What in the world?? Surely there must be something more than "generate a HTML page with 500 elements". Any edge cloud hosting lets you do that for free.
With hetzner AX162-s [1], 1TB DDR5, ($600/month) you can load your freaking database in Memory and not blink !!
I don't understand why people go to cloud.
The situation is interesting, and self-hosting is indeed a very nice solution often. However, I wanted to comment on the article itself - it seems to be very heavily AI-edited. Anyone who has spent time with LLMs will easily see it. But even that's not the issue; the main issue is that the article is basically a marketing piece.
For example, the "Bridging the Gap: Why Not Just Docker Compose?" section is a 1:1 copy of the points in the "Powerful simplicity" on the landing page - https://disco.cloud/
And this blog post is the (only) case study that they showcase on their main page.
You're absolutely right! Here are some three points why:
- ...
I'm kidding :-)
Our library is open source, and we're very happy and proud that Idealist is using us to save a bit of cash. Is it marketing if you're proud of your work? :-) Cheers
Marketing should be marketing and clearly so. Tech blogs are about sharing information with the community (Netflix Tech blog is a good example) NOT selling something. Marketing masquerading as a tech blog is offputting to a lot of people. People don't like being fooled with embedded advertising and putting ad copy into such pieces is at best annoying.
Nah, people are stupid. Including me. It's all marketing. Netflix's tech blog is marketing to engineers to want to go work there and to promote their product. If you want to see things though the lense that all advertising is bad, you'll make your life miserable because it's all advertising in one way or another.
That's the problem with framing everything that way. This HN comment is marketing for my brand, my username, I sell t-shirts on my website! That's not why I'm commenting here, but the term is that broad because we're using it colloquially. It's a human psychology thing that I get entrapped into too. Calling it out doesn't make it not work. When you use the lens of marketing, your comment is marketing that you are not marketing, which is a specific category and advertising profile to be filed away in a database somewhere, if we go to the theoretical extremes.
What you've done is taken something I've written, redefined a core term in a way I obviously didn't mean, and then told me I'm wrong because of your redefinition.
When you put it that way, you make me sound like an ass. Is that how I'm coming across? What did I redefine? I'm refuting the fairytale where some content is pure and untainted by marketing. Netflix writes posts that make engineers want to work there and people think, "hey, that's smart!" That’s marketing.
I think a big difference is when someone is pretending to be all about something else and tries to sneakily market to you. One thing is getting a free water bottle with an ad, another thing is when someone is inviting you to a "party" with free food and drinks and it turns out to be a MLM "party".
Netflix is giving away free water bottles (I hate them, but I use their fast.com super often to test the speeds), another is pretending to be a blog post, but actually being an ad (if that was the case here). You just feel lied to. You cannot take anything seriously you read there, as it will probably be super biased and you cannot get your time back now.
Maybe not an ass, that's too strong, but it's a common online pattern where someone transforms your point into an entirely different meaning and then disagrees with that transformation. It's annoying.
I'm complaining about thinly veiled ad copy wearing the mask of shared technical notes. This is seen as a bad faith effort by the publisher of such notes and a dirty trick played on the reader. Advertising should announce itself for what it is.
I'm very clearly making a distinction, I like A, I don't like B.
You're taking that, saying I must actually hate both A and B, and by the way C through Z because nobody is 111% pure of heart and everybody must have at least some motivation for doing something and nobody is entirely altruistic.... which is just this crazy extreme that it's clear I don't believe at all.
I like the incentive structure that leads Netflix to produce objectively high quality articles sharing with the community in a way that really seems to be entirely untainted by the motivation.
Ad copy in tech notes does seem to taint the motivation and quality of them, it can be innocent but it doesn't seem like it and is generally irritating to a lot of people.
Dislike of a certain kind of advertising doesn't mean I'm sitting around miserable because nobody is truly altruistic as you suggest, and that the issue. My lines of thinking aren't taken to a silly extreme. A lot of disagreements these days are people reinterpreting their opposition as exclusively extremist and that's a problem.
You keep saying it's clear when it isn't. We don't know what's going on in your mind. Did you know there are people out there that won't eat anything that came from any animal products? That's crazy extreme! But there are tons of vegans out there. So what's seems extreme to one person is someone else's normal, and someone else's normal is extreme.
You say you like A and don't like B. You don't like B because it has X in it. But A also has X in it. So why do you like A but not B? It's not logically consistent. We disagree on how much X is in A. You want X to be clearly marked with red tape. It's not clear how reasonable and feasible that is or isn't. I'm saying if you're looking for X, you're going to find trace amounts of it everywhere once you start looking for it. X isn't some previously unheard of chemical that's gonna give you cancer or leaky gut though, it's other people making money. It's been chosen for us, that money is how the world works. It's not how I would do it, but I'm not in charge of the world, so it's a moot point. Everyone is weird about money in their own special way. I am no exception. What sticks in my craw is when people have problems with other people making money. How they make money is material. I'm not okay with making money off of sex trafficking or CSAM, for example, but advertising a product with an interesting bit of writing beforehand isn't that. So on the spectrum of your kid's painting that they made for you in school with crayon that were ethically sourced and drew on recycled paper, to the in your face red plastic Coca-Cola banner wrapped around the side of a bus that's gonna be fed to whales to choke and die on, where this particular blog post lies is for you to determine for yourself. Where I'm really getting at is that requiring X to be at a certain level has the unintended consequence that only big corporations with giant bags of money can create content that passes this purity test of yours, is, if we do some extrapolating, self-defeating.
I'm not sure you're functionally literate and you're beginning to ramble. So yes you're coming off as an asshole and just shouldn't respond like this. When I glance at your reply and you're bringing up sex trafficking somehow... yeah no thanks. This is the kind of reply definitely not worth engaging in.
> But even that's not the issue; the main issue is that the article is basically a marketing piece.
Why is that an issue? Is it forbidden by HN guidelines? Or would you like all marketing to be marked as such? Which articles aren't marketing, one way or another?
It's funny they have this marketing blog post based on competing on price yet don't disclose any of their pricing on their site only a schedule a meeting which is just about the biggest RED FLAG on pricing there is.
Our library is open source, the price is 0!! :-) Haha
We're actually mostly talking to people (that "schedule a meeting") to see how we can help them migrate their stuff away (from Heroku, Vercel, etc.)
But we're not sure of the pricing model yet - probably Entreprise features like Gitlab does, while remaining open source. It's a tough(er) balance than running a hosted service where you can "just" (over)charge people.
This isn't the first time an article is also marketing. Besides, what is wrong with marketing something via a use case article?
This is a fairly tame example of it and I found it an interesting and useful read, knowing full well it was also marketing.
I guess I'm not quite understanding why you need six staging servers provisioned at $500 a pop? And if you need that because you have a large team...what percentage of your engineering spend is $3000 vs $100k+/yr salaries?
Especially when I got look at the site in question (idealist.org) and it seems to be a pretty boring job board product.
6 staging servers: main, dev, and any branches that you want to let other (non tech people) QA.
As for the staging servers, for each deployment, it was a mix of Performance-M dynos, multiple Standard dynos, RabbitMQ, a database large enough, etc. - it adds up quickly.
Finally, Idealist serves ~100k users per day - behind the product is a lot of boring tech that makes it reliable & fast. :-)
From what I read, they're using them as dev environments. Like running many services at once for a single developer to tie into. That's why they wanted multiple ones, one for each dev.
This thinking definitely drives enterprise products, and is exactly what makes it hard for small companies. "You can pay a lot simply because you clearly can afford to" doesn't lead to great products, even if it often does lead to profitable companies.
> Each new staging environment still required DNS and CDN configuration - not difficult, but not automatic either. The team also took on responsibility for server monitoring, security updates, and handling any infrastructure issues themselves - operational overhead that Heroku had previously covered. And while Hetzner's price-performance is exceptional, its limited presence in the US was a consideration; for this staging workload, it wasn't an issue, but it's a factor for production services targeting US users.
If you factor in salaries, I can see this ending up costing more than Heroku. $3000/m is one (underpaid) developer here in Scandinavia.
That's the equation that's often lacking, and why Heroku is still worth it where I work.
Any VPS you fancy that fits the price/performance/location/support you want, then point Coolify/Dokploy/whatever at it.
I did just this using Coolify, Mythic Beasts running Django & Postgres the other month from Google App Engine. Hilariously easy, even with my extremely rusty skills.
It is worth learning to use Docker Swarm. Deployments are as simple as pushing a new container to your registry and running one command. I built a free CLI tool rove.dev that simplifies provisioning and does service diffing.
Either that or use a PaaS that deploys to VMs. Can't make recommendations here but you could start by looking at Semaphore, Dokku, Dokploy.
Definitely check out swarm. I've heard so many great things from engineers that use it on large projects, and it takes very little time to learn if you already know the docker cli.
I think the bar has been lowered since ChatGPT started instructing people to do things. This question about Python wouldn't even have come up 10 years ago.
https://render.com/ is probably the closest, I'm really enjoying using them. Workflow is the same as heroku, but cheaper, no nightly restarts, supports new python versions etc..
They replaced a $500/mo bill with a $55/mo server. And at the same time increased what they could do before - run 5 extra staging environments!
While saving $445/mo isn't nothing, in my book enabling teams to freely run the staging environments they need is the real win here! Limiting testing resources can be a real drag on momentum in a project!
From looking at your docs, it appears like using and connecting GitHub is a necessary prerequisite for using Disco. Is that correct? Can disco also deploy an existing Docker image in a registry of my choosing without a build step? (Something like this with Kamal: `kamal --skip-push --version latest`)
Correct, GitHub is necessary at this point to deploy code.
However, yes, you can ask Disco to fetch an existing Docker image (we use that to self-host RabbitMQ). An example of deploying Meilisearch's image is here [0] with the tutorial here [1].
Do you typically build your Docker images and push them to a registry? Curious to learn more about your deployment process.
Yes, I try to keep my CI pipelines somewhat platform-agnostic so even though I'm mostly using GitHub, my workflow is typically to first build a Docker image and push it to a registry, then use Kamal to deploy that image.
These discussions often devolve into how Heroku is “expensive”. You’re paying for convenience and not having to worry about something. Every business will evaluate this for themselves. For a profitable business with a lean team, Heroku works great.
Yes, I'm just as curious as you on _why_ does a staging setup needs the same amount of resources as prod.
All of my staging setups are on a ~$15 Hetzner server, with a GitHub Action to `docker compose build && docker compose up -d` remotely, with an Apache service with a wildcard certificate and dynamic host names. We have 3..n staging setups, with each PR spinning up a new staging site just for that PR.
It's been working with us for years, for a team of 10 developers.
It is hilarious, don't get me wrong - I really appreciate more people moving away from these "Hi-Tech" deployment styles and cloud services and the rest, but it is like rediscovering hot water.
It is clear that Heroku is not interested in reducing their prices. But I don’t think this is a Heroku problem. Vercel is also the same, which makes me think there is a fundamental issue with the PaaS business model that stops it from competing on price while the commoditised part their business (data centers) are always reducing their prices.
The challenge I always face with homebrew PaaS solutions is that you always end up moving from managing your app to managing your PaaS.
This might not be true right now but as complexity of your app grows it’s almost always the eventual outcome.
I like that this post and the migration focus solely on staging. Far too often, people report how much money they saved by migrating from Heroku/Render/Fly to a VPS with Coolify/Dokploy/Kamal. What these calculations usually fail to take into account is the additional management and maintenance effort required. While this is fine for side projects and non-critical parts (such as staging/preview environments), it represents a significant burden for productive environments with business-critical services. In this case, a correct and understandable decision was made. Thanks for sharing!
I recently moved a couple of Rails projects to a VPS using Dokploy and couldn't be happier. Apart from the cost savings the bump in available resources is a big plus. It is astonishing how Heroku still gets away with 512 mb memory for $25/m
background: I have been self-hosting apps on VPS for a while now but just started using a tool
Title seems slightly exaggerated since by my reading there was no actual $3000 / month bill? Still a great use-case
This seems like a good idea to have plentiful dev environments and avoid a bad pricing model. If your production instance is still on Heroku, you might still want a staging environment on Heroku since a Hetzner server and your production instance might have subtle differences.
> And while Hetzner's price-performance is exceptional, its limited presence in the US was a consideration; for this staging workload, it wasn't an issue, but it's a factor for production services targeting US users.
What is this referring to? Concerns about capacity if you need to scale up quickly? Or just "political"/marketing considerations about people not being used to being served by a Hetzner server?
Correct! Re: latency, as I just noted elsewhere, if you run your prod database using Crunchy Bridge or Supabase or another big provider (which you absolutely should for prod), that typically means that your db will be running within an AWS region. You would, in most cases, need to run your compute in the same region. So yeah, at that point, Hetzner would be out.
> The Real Insight: Staging Became a Free Commodity
Not free, it became a productivity boost.
You now have a $35k annual budget for the maintenance, other overhead, and lost productivity. What do you spend it on?
> The team also took on responsibility for server monitoring, security updates, and handling any infrastructure issues themselves
For a place that’s paying devs $150k a year that might math out. It absolutely does not for places paying devs $250k+ a year.
One of the great frustrations of my mid career is how often people tried to bargain for more speed by throwing developers at my already late project when what would have actually helped almost immediately was more hardware and tooling. But that didn’t build my boss’ or his bosses’ empires. Don’t give me a $150k employee to train, give me $30k in servers.
Absolutely no surprise at all when devs were complicit with Cloud migrations because now you could ask forgiveness instead of permission for more hardware.
I know everyone in thread today is having fun embracing the hacker ethos and running things themselves.
But I've migrated plenty of companies off custom deployment setups to PAAS and told many ceo's simply what OP above has shared. Even a part time dev ops engineer is still $60000 a year, and that can buy us a LOT on PAAS. Using PAAS you can have effectively zero dev ops, I've also trained non technical people on how to scale their own servers if no devs are around because you just have a web based UI slider.
I consider myself a developer who cares more about the business, risk, profits and runway. A lot of developers don't share this mentality (which is fine btw always need engineers who like engineering for engineering sakes) but in meetings you will have a hard time beating me in an argument if you try to say that running servers ourselves would be "cheaper", and/or even faster, safer and definitely not more stable. (obviously not in all situations, but kind of most for modern crud web apps that don't require complicated compute setups)
I'm probably being overly antagonistic, forgive me for that, though highly recommend questioning the real cost of running your own setups.
Hetzner is flaky though, you can't trust them. You don't want your website taken down because they decided something is suspicious and they randomly banned you without warning.
<Insert every cloud provider here>.
There are countless posts on HN that describe people and businesses being blocked by a cloud provider like GCP.
Hetzner is by all means not an outlier.
They are for me. Got banned and received the explanation that my actual billing details were suspicious (there was nothing suspicious other than possibly my geolocation which is SE Asia). I asked them to clarify and they declined.
Signed up with Contabo without incident and have been a happy customer ever since.
Look, whether or not Hetzner chose to block me out of blatant geo-racism is not at issue here, I wouldn't want to do business with them either way since they declined to give me a reasonable explanation and I wouldn't feel secure with a provider like that.
The draw of a docker-compose-like interface for deployment is so alluring that I have spent the last year or so working on a tool called Defang that takes a compose file and deploys it to the cloud. We don't support Hetzner (yet), but we do support AWS, GCP, and DO. We provision networking, IAM, compute, database, secrets, etc in your cloud account, so you maintain full control, but you also get the ergonomics of compose.
If you are on a PaaS and you want to reduce cost without losing ergonomics and scalability, it might be interesting.
Cool to hear on the savings.
But now the team has to maintain two different deployment models, so you have account for the ongoing cost that your team now has to own and maintain two different processes of deployment (prod & staging).
The key element here is the need to continuously exercise both processes (Heroku + your staging server), to work out both processes & maintain familiarity on both.
Depending on the amount of staff involved in the above, it might eclipse the compute savings, but only OP knows those details. I'm sure they are a smart bunch.
I don't mean to hate but i find it incredibly alarming that i'm lately seeing all the seemingly seniorly positioned people writing articles about how they just realized that you can actually just buy a vps, setup a deployment workflow and write a revealing blog about "drastically cutting costs".
It's like juniors who did not recieve a proper training/education got hired into companies where someone told them to go serverless on some heroku or vercel, or use some incredibly expensive aws service because that's a "modern correct way" to do it, except now they were a developer for long enough to get a "senior" title in their job title now are in positions of actually modelling this architecture themselves
As one of those people who's experiencing this now (I'm a person who's worked in this industry for 25 years and done everything from colocated servers, to VMs, to VPS's, to AWS/GCP, and now going back to VPS/dedicated), I can tell you it's more of a boiling frog situation. The number of servers slowly grows over time and before you know it, you're spending way more than you really thought about.
At a certain scale (or when you do more custom dev), you just don't prioritize it because it's a blip in your financials compared to salaries. And it's not worth spending what might be thousands (or tens of thousands) of dollars of engineering and testing effort to migrate off, when there are other priorities.
And one day you (in our case) realize that you're spending $600/month between Linode/AWS/Fly/Render (cuz we experimented a lot), claude code can help you with some of the stuff, and you want to consolidate it all back to the olden days where you just had 2-3 servers paying a total of $60-75 a month . But you still have to set aside probably a day or two to do it (no longer a couple sprints). So then the math works out where we make that money back within a few months.
Quite sad to see devs nowadays has lost abilities to self-host. I know it can be overwhelming with Linux, networking, db, backup, hardware load.... However, it's not rocket science!
I work at Render (render.com); we have over 4 million developers on the platform, and we've migrated many large (and small) Heroku customers over because of our more modern capabilities and scalable pricing.
You have your range of options - it depends on the size of your team, the kind of apps you're running, etc. The answer can be anything from an "ssh script" to AWS (or K8S), etc.
If you're running something that's too expensive for your taste and can share more information, happy to brainstorm some options.
I was looking on Hetzner after that recent article and their server marketplace has $34/month server that had something like an Intel Core i7 with 64GB RAM and 2x512GB SSDs. Compare that to EC2 pricing.
From having talked to many folks, migrations are psychologically very, very, very very hard.
At least, the "fear" factor (will the new system work? what bugs will it introduce? how much time will I spend, etc.) pushes a lot of folks to accept a very big price differential aka known knowns versus unknowns...
It's understandable really. It's just that once you've migrated, you almost definitely never want to go back :-)
Exactly. For most people, even at $500/month, yes it might be painful but is still dwarfed by the cost of engineer time. And it's also one of those "let's kick the can down the road, everything's working right now, do we want the headache of potentially destabilizing things"
But this is also one of those things where, almost invariably after you do it, you think "Oh man, we should've done this 3 years ago!!! It just ended up taking a day and everything's working beautifully"
...but this CX33 "server" being discussed - is a 6 bucks a month VPS [0]
Normally you build a prototype on laptop and move it out to fat hardware when it outgrows that. Here they started with 3k infra and then later realized it runs on toaster. Completely back to front.
Maybe they just never iterated on a local version and nobody developed an intuition for requirements. Switched straight to iterating on a nebulous cloud where you can't tell how much horsepower is behind the cloudfunctions etc.
Presumably there is a perfectly reasonably explanation and it's just not spelled out, it just seems weird based on given info
The longest is to adapt your app to a Dockerfile-based deployment, if it isn't already containerized. We have examples for most languages - for Flask, for example, the whole file is 10 lines long [0]
But to provision a new server, as these are "stateless" (per 12 Factor) servers, it's just 1) get a VPS 2) install Docker+Disco using our curl|sh install script 3) authorize github 4) deploy a "project" (what we call an app), setting the env vars.
Is this really all it takes? I’m curious because for me the value of heroku etc is not the PaaS itself but the infra: hardening the server, load balancing, redundancy, monitoring, alerting, connectivity, storage… and that’s always the hole in the selfhosted PaaS story for me. I’d love to solve it but it’s handwaved every time whenever these topics come up. Any pointers?
It's a good point. I'd say that generally, there are answers to most of these points that either get you 80% of the way there, or are functionally equivalent.
Re Load balancing for example, Disco is built on top of Docker Swarm, so you can add nodes (ie machines) to scale horizontally - `disco nodes:add root@<ip>`
For monitoring/alerting, we offer some real time cpu/memory metrics (ie `docker stats`) and integrate with external syslog services.
Do you have specific use cases in mind which current PaaS providers satisfy? Would you say that these kinds of concerns are what's holding you back from leaving Heroku or others (and are you considering leaving because of price, support, etc.)? Cheers
No, all good, this is super useful - if you're thinking about this, then it matters to us!
Our philosophy is built on the "cattle, not pets" [0] and 12-factor [1] app methodologies. To some extent, the Disco server itself should be treated as disposable.
Disco runs your applications, which are just deployments of your code (ie git pulls). There's nothing on the server itself to back up. If a server were to die, you'd spin up a new one, run the install.sh script, and redeploy your apps in about 15 minutes.
For application data, our stance is that we believe you should use a dedicated, managed database provider for prod workloads. While we can run a "good enough" postgres as noted, we treat that as a dev/staging tool. Disco handles the stateless application layer, you should entrust your critical stateful data to a service that specializes in that.
Finally, re: security, we recommend a fresh Ubuntu 24.04 LTS server, which handles its own OS security updates. Disco only exposes the necessary web and SSH ports, so the attack surface is minimal by default.
One thing however to note, is that by having a different non-prod and prod environment, it will be possible to test only the application, and not the infra.
Which means, that if they want to test what it will look like running in cloud for prod, they are going to either need a pre-prod environment or go yolo
The connection count costs also balloon on most cloud providers (any busy protocol like telemetry or log-consolidation costs can be huge.)
If your group pushes over 23TiB/Month, than most usually look for un-metered colo hosts, and or CDN services with your resource integrity Attribute tags set.
Cloud makes sense for small IT shops that can't afford a fully staffed team. The general rule is a 5:1 labor to hardware cost amortized over 3 years.
AWS was successful as IT had enormous upfront costs most people found difficult to justify, and data-centers could absorb a DDoS that would be expensive to mitigate.
Rule #17: "Only forward meaningful data at every layer of a design, as it often naturally ensures a feasible convergent behavior."
There are a number of great options, but depends on the use-cases. =3
It sounds more like poor choices. 6 staging environments sounds a bit overkill.
If you can fit them all on a 4 cpu / 32gb machine, you can easily forgo them and run the stack locally on a dev machine. IME staging environments are generally snowflakes that are hard to stand up (no automation).
> you can easily forgo them and run the stack locally
Not if you're running with external resources of specific type, or want to share the ongoing work with others. Or need to setup 6 different projects with 3 different databases at the same time. It really depends on your setup and way of working. Sometimes you can do local staging easily, sometimes it's going to be a lot of pain.
i'd be interested what the load is like on that CCX33 server - i've got a lower-spec VPS from Hetzner and even from there I'm only using about 25%-30% CPU/RAM with a moderate load
> Even with all 6 environments and other projects running, the server's resource usage remained low. The average CPU load stayed under 10%, and memory usage sat at just ~14 GB of the available 32 GB.
Once everything is installed/running, a very tldr diagram would be:
GitHub (webhook on git push) -> Docker swarm running Caddy -> Disco Daemon REST API which will ask Docker to build the image, and then does a blue-green zero-time deployment swap
But yeah, a clearer/better diagram would be great. Thanks for the push!
Every time I've worked somewhere without one, we've wanted it and wasted more developer hours than the cost of having it trying to reproduce issues while working around the differences in the environments.
Why people discover it only today? I remember making comments about it years ago.
I even shown one customer that their elaborate cluster costing £10k a month could run on a £10 vps faster and with less headache (they set it up for "big data" thinking 50GB is massive. There was no expectation of the database growing substantially beyond that).
Their response? Investors said it must run on the cloud, because they don't want to lose their money if homegrown setup goes down.
Yes. The "cloud" is sold on grounds of "efficiency" but really it's just an ideological decision to increase outsourcing and reduce the employees' bargaining power.
(Except this backfires, because a service running on a RHEL or Debian machine might go on for 5-10 years untouched without any particular issue, security aside, while anything relying on kubernetes or the hyperscaler's million little services needs to be tweaked every 6 months and re-engineered every few years or it will completely stop working.)
Having been in the industry for 20 years, I can remember we were processing high loads with... Algorithms. It wasn't a cloud cost saving initiative back then, but a necessity if you had scale, you could just not throw money at scaling. Feels like we shifted optimization from algorithms to cloud costs savings...
I have been using RackNerd for a few non-critical, hobby projects and can't fault them.
Their advertised prices are not half of Hetzner, but you can find hidden deals if you do a Google search, or via sites like https://racknerdtracker.com (no affiliation).
We do have a UI, we're just so behind on the documentation, it's not even funny ha.
If you setup a server with the curl|sh install script on the homepage, you'll get a url at the end that directs you there. And you can use the CLI too of course.
Just something to consider if you are in a professional environment before switching your entire infra: maintenance cost is expensive. I strongly suggest to throw man-days in your cost calculation.
To prevent security vulnerabilities, the team will need to write some playbooks to auto-update regularly your machine, hoping for no breaking changes. Or instead write a pipeline for immutable OS images updates. And it often mean testing on an additional canary VM first.
Scaling up the VM from a compute point of view is not that straightforward as well, and will require depending of the provider either downtime or to migrate the entire deployments to a new instance.
Scaling from a disk size point of view, you will need to play with filesystems.
And depending on the setup you are using, you might have to manage lets encrypt, authentication and authorization, secrets vaults, etc (here at least Disco manages the SSL certs for you)
If you are large enough, you will need an ops team to manage allowing your developers to write terraform and manage AWS costs already.
If you are small enough, you are not going to be truly affected by downtime. If you are just a little bigger, a single hot spare is going to be sufficient.
The place where you get dinged is heavy growth in personnel and bandwidth. You end up needing to solve CPU bound activities quicker because it hurts the whole system. You need to start thinking about sticky round robin load balancing and other fun pieces.
This is where the cloud can allow you to trade money for velocity. Eventually, though, you will need to pay up.
That said, the average SaaS can go a long way with a single server per product.
> I strongly suggest to throw man-days in your cost calculation.
Only if those man-days actually incur a marginal cost. If it's just employees you already have spending their time on things, then it's not worth factoring in because it's a cost you pay regardless.
That's the case if said employees have nothing better to do and are currently twiddling their thumbs. Usually the server maintenance hours come out of project development hours.
It's precisely why we moved from a self-hosted demo environment server to heroku - the developers that had both the skills to manage a server and enough seniority to have access accross all the different projects could bring in more by building.
If they are spending their time fiddling around disk resizing and package upgrades, it's time they are not spending helping deliver real business features.
This part can be outsourced to a PaaS company, so that the company engineers can be focused on what is the company actually making money from.
You can also enable zram to compress ram, so you can over-provision like the pros'. A lot of long-running software leaks memory that compresses pretty well.
Here is how I do it on my Hetzner bare-metal servers using Ansible: https://gist.github.com/fungiboletus/794a265cc186e79cd5eb2fe... It also works on VMs.
systemd-oomd and oomd use the kernel's PSI[2] information which makes them more efficient and responsive, while earlyoom is just polling.
earlyoom keeps getting suggested, even though we have PSI now, just because people are used to using it and recommending it from back before the kernel had cgroups v2.
[0]: https://www.freedesktop.org/software/systemd/man/latest/syst...
[1]: https://github.com/facebookincubator/oomd
[2]: https://docs.kernel.org/accounting/psi.html
I think it's off on the server variants.
> systemd-oomd periodically polls PSI statistics for the system and those cgroups to decide when to take action.
It's unclear if the docs for systemd-oomd are incorrect or misleading; I do see from the kernel.org link that the recommended usage pattern is to use the `poll` system call, which in this context would mean "not polling", if I understand correctly.
What I was trying to say is that the actual information on when there's memory pressure is more accurate for systemd-oomd / oomd because they use PSI, which the kernel itself is updating over time, and they just poll that, while earlyoom is also internally making its own estimates at a lower granularity than the kernel does.
See https://docs.kernel.org/accounting/psi.html
Another option is to limit memory per application in cgroups but that requires more explaining than I am putting in an HN comment.
Another useful thing is to never OOM kill in the first place on servers that are only doing things in memory and need not commit anything to disk. So don't do this on a disked database. This is for ephemeral nodes that should self heal. Wait 60 seconds so drac/ilo can capture crash message and then earth shattering kaboom...
For a funny side note, those options can also be used as a holy hand grenade to intentionally unsafely reboot NFS diskless farms when failing over to entirely different NFS server clusters. setting panic to 15 mins, triggering OOM panic by setting min_free to 16TB at the command line via Ansible not in sysctl.conf, swapping clusters, arp storm and reconverge.Companies that use k3/k8's they may still have bare metal nodes that are dedicated to a role such as databases, ceph storage nodes, DMZ SFTP servers, PCI hosts that were deemed out of scope for kube clusters and of course any "kittens" such as Linux nodes turned into proprietary appliances after installing some proprietary application that will blow chunks if shimmed into k8's or any other type of abstraction layer.
Maybe back in the 90s, it was okay to wait 2-3 seconds for a button click, but today we just assume the thing is dead and reboot.
In the age of microservices and cattle servers, reboot/reinstall might be cheap, but in the long run it is not. A long running server, albeit being cattle, is always a better solution because esp. with some excess RAM, the server "warms up" with all hot data cached and will be a low latency unit in your fleet, given you pay the required attention to your software development and service configuration.
Secondly, Kernel swaps out unused pages to SWAP, relieving pressure from RAM. So, SWAP is often used even if you fill 1% of your RAM. This allows for more hot data to be cached, allowing better resource utilization and performance in the long run.
So, eff it, we ball is never a good system administration strategy. Even if everything is ephemeral and can be rebooted in three seconds.
Sure, some things like Kubernetes forces "no SWAP, period" policies because it kills pods when pressure exceeds some value, but for more traditional setups, it's still valuable.
So no, my experience with swap isn't that it's invisible with SSD.
I've had good experience with linux's multi-generation LRU feature, specifically the /sys/kernel/mm/lru_gen/min_ttl_ms feature that triggers OOM-killer when the "working set of the last N ms doesn't fit in memory".
On my 8gb M1 Mac, I can have a ton of tabs open and it'll swap with minimal slowdown. On the other hand, running a 4k external display and a small (4gb) llm is at best horrible and will sometimes require a hard reset.
I've seen similar with different combinations of software/hardware.
Dirty buffer should also be tuned (limited), absolutely. Default is 20% of RAM, (with 5 second writeback and 30 second expire intervals), which is COMPLETELY insane. I limit it to 64 MB max usually, with 1 second writeback and 3 second expire intervals.
I once used an Intel Optane drive as swap for a job that needed hundreds of gigabytes of ram (in a computer that maxed out at 64 gigs). The latency was so low that even while the task was running the machine was almost perfectly usable; in fact I could almost watch videos without dropping frames at the same time.
The free memory won't go below a configurable percentage and the contiguous io algorithms of the swap code and i/o stack can still do their work.
Here’s a screenshot from atop while the task was running: <https://db48x.net/temp/Screenshot%20from%202019-11-19%2023-4...>. Note the number of page faults, the swin and swout (swap in and swap out) numbers, and the disk activity on nvme0n1. Swap in is 150k, and the number of disk reads was 116k with an average size of 6KB. Swap out was 150k with 150k disk writes of 4KB. It’s also reading from sdh at a fair clip (though not as fast as I wanted!)
<https://db48x.net/temp/Screenshot%20from%202019-12-09%2011-4...> is interesting because it actually shows 24KB average write size. But notice that swout is 47k but there were actually 57k writes. That’s because the program I was testing had to write data out to disk to be useful, and I had it going to a different partition on the same nvme disk. Notice the high queue depth; this was a very large serial write. The swap activity was still all 4KB random IO.
> Secondly, Kernel swaps out unused pages to SWAP, relieving pressure from RAM. So, SWAP is often used even if you fill 1% of your RAM. This allows for more hot data to be cached, allowing better resource utilization and performance in the long run.
Yes, and you can observe that even in your desktop at home (if you are running something like Linux).
> So, eff it, we ball is never a good system administration strategy. Even if everything is ephemeral and can be rebooted in three seconds.
I wouldn't be so quick. Google ran their servers without swap for ages. (I don't know if they still do it.) They decided that taking the slight inefficiency in memory usage, because they have to keep the 'leaked' pages around in actual RAM, is worth it to get predictability in performance.
For what it's worth, I add generous swap to all my personal machines, mostly so that the kernel can offload cold / leaked pages and keep more disk content cached in RAM. (As a secondary reason: I also like to have a generous amount of /tmp space that's backed by swap, if necessary.)
With swap files, instead of swap partitions, it's fairly easy to shrink and grow your swap space, depending on what your needs for free space on your disk are.
Yup, that part of my comment was culmination of using Linux desktops for the last two decades. :)
> I wouldn't be so quick. Google ran their servers without swap for ages.
If you're designing this from get go and planning accordingly, it doesn't fit into my definition of eff it, we ball, but let's try this and see whether we can make it work.
> With swap files, instead of swap partitions,...
I'm a graybeard. I eyeball a swap partition size while installing the OS, and just let it be. Being mindful and having good amount of RAM means that SWAP acts as a eviction area for OS first, and as an escape ramp second, in very rare cases.
--
Sent from my desktop.
It doesn't. SSDs came a long way but so did memory dies and buses, and with that the way programs work also changed as more and more they are able to fit their stacks and heaps on memory more often than not.
I have had a problem with shellcheck that for some reason eats up all my ram when I open I believe .zshrc and trust me, it's not invisible. The system crawls to a halt.
If we're talking about SATA SSDs which top at 600MBps, then yes, an aggressive application can make itself known. However, if you have a modern NVMe, esp. a 4x4 one like Samsung 9x0 series or if you're using a Mac, I bet you'll notice the problem much later, if ever. Remember the SSD trashing problem on M1 Macs? People never noticed that system used SWAP that heavily and trashed the SSD on board.
Then, if you're using a server with a couple of SAS or NVMe SSDs, you'll not notice the problem again, esp. if these are backed by RAID (even md counts).
I’m gonna guess you’re not old enough to remember computers with memory measured in MB and IDE hard disks? Swapping was absolutely brutal back then. I agree with the other poster, swap hitting an SSD is a barely noticeable in comparison.
( ) a 1% chance the system would crawl to a halt but would work
( ) a 1% change the kernel would die and nothing would work
Eg Google used to (and perhaps still does?) run their servers without swap, because they had built fault tolerance in their fleet anyway, so were happier to deal with the occasional crash than with the occasional slowdown.
For your desktop at home, you'd probably rather deal with a slowdown that gives you a chance to close a few programs, then just crashing your system. After all, if you are standing physically in front of your computer, you can always just manually hit the reset button, if the slowdown is too agonising.
So most of the time I'd prefer option 3: the OOM killer to reap a few offending programs and let me handle restarting them.
If your problem doesn't keep growing, and you just have more data that programs want to keep in memory than you have RAM, but the actual working set of what's accessed frequently still fits in RAM, then swap perfectly solves this.
Think lots of programs open in the background, or lots of open tabs in your browser, but you only ever rapidly switch between at most a handful at a time. Or you are starting a memory hungry game and you don't want to be bothered with closing all the existing memory hungry programs that idle in the background while you play.
Adding a couple of gb of swap means the image resizing is _slow_, but completes without causing issues.
> Doesn't swap just delay the fundamental issue?
The fundamental issue here is what the linux fanboys literally think what killing a working process and most of the time the process[0] is a good solution for not solving the fundamental problem of memory allocation in the Linux kernel.
Availability of swap allows you to avoid malloc failure in a rare case your processes request more memory than physically (or 'physically', heh) present in the system. But in the mind of so called linux administrators even if a one byte of the swap would be used then the system would immediately crawl to a stop and never would recover itself. Why it always should be the worst and the most idiotic scenario instead of a sane 'needed 100MB more, got it - while some shit in the memory which wasn't accessed since the boot was swapped out - did the things it needed to do and freed that 100MB' is never explained by them.
[0] imagine a dedicated machine for *SQL server - which process would have the most memory usage on that system?
Also: When those processes that haven't been active since boot (and which may never be active again) are swapped out, more system RAM can become available for disk caching to help performance of things that are actively being used.
And that's... that's actually putting RAM to good use, instead of letting it sit idle. That's good.
(As many are always quick to point out: Swap can't fix a perpetual memory leak. But I don't think I've ever seen anyone claim that it could.)
And maybe plan on getting more RAM.
(It's your system. You're allowed to tune it to fit your usage.)
What if I've got 64G of ram and 64G of swap and need the same amount of memory?
Detecting things are down is far easier than detecting things are slow.
I'd rather that oom started killing things though than a kernel panic or a slow system. Ideally the thing that is leaking, but if not the process using the most memory (and yes I know that "using" is tricky)
I still don’t use it though.
I'm not an AWS guy. I can see and touch the servers I manage, and in my experience, SWAP works, and works well.
However, from my experience, normal (eviction based) usage of SWAP doesn't impact the life of an SSD in a measurable manner. My 256GB system SSD (of my desktop system) shows 78% life remaining after 4 years of power on hours, which also served as /home for at least half of its life.
If the slowest drive on the machine is the SSD, how does caching to swap help?
This cache is evictable, but it'll be there eventually.
Linux used to don't touch unused pages in the RAM in the older days if your RAM was not under pressure, but now it swaps out pages unused for a long time. This allows more cache space in RAM.
> how does caching to swap help?
I think I failed to convey what I tried to say. Let me retry:
Kernel doesn't cache to SSD. It swaps out unused (not accessed) but unevictable pages to SWAP, assuming that these pages will stay stale for a very long time, allowing more RAM to be used as cache.
When I look to my desktop system, in 12 days, Kernel moved 2592MB of my RAM to SWAP despite having ~20GB of free space. ~15GB of this free space is used as disk cache.
So, to have 2.5GB more disk cache, Kernel moved 2592 MB of non-accessed pages to SWAP.
I DON’T WANT THE KERNEL PRIORITIZING CACHE OVER NRU PAGES.
The easiest way to do this is to disable swap.
Then tell the Kernel about it. Don't remove a feature which might benefit other things running on your system.
When you call malloc(), it requests a big chunk of memory from the OS, in units of pages. It then uses an allocator to divide it up into smaller, variable length chunks to form each malloc() request.
You may have heard of “heap” memory vs “stack” memory. The stack of course is the execution/call stack, and heap is called that because the “heap allocator” is the algorithm originally used for keeping track of unused chunks of these pages.
(This is beginner CS stuff so sorry if it came off as patronizing—I assume you’re either not a coder or self-taught, which is fine.)
How about this server:
Uptime of 2,105 days - nearly 6 years.How long does the server have to run to reach 100% of ram?
25MB swap use seems normal for a server which doesn't juggle much tasks, but works on one.
This is from another system I have close:
2MB of SWAP used, 1423 MB RAM used, 29GB cache, 1042 MB Free. Total RAM 32 GB.ls -lrt, ls -lSh and ls -lShr are also very common in my daily use, depending on what I'm doing.
If I were to increase this to 8G of ram instead of 2G, but for arguments sake had to have no swap as the tradeoff, would that be better or worse. Swap fans say worse.
As I noted somewhere, my other system has 2,5GB of SWAP allocated over 13 days. That system is a desktop system and juggles tons of things everyday.
I have another server with tons of RAM, and the Kernel decided not to evict anything to SWAP (yet).
> If I were to increase this to 8G of ram instead of 2G, but for arguments sake had to have no swap as the tradeoff, would that be better or worse. Swap fans say worse.
I'm not a SWAP fan, but I support its use. On the other hand I won't say it'd be worse, but it'd be overkill for that server. Maybe I can try 4, but that doesn't seem to be necessary if these numbers are stable over time.
The easiest way affecting everything running on the system might not be the best or even the correct way to do things.
There's always more than one way to solve a problem.
Reading the Full Manual (TM) is important.
This is not about belief, but lived experience. Setting up swap to me is a choice between a unresponsive system (with swap) or a responsive system with a few oom kills or downed system.
I mean, I manage some servers, and this is my experience.
> Setting up swap to me is a choice between a unresponsive system (with swap) or a responsive system with a few oom kills or downed system.
Sorry, but are you sure that you budgeted your system requirements correctly? A Linux system shall neither fill SWAP nor trigger OOM regularly.
With a good amount of swap, you don't have to worry about closing programs. As long as your 'working set' stays smaller than your RAM, your computer stays fast and responsive, regardless of what's open and idling in the background.
Old habits die hard, but I'm not complaining about this one. :)
Without swap oom killer runs and things become responsive.
Swap is good to have. The value is limited but real.
Also not having swap doesn't prevent thrashing, it just means that as memory gets completely full you start dropping and re-reading executable code over and over. The solution is the same in both cases, kill programs before performance falls off a cliff. But swap gives you more room before you reach the cliff.
Swap helps you use ram more efficiently, as you put the hot stuff in swap and let the rest fester on disk.
Sure if you overwhelm it, then you're gonna have a bad day, but thats the same without swap.
Seriously, swap is good, don't believe the noise.
That is my interpretation of what people are saying upthread, at least. To which posters such as yourself are saying “you still need swap.” Why?
Right, you seem to be not understanding what I'm getting at.
Memory exhaustion is bad, regardless of swap or not.
Swap gets you a better performing machine because you can swap out shit to disk and use that ram for vfs cache.
the whole "low latency" and "I want my VM to die quicker" is tacitly saying that you haven't right sized your instances, your programme is shit, and you don't have decent monitoring.
Like if you're hovering on 90% ram used, then your machine is too small, unless you have decent bounds/cgroups to enforce memory limits.
It's a bit wasteful to provision your computers so that all the cold data lives in expensive RAM.
But that's a job applications are already doing. They put data that's being actively worked on in RAM they leave all the rest in storage. Why would you need swap once you can already fit the entire working set in RAM?
Swap ram by itself would be stupid but no one doing this isn’t also turning on compression.
I'm not sure what you mean here? Swapping out infrequently accesses pages to disk to make space for more disk cache makes sense with our without compression.
In that case, and if you are only running these applications, the need for swap is much less.
Almost by definition, that leaked memory is never accessed again, so it's very cold. But the applications don't put this on disk by themselves. (If the app's developers knew about which specific bit is leaking, they'd rather fix the leak then write it to disk.)
If you size your RAM and swap right, you get no service degradation, but still get away with using less RAM.
But when I was at Google (about a decade ago), they followed exactly the philosophy you were outlining and disabled swap.
Many won't enable swap. For some swap wouldn't help anyways, but others it could help soak up spikes. The latter in some cases will upgrade to a larger instance without even evaluating if swap could help, generating AWS more money.
Either way it's far-fetched to derive intention from the fact.
Nowadays when a program hits swap it's not going to fallback to a different memory usage profile that prioritises disk access. It's going to use swap as if it were actual ram, so you get to see the program choking the entire system.
If your GC is a moving collector, then absolutely this is something to watch out for.
There are, however, a number of runtimes that will leave memory in place. They are effectively just calling `malloc` for the objects and `free` when the GC algorithm detects an object is dead.
Go, the CLR, Ruby, Python, Swift, and I think node(?) all fit in this category. The JVM has a moving collector.
Tracing garbage collectors solve a single problem really really well - managing a complex, possibly cyclical reference graph, which is in fact inherent to some problems where GC is thus irreplaceable - and are just about terrible wrt. any other system-level or performance-related factor of evaluation.
There's a lot of "it depends" here.
For example, an RC garbage collector (Like swift and python?) doesn't ever trace through the graph.
The reason I brought up moving collectors is by their nature, they take up a lot more heap space, at least 2x what they need. The advantage of the non-moving collectors is they are much more prompt at returning memory to the OS. The JVM in particular has issues here because it has pretty chunky objects.
If the implementer cares about memory use it won't. There are ways to compact objects that are a lot less memory-intensive than copying the whole graph from A to B and then deleting A.
Even not so modern ones: have you heard of generational garbage collection?
But even in eg Python they introduced 'immortal objects' which the GC knows not to bother with.
Also since the refcounts are inline, adding a reference to a cold object will update that object. IIRC Swift has the latter issue as well (unless the heap object’s RC was moved to the side table).
It's that "touching" of all the pages controlled by the GC that ultimately wrecks swap performance. But also the fact that moving collector like to hold onto memory as downsizing is pretty hard to do efficiently.
Non-moving collectors are generally ultimately using C allocators which are fairly good at avoiding fragmentation. Not perfect and not as fast as a moving collector, but also fast enough for most use cases.
Java's G1 collector would be the worst example of this. It's constantly moving blocks of memory all over the place.
The memory that's now not in use, but still held onto, can be swapped out.
I regularly use it on my Snapdragon 870 tablet (not exactly a top of the line CPU) to prevent OOM crashes (it's running an ancient kernel and the Android OOM killer basically crashes the whole thing) when running a load of tabs in Brave and a Linux environment (through Tmux) at the same time.
ZRAM won't save you if you do actually need to store and actively use more than the physical memory but if 60% of your physical memory is not actively used (think background tabs or servers that are running but not taking requests) it absolutely does wonders!
On most (web) app servers I happily leave it enabled to handle temporary spikes, memory leaks or applications that load a whole bunch of resources that they never ever use.
I'm also running it on my Kubernetes cluster. It allows me to set reasonable strict memory limits while still having the certainty that Pods can handle (short) spikes above my limit.
Would love to be corrected if I'm wrong
Example on my personal VPS
But also false. Swap is there so anonymous pages can be evicted. Not as a “slow overflow for RAM”, as a lot of people still believe.
By disabling swap you can actually *increase* thrashing, because the kernel is more limited in what it can do with the virtual memory.
If swapping to SSD is 'extremely slow', what's your term for swapping to HDD?
My experience is the exact opposite. If anything 2-3 second button clicks are more common than ever today since everything has to make a roundtrip to a server somewhere whereas in the 90s 2-3s button click meant your computer was about to BSOD.
Edit: Apple recently brought "2-3s to open tab" technology to Safari[1].
[1] https://old.reddit.com/r/MacOS/comments/1nm534e/sluggish_saf...
Yup, this is a thing. It happens because file-backed program text and read-only data eventually get evicted from RAM (to make room for process memory) so every access to code and/or data beyond the current 4K page can potentially involve a swap-in from disk. It would be nice if we had ways of setting up the system so that pages of code or data that are truly critical for real-time responsiveness (including parts of the UI) could not get evicted from RAM at all (except perhaps to make room for the OOM reaper itself to do its job) - but this is quite hard to do in practice.
As a small example from a default Ubuntu installation, "unattended-upgrades" is holding 22MB of RSS, and will not impact system performance at all if it spends next week swapped out. Bigger examples can be found in monolithic services where you don't use some of the features but still have to wire them into RAM. You can page those inactive sections of the individual process into swap, and never notice.
Gives you some time to upgrade, or tune services before it goes ka-boom.
If your memory usage spikes suddenly, a nominal amount of swap isn't stopping anything from getting killed; you're at best buying yourself a few seconds, so unless you spend your time just staring at the server, it'll be dead anyways.
Therefore it is better to always tune "vm.swappiness" to 1 in /etc/sysctl.conf
You can also configure your web server / TCP stack buffers / file limits so they never allocate memory over the physical ram available. (eg. in nginx you can setup worker/connection limits and buffer sizes.)
Is earlyoom a better solution than that to prevent an erratic process from making an instance unresposnsive?
- hardware, networks, monitoring, provisioning, server room locations in existing buildings, how to prepare server rooms
- and so on up to hiring and firing sysadmins, salary negotiations[2], vendor negotiations and the first book even had a whole chapter dedicated to "Being happy"
[1] There is a third author as well now, but those two were the ones that are on the cover of my book from 2005 and that I can remember
[2] Has mostly worked well after I more or less left sysadmin behind as well
And of course the overhead is zero when you don't page-out to swap.
For an algorithm using the whole memory, that’s a terrible idea.
I understand all of those words, but none of the meaning. Why would I reserve RAM in order to put fast swap on it?
This has a number of benefits: in practice more “active” space is freed up as unused pages are compressed and often compressible. Often times that can be freed application memory that is reserved within application space but in the free space of the allocator, especially if that allocator zeroes it those pages in the background, but even active application memory (eg if you have a browser a lot of the memory is probably duplicated many times across processes). So for a usually invisible cost you free up more system RAM. Additionally, the overhead of the swap is typically not much more than a memcpy even compressed which means that you get dedup and if you compressed erroneously (data still needed) paging it back in is relatively cheap.
It also plays really well with disk swap since the least frequently used pages of that compressed swap can be flushed to disk leaving more space in the compressed RAM region for additional pages. And since you’re flushing retrieving compressed pages from disk you’re reducing writes on an SSD (longevity) and reducing read/write volume (less overhead than naiive direct swap to disk).
Basically if you think of it as tiered memory, you’ve got registers, l1 cache, l2 cache, l3 cache, normal RAM, compressed swap RAM, disk swap - it’s an extra interim tier that makes the system more efficient.
I do: https://postimg.cc/G8Gcp3zb (casualmeasurement.png)
To clarify OP's represention of the tool, it compresses swap space not resident ram. Outside of niche use-cases, compressing swap has overall little utility.
It has the benefit of absorbing memory leaks (which for whatever reason compress really well) and compressing stale memory pages.
Under actual memory pressure performance will degrade. But in many circumstances where your powerful CPU is not fully utilized you can 2x or even 3x your effective RAM (you can opt for zstd compression). zram also enables you to make the trade-off of picking a more powerful CPU for the express purpose of multiplying your RAM if the workload is compatible with the idea.
PS: On laptops/workstations, zram will not interfere with an SSD swap partition if you need it for hibernation. Though it will almost never be used for anything else if you configure your zram to be 2x your system memory.
That reads like what they said? You reserve part of the RAM as a swap device, and memory is swapped from resident RAM to the swap ramdisk, as long as there’s space on there. And AFAIK linux will not move pages between swap devices because it doesn’t understand them beyond priority.
Zswap actually seems strictly better in many cases (especially interactive computers / dev machines) as it can more flexibly grow / shrink, and can move pages between the compressed RAM cache and the disk swap.
The case of swap thrashing sounds like a misbehaving program, which can maybe be tamed by oomd.
System responsiveness though needs a complete resource control regime in place, that preserves minimum resources for certain critical processes. This is done with cgroupsv2. By establishing minimum resources, the kernel will limit resources for other processes. Sure, they will suffer. That’s the idea.
Partly it's a money thing (they want to sell you RAM), partly it's so that the shared disk isn't getting thrashed by multiple VPS
And that was like... two years ago? 1GB of RAM and actually ~700MB usable before I found the proper magik incantations to really disable kdump.
Also have used 1GB machines for literally years.
Strongly suggest you shouldn't strongly suggest.
You do understand what's being discussed... right?
Or you have a very peculiar understanding what 'VPS' means.
As I said earlier
https://news.ycombinator.com/threads?id=awesome_dude#4566311...
We have a policy that our customers are responsible for all their business-related input, but we make the decisions about the technical implementation. Every technical decision that the customer wants to make basically costs extra.
In this case we built a rather simple multi-tenancy B2B app using Laravel, with one database per tenant. They planned to start with a single customer/tenant, scaling up to maybe a few dozen within the next years, with less than a hundred concurrent users over the first five years. There were some processes with a little load, but they were few, running less that a minute each and already built up to run asynchronous.
We planned a single Hetzner instance and to scale up as soon as we would see it reaching its limits. So less than 100 €/month.
The customer told us that they have a cooperation with their local hosting provider (with "special conditions!") and that they wanted to use them instead.
My colleague did all the setup, because he is more experienced in that, but instead of our usual five-minute-setup in Forge (one of the advantages of the Laravel ecosystem), it took several weeks with the hosting provider, where my colleague had to invest almost full time just for the deployment. The hosting provider "consulted" out customer to invest in a more complex setup with a load balancer in front, to be able to scale right away. They also took very long for each step, like providing IP addresses or to handle the SSL certificates.
We are very proud of our very fast development process and having to work with that hosting provider cost us about one third of our first development phase for the initial product.
It's been around two years since then. While the software still works as intended, the customer could not grow as expected. They are still running with only one single tenant (basically themselves) and the system barely had to handle more than two concurrent users. The customer recently accidentally mentioned that they pay almost 1000€/month for the hosting alone. But it scales!
They don't want our hosting solutions but insist on using their own hosting partners
The result are similar:
- its at least five times more expensive on pure hosting costs
- we lose a considerable amount of time dealing with the hosting partner (which we bill to the customer)
- it's always a security nightmare, either because they put so much "safety protections" in place that it's unusable (think about the customer wanting an Internet-facing website, but the servers are private...) or because they don't put any safety settings in place so the servers are regularly taken down through awfully simple exploits (think about SSH root access with "passw0rd" as password...)
- customer keep complaining about performances to us, but what can you do when the servers are sharing a 100Mbps connection, or the filesystem is on an NFS with <20Mbps bandwidth
Forge seems like a great integrated solution (I subscribe to their newsletter and their product updates seem quite frequent and useful). What's been your experience with them? Any particular things you like, or dislike about them?
I'm also curious when you talk about scaling up Forge - is that something you've done, and is that generally easy to do?
Thanks a lot!
Local hosting can also be comparing apple with oranges. A local data center that provide a physical machine is very different from a cloud provider, especially if that cloud is located in a different continent and under different jurisdictions. Given that they were providing SSL certificates, was this a local php webshop? Data centers should be a bit more proficient with things like IP addresses and setting up any cast, but less so in providing help with php or certificates, and if they sell that it may not be their area of expertise.
What prevented them from scaling to more tenants?
It's a shame they don't just license all their software stack at a reasonable price with a similar model like Sidekiq and let you sort out actually decent hardware. It's insane to consider Heroku if anything has gotten more expensive and worse compared to a decade ago yet in comparison similar priced server hardware has gotten WAY better of a decade. $50 for a dyno with 1 GB of ram in 2025 is robbery. It's even worse considering running a standard rails app hasn't changed dramatically from a resources perspective and if anything has become more efficient. It's comical to consider how many developers are shipping apps on Heroku for hundreds of dollars a month on machines with worse performance/resources than the macbook they are developing it on.
It's the standard playback that damn near everything in society is going for though just jacking prices and targeting the wealthiest least price sensitive percentiles instead of making good products at fair prices for the masses.
Heroku's pricing has _remained the same_ for at least seven years, while hardware has improved exponentially. So when you look at their pricing and see a scam, what you're actually doing is comparing a 2025 anchor to a mid-2010s price that exists to retain revenue. At the big cloud vendors, they differentiate customers by adding obstacles to unlocking new hardware performance in the form of reservations and updated SKUs. There's deliberate customer action that needs to take place. Heroku doesn't appear to have much competition, so they keep their prices locked and we get to read an article like this whenever a new engineer discovers just how capable modern hardware is.
Heroku has obviously stagnated now but their stack is _very cool_ for if you have a fairly simple system but still want all the nice parts of a mode developed ops system. It almost lets you get away with not having an ops team for quite a while. I don't know any other provider that is low-effort "decent" ops (Fly seems to directionally want to be new Heroku but is still missing a _lot_ in my book, though it also has a lot)
There are some folks with good offerings (Fly, Railway, etc), but the feature set of Heroku is deeper, and more important for production apps, than most people realize. They aren’t a good place for hobbyists anymore though. I agree with that.
To be fair, AWS quite proudly talk about all the times they've lowered prices on existing services, or have introduced new generations that are cheaper (e.g. their Graviton EC2 instances).
We built and open sourced https://canine.sh for exactly that reason. There’s no reason PaaS providers should be charging such a giant markup over already marked up cloud providers.
You can also self host almost any open source service without any fuss, and perform internal networking with telepresence. (For example, if you want to run an internal metabase that is not available on public internet, you can just run `telepresence connect`, and then visit the private instance at metabase.svc.cluster.local).
Canine tries to leverage all the best practices and pre-existing tools that are already out there.
But agreed, business critical databases probably shouldn't belong on Kubernetes.
Regardless, you're going to have a much easier time developing your app if your datastore access latency is submillisecond rather than tens of milliseconds.
So that extra trouble might be worth it...
Also, "small scale" means different things to different people. Given the full topic at hand, I would call it "nano scale". Depending upon your specific schema, you can serve tens of thousands of queries per second with a single server on modern hardware, which is way more than enough for the vast majority of workloads.
AWS isn't much better honestly.. $50/month gets you an m7a.medium which is 1 vCPU (not core) and 4GB of RAM. Yes that's more memory but any wonder why AWS is making money hand-over-fist..
Where they get you is all the ancillary shit, you buy some database/backup/storage/managed service/whatever, and it is priced in dollars per boogaloo, you also have to pay water tax on top, and of course if you use more than the provisioned amount of hafnias the excess ones cost 10x as much.
Most customers have no idea how little compute they are actually buying with those services.
If you reserve that instance you can get it for 40% cheaper, or get 4 cores instead.
Yes it's more expensive than OVH but you also get everything AWS to offer.
To compare to Heroku's standard dynos (which are shared hosting) you want the t3a family which is also shared, and much cheaper.
Netlify sets the same prices.
Just throw it into a cloud bucket from CI and be done with it.
That said, they are emerging. I'm actually working on a drop-in Vercel competitor at https://www.sherpa.sh. We're 70% lower cost by running on EU based CDN and dedicated servers (Hetzner, etc). But we had to build the relationships to solve all the above challenges first.
Every other time I login to the admin site I get a Heroku error.
It's insane how much a restaurant charges for a decent steak, I can do it much cheaper myself!
...!
By the time the product is a success and reaches a scale where it becomes cost prohibitive, they have enough resources to expand or migrate away anyway.
I suppose for solo devs it might be cheaper to setup a box for fun, but even then, I would argue that not everyone enjoys doing devops and prefers spending their time elsewhere.
Honestly, reading these threads it sounds to me like a lot of people are still launching new projects on Heroku. I wouldn’t have guessed that was true before reading this.
No one gets hurt if someone else chooses to waste their money on Heroku so why are people complaining? Of course it applies in cases where there aren't a lot of competitors but there are literally hundreds of different of different options for deploying applications and at least a dozen of them are just as reliable and cheaper than Heroku.
Or they didn't check. A business still existing is pretty weak evidence that the pricing is reasonable.
Really? I mean oil changes are pretty cheap. You can get an oil change at walmart for like 40 bucks.
Hosting staging envs in pricey cloud envs seems crazy to me but I understand why you would want to because modern clouds can have a lot of moving parts.
Cloud isn't worth it until suddenly it is because you can't deploy your own servers fast enough, and then it's worth it until it exceeds the price of a solid infrastructure team and hardware. There's a curve to how much you're saving by throwing everything in the cloud.
It's fun the first time, but becomes an annoying faff when it has to be repeated constantly.
In Heroku, Vercel and similar you git push and you're running. On a linux server you set up the OS, the server authentication, the application itself, the systemctl jobs, the reverse proxy, the code deployment, the ssl key management, the monitoring etc etc.
I still do prefer a linux server due to the flexibility, but the UX could be a lot better.
I have to ask - do scripts not work for you?
When I had to do this back in 2005 it was automated with 3 main steps:
1. A preseed (IIRC) debian installation disc (all the packages I needed where installed at install time), and
2. Which included a first-boot bash script that retrieved pre-compiled binaries from our internal ftp site, and
3. A final script that applied changes to the default config files and ran a small test to ensure everything started.
Zero human interaction after powering a machine on with the disc in the drive.
These days I would do it even better (system-d configs, Nix perhaps, text files (such as systemd units) can be retrieved automagically after boot, etc).
And the overlap between what Nix does and what the 'cloud' does for you is only partial. (Eg it can still make sense to use Nix in the cloud.)
Certainly true, but there are a whole lot of tools to automate those operations so that you aren't doing them constantly.
Ansible basically automates the workflow of: log in to X, do step X (if Y is not present). It has broad support for distros and OSes. It's mostly imperative and can be used like a glorified task runner.
Salt let's you mostly declaratively describe the state of a system. It comes with a agent/central host system for distributing this configuration from the central host to the minions (push).
Puppet is also declarative and also comes with an agent/central host system but uses a pull based approach.
Specialized/ exotic options are also available, like mgmt or NixOS.
Actually I am looking for tools to automate DevOps and security for self-hosting
Ansible can also do that, on top of literally anything else you could want - network configuration, infrastructure automation, deployment pipelines, migrations, anything. As always, that flexibility can be a blessing or a curse, but I think Ansible manages it well because it's so KISS.
RedHat's commercial Ansible Automation Platform gives you more power for when you need it, but you don't need it starting out.
It is in general the simplest of these systems to get started with and you should be able to incrementally adopt it. There is also a plethora of free online resources available for it.
Ansible-Lockdown is another excellent example of how Ansible can be used to harden servers via automation.
cloud is easy until is not, for 90% of us maybe we dont need a multi region with hot and cold storage
for those that need it, its neccesary
This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
It offloads things like - Power Usage - Colo Costs - Networking (a big one) - Storage (SSD wear / HDD pools) - etc
It is a long list but what doesnt allow you do it make trade offs like spending way less but accept downtime if your switch dies etc etc.
For a staging env these are things you might want to do.
Is it mostly developer insecurity, or mostly tech leadership insecurity?
I think this is partly responsible for the increased popularity of sqlite as a backend. It's super simple and lightstream for recovery isn't that complicated.
Most apps don't need 5 9s, but they do care about losing data. Eliminate the possibility of losing data, without paying tons of $ to also eliminate potential outages, and you'll get a lot of customers.
It’s a lot cheaper than me learning to bake as well as he does—not to mention dedicating the time every day to get my daily bread—and I’ll never need bread on the kind of scale that would make it worth my time to do so.
But the cloud is different. None of the financial scale benefits are passed on to you. You save serious money running it in-house. The arguments around scale have no validity for the vast, vast majority of use cases.
Vercel isn't selling bread: they're selling a fancy steak dinner, and yes, you can make steak at home for much less, and if you eat fancy steak dinners at fancy restaurants every night you're going to go broke.
So the key is to understand whether your vendors are selling you bread, or a fancy steak dinner, and to not make the mistake of getting the two confused.
I wonder, though—at the risk of overextending the metaphor—what if I don’t have a kitchen, but I need the lunch meeting to be fed? Wouldn’t (relatively expensive) catering routinely make sense? And isn’t the difference between having steak catered and having sandwiches catered relatively small compared to the alternative of building out a kitchen?
What if my business is not meaningfully technical: I’ll set up applications to support our primary function, and they might even be essential to the meat of our work. But essential in the same way water and power are: we only notice it when it’s screwed up. Day-to-day, our operational competency is in dispatching vehicles or making sandwiches or something. If we hired somebody with the expertise to maintain things, they’d sit idle—or need a retainer commensurate with what the Vercels and Herokus of the world are charging. We only need to think about the IT stuff when it breaks—and maybe to the extent that, when we expect a spike, we can click one button to have twice as much “application.”
In that case, isn’t it conceivable that it could be worth the premium to buy our way out of managing some portion of the lower levels of the stack?
In practice, there are two situations where cloud makes sense:
1. You infrequently need to handle traffic that unpredictably bursts to a large multiple of your baseline. (Consider: you can over provision your baseline infrastructure by an order of magnitude before you reach cloud costs) 2. Your organization is dysfunctional in a way that makes provisioning resources extremely difficult but cloud can provide an end run around that dysfunction.
Note that both situations are quite rare. most industries that handle that sort of large burst are very predictable: event management know when a client will be large and provision ticket sales infra accordingly, e-commerce knows when the big sale days will be, and so on. In the second case, whatever organizational dysfunction caused the cloud to be appealing will likely wrap itself around the cloud initiative as well.
Water is cheap, yes. Salt isn't all that cheap, but you only need a little bit.
> [...] and I’ll never need bread on the kind of scale that would make it worth my time to do so.
If you need bread by hand, it's a very small scale affair. Your physique and time couldn't afford you large scale bread making. You'd a big special mixer and a big special oven etc for that. And you'd probably want a temperature and moisture controlled room just for letting your dough rise.
https://postmates.com/store/restaurant-depot-4538-s-sheridan...
I blush to admit that I do from time to time pay $21 for a single sourdough loaf. It’s exquisite, it’s vastly superior to anything I could make myself (or anything I’ve found others doing). So I’m happy to pay the extreme premium to keep the guy in business and maintain my reliable access to it.
It weighs a couple of pounds, though I’m not clear how the water weight factors in to the final weight of a loaf. And I’m sure that flour is fancier than this one. I take your point—I don’t belong in the bread industry :)
(Similarly to how you pay Amazon or Google etc not just for the raw cloud resources, but for the system they provide.)
I grew up in Germany, but now live in Singapore. What's sold as 'good' sourdough bread here would make you fail your baker's training in Germany: huge holes in the dough and other defects. How am I supposed to spread butter over this? And Mischbrot, a mixture of rye and wheat, is almost impossible to find.
So we make our own. The goal is mostly to replicate the everyday bread you can buy in Germany for cheap, not to hit any artisanal highs. (Though they are massively better IMHO than anything sold as artisanal here.)
Interestingly, the German breads we are talking about are mostly factory made. Factory bread can be good, if that's what customers demand.
See https://en.wikipedia.org/wiki/Mischbrot
Going on a slight tangent: with tropical heat and humidity, non-sourdough bread goes stale and moldy almost immediately. Sourdough bread can last for several days or even a week without going moldy in a paper bag on the kitchen counter outside the fridge, depending on how sour you go. If you are willing to toast your bread, going stale during that time isn't much of an issue either.
(Going dry is not much of an issue with any bread here--- sourdough or not, because it's so humid.)
Of course, the difference between sourdough and anything else is astonishing, I just can't comprehend someone charging $21 for it!
Wait, what? Salt is literally one of the cheapest of all materials per kilogram that exists in all contexts, including non-food contexts. The cost is almost purely transportation from the point of production. High quality salt is well under a dollar a pound. I am currently using salt that I bought 500g for 0.29 euro. You can get similar in the US (slightly more expensive).
This was a meme among chemical engineers. Some people complain in reviews on Amazon that the salt they buy is cut with other chemicals that make it less salty. The reality is that there is literally nothing you could cut it with that is cheaper than salt.
But sure, it's cheap otherwise. Point granted.
One way or another, salt is not a major driver of cost in bread, because there's relatively little salt in bread. (If there's 1kg of flour, you might have 20g of salt.)
also skills, some people just bake better than others
It's actually not too bad, if look at the capital cost of a bread factory amortised over each loaf of bread.
The equipment is comparatively more expensive for a home baker who only bakes perhaps two loafs a week.
Configuring a web server is a low-difficulty task that should be available for any good software developer with 3 days to study for it. It's absurd for a developer to need to configure a web server, but insist on paying a large rent and cede control to some 3rd party instead of just doing it.
I bet you could figure out `apt install nginx` and a basic config pretty quickly, definitely faster than a web dev could learn game programming. “What do you mean, I have to finish each loop in 16 msec?”
Today the smallest, and even large, aws machines are a joke, comparable to a mobile phone from 15 years ago to a terrible laptop today, and take about three to six months to in rent as buying the hardware outright.
If you're on the cloud without getting 75% discount you will save money and headcount by doing everything on prem.
I'd still like a staging + prod, but keeping the dev environments on a separate beefy server seems smart.
You get X resources in the cloud and know that a certain request/load profile will run against it. You have to configure things to handle that load, and are scored against other people.
Things like Lambda do fit in this model, but they are too inefficient to model every workload.
Amazon lacks vision.
As an example: my Macbook Pro from 2015 had 16 GiB RAM, and that's what my MacBook Air from 2025 also has.
Oh, and the new machine has unified RAM. The old machine had a bit of extra RAM in the GPU that I'm not counting here.
As far as I can tell, the new RAM is a lot faster. That counts for something. And presumably also uses less power.
Simplicity is uncomfortable to a lot of people when they're used to doing things the hard way.
* The big caveat: If you don't incur the exact same devops costs that would have happened with a linux instance.
Many tools (containers in particular) have cropped up that have made things like quick, redundant deployment pretty straightforward and cheap.
As cloud marches on it continues to seem like a grift.
Breaking into a home is relatively easy.
And unless you live in the US and is willing to actually shot someone (with all the paperwork that entails, as well as physical and legal risks), the fact is that you can't actually stop a burglary.
The cloud costs includes everything.
We used to be on Heroku and the cost wasn't just the high monthly bill - it was asking "is this little utility app I just wrote really worth paying $15/month to host?" before working on it.
This year we moved to a self-hosted setup on Coolify and have about 300 services running on a single server for $300/month on Hetzner. For the most part, it's been great and let us ship a lot more code!
My biggest realization is that for an organization like us, we really only need 99% uptime on most of our services (not 99.99%). Most developer tools are around helping you reach 99.99% uptime. When you realize you only need 99%, the world opens up.
Disco looks really cool and I'm excited to check it out!
We know of two similar cases: a bootcamp/dev school in Puerto Rico that lets its students deploy all of their final projects to a single VPS, and a Raspberry Pi that we've set up at the Recurse Center [0] which is used to host (double checking now) ~75 web projects. On a single Pi!
[0] https://www.recurse.com/
(Just remember to take regular backups now, so that when this 5 year deal expires you don’t get into the same situation again :-)
Lots of conversation & discussion about self-hosting / cloud exits these days (pros, cons, etc.) Happy to engage :-)
Cheers!
The load average in htop is actually per CPU core. So if you have 8 CPU cores like in your screenshot, a load average of 0.1 is actually 1.25% (10% / 8) of total CPU capacity - even better :).
Cool blog! I've been having so much success with this type of pattern!
I'd say the main differences is that we 1) we offer a more streamlined CLI and UI rather than offering extensive app/installation options 2) have an api-key based system that lets team members collaborate without having to manage ssh access/keys.
Generally speaking, I'd say our approach and tooling/UX tends to be more functional/pragmatic (like Heroku) than one with every possible option.
Would be great to have a comparison on the main page of Disco
I am building https://github.com/openrundev/openrun/. Main difference is that OpenRun has a declarative interface, no need for manual CLI commands or UI operations to manage apps. Another difference is that OpenRun is implemented as a proxy, it does not depend on Traefik/Nginx etc. This allows OpenRun to implement features like scaling down to zero, RBAC access control for app access, audit logs etc.
Downside with OpenRun is that is does not plan to support deploying pre-packaged apps, no Docker compose support. Streamlit/Gradio/FastHTML/Shiny/NiceGUI apps for teams are the target use case. Coolify has the best support and catalog of pre-packaged apps.
https://news.ycombinator.com/item?id=44292103
https://news.ycombinator.com/item?id=44873057
> disco provides a "good enough" Postgres addon.
> This addon is a great way to quickly setup a database when Postgres is not mission critical to your system. If you need any non-basic features, like replication, automatic failover, monitoring, automatic backups and restore, etc. you should consider using a managed Postgres provider, such as Neon or Supabase.
How come automatic backups is considered an “advanced” feature?
Also I can’t think of a single application since 2012 that I have worked on that did not have a secondary/follower instance deployed. Also suggesting Neon and friends is fine, but I wonder what is your average latency, Hetzner does not have direct connection to the DCs these databases are hosted.
I fully agree with you though, it's table stakes (unintended pun!) for any prod deployment, just as read-only followers, etc. Our biggest, most important point, is that folks should be using real dbs hosted by people who know what they're doing. The risk/reward ratio is out of whack in terms of doing it yourself.
And finally, re Hetzner and cross-DC latency, that's unfortunately a very good issue that we had to plan for in another case - specifically, a customer using Supabase (which is AWS-based). The solution was to simply use an EC2 machine in the same region. Thankfully, some db providers end up being explicit about which AWS region they run in - and obviously, using AWS RDS is also an option! It's definitely a consideration.
> Critically, all staging environments would share a single "good enough" Postgres instance directly on the server, eliminating the need for expensive managed database add-ons that, on Heroku, often cost more than the dynos themselves.
Heroku also has cheaper managed database add-ons, why not use something like that for staging? The move to self hosting might still make sense, my point is that perhaps the original staging costs of $500/mo could have been lower from the start.
The lead who wrote it had never even profiled code before, after some changes we cut it down to ~$0.01/per, but that's still insane.
[1] https://www.hetzner.com/dedicated-rootserver/ax162-s/configu...
For example, the "Bridging the Gap: Why Not Just Docker Compose?" section is a 1:1 copy of the points in the "Powerful simplicity" on the landing page - https://disco.cloud/
And this blog post is the (only) case study that they showcase on their main page.
- ...
I'm kidding :-)
Our library is open source, and we're very happy and proud that Idealist is using us to save a bit of cash. Is it marketing if you're proud of your work? :-) Cheers
Marketing should be marketing and clearly so. Tech blogs are about sharing information with the community (Netflix Tech blog is a good example) NOT selling something. Marketing masquerading as a tech blog is offputting to a lot of people. People don't like being fooled with embedded advertising and putting ad copy into such pieces is at best annoying.
https://netflixtechblog.com/
Mine isn't, unless you make the meaning of that term so broad that it essentially lost any meaningful meaning. (Intentionally meta.)
Netflix is giving away free water bottles (I hate them, but I use their fast.com super often to test the speeds), another is pretending to be a blog post, but actually being an ad (if that was the case here). You just feel lied to. You cannot take anything seriously you read there, as it will probably be super biased and you cannot get your time back now.
I'm complaining about thinly veiled ad copy wearing the mask of shared technical notes. This is seen as a bad faith effort by the publisher of such notes and a dirty trick played on the reader. Advertising should announce itself for what it is.
I'm very clearly making a distinction, I like A, I don't like B.
You're taking that, saying I must actually hate both A and B, and by the way C through Z because nobody is 111% pure of heart and everybody must have at least some motivation for doing something and nobody is entirely altruistic.... which is just this crazy extreme that it's clear I don't believe at all.
I like the incentive structure that leads Netflix to produce objectively high quality articles sharing with the community in a way that really seems to be entirely untainted by the motivation.
Ad copy in tech notes does seem to taint the motivation and quality of them, it can be innocent but it doesn't seem like it and is generally irritating to a lot of people.
Dislike of a certain kind of advertising doesn't mean I'm sitting around miserable because nobody is truly altruistic as you suggest, and that the issue. My lines of thinking aren't taken to a silly extreme. A lot of disagreements these days are people reinterpreting their opposition as exclusively extremist and that's a problem.
You say you like A and don't like B. You don't like B because it has X in it. But A also has X in it. So why do you like A but not B? It's not logically consistent. We disagree on how much X is in A. You want X to be clearly marked with red tape. It's not clear how reasonable and feasible that is or isn't. I'm saying if you're looking for X, you're going to find trace amounts of it everywhere once you start looking for it. X isn't some previously unheard of chemical that's gonna give you cancer or leaky gut though, it's other people making money. It's been chosen for us, that money is how the world works. It's not how I would do it, but I'm not in charge of the world, so it's a moot point. Everyone is weird about money in their own special way. I am no exception. What sticks in my craw is when people have problems with other people making money. How they make money is material. I'm not okay with making money off of sex trafficking or CSAM, for example, but advertising a product with an interesting bit of writing beforehand isn't that. So on the spectrum of your kid's painting that they made for you in school with crayon that were ethically sourced and drew on recycled paper, to the in your face red plastic Coca-Cola banner wrapped around the side of a bus that's gonna be fed to whales to choke and die on, where this particular blog post lies is for you to determine for yourself. Where I'm really getting at is that requiring X to be at a certain level has the unintended consequence that only big corporations with giant bags of money can create content that passes this purity test of yours, is, if we do some extrapolating, self-defeating.
Why is that an issue? Is it forbidden by HN guidelines? Or would you like all marketing to be marked as such? Which articles aren't marketing, one way or another?
We're actually mostly talking to people (that "schedule a meeting") to see how we can help them migrate their stuff away (from Heroku, Vercel, etc.)
But we're not sure of the pricing model yet - probably Entreprise features like Gitlab does, while remaining open source. It's a tough(er) balance than running a hosted service where you can "just" (over)charge people.
Especially when I got look at the site in question (idealist.org) and it seems to be a pretty boring job board product.
As for the staging servers, for each deployment, it was a mix of Performance-M dynos, multiple Standard dynos, RabbitMQ, a database large enough, etc. - it adds up quickly.
Finally, Idealist serves ~100k users per day - behind the product is a lot of boring tech that makes it reliable & fast. :-)
That's more than 1/3 of the cost of a developer there.
That will save you some week of a person's work to set things up and half-a-day every couple of months to keep it running. Rounding way up.
If you factor in salaries, I can see this ending up costing more than Heroku. $3000/m is one (underpaid) developer here in Scandinavia.
That's the equation that's often lacking, and why Heroku is still worth it where I work.
I did just this using Coolify, Mythic Beasts running Django & Postgres the other month from Google App Engine. Hilariously easy, even with my extremely rusty skills.
Either that or use a PaaS that deploys to VMs. Can't make recommendations here but you could start by looking at Semaphore, Dokku, Dokploy.
I just want someone else handling all that.
They replaced a $500/mo bill with a $55/mo server. And at the same time increased what they could do before - run 5 extra staging environments!
While saving $445/mo isn't nothing, in my book enabling teams to freely run the staging environments they need is the real win here! Limiting testing resources can be a real drag on momentum in a project!
From looking at your docs, it appears like using and connecting GitHub is a necessary prerequisite for using Disco. Is that correct? Can disco also deploy an existing Docker image in a registry of my choosing without a build step? (Something like this with Kamal: `kamal --skip-push --version latest`)
However, yes, you can ask Disco to fetch an existing Docker image (we use that to self-host RabbitMQ). An example of deploying Meilisearch's image is here [0] with the tutorial here [1].
Do you typically build your Docker images and push them to a registry? Curious to learn more about your deployment process.
[0] https://github.com/letsdiscodev/sample-meilisearch/blob/main...
[1] https://disco.cloud/docs/deployment-guides/meilisearch
Genuine question.
All of my staging setups are on a ~$15 Hetzner server, with a GitHub Action to `docker compose build && docker compose up -d` remotely, with an Apache service with a wildcard certificate and dynamic host names. We have 3..n staging setups, with each PR spinning up a new staging site just for that PR.
It's been working with us for years, for a team of 10 developers.
The challenge I always face with homebrew PaaS solutions is that you always end up moving from managing your app to managing your PaaS.
This might not be true right now but as complexity of your app grows it’s almost always the eventual outcome.
They offer convenience
background: I have been self-hosting apps on VPS for a while now but just started using a tool
This seems like a good idea to have plentiful dev environments and avoid a bad pricing model. If your production instance is still on Heroku, you might still want a staging environment on Heroku since a Hetzner server and your production instance might have subtle differences.
What is this referring to? Concerns about capacity if you need to scale up quickly? Or just "political"/marketing considerations about people not being used to being served by a Hetzner server?
Not free, it became a productivity boost.
You now have a $35k annual budget for the maintenance, other overhead, and lost productivity. What do you spend it on?
> The team also took on responsibility for server monitoring, security updates, and handling any infrastructure issues themselves
For a place that’s paying devs $150k a year that might math out. It absolutely does not for places paying devs $250k+ a year.
One of the great frustrations of my mid career is how often people tried to bargain for more speed by throwing developers at my already late project when what would have actually helped almost immediately was more hardware and tooling. But that didn’t build my boss’ or his bosses’ empires. Don’t give me a $150k employee to train, give me $30k in servers.
Absolutely no surprise at all when devs were complicit with Cloud migrations because now you could ask forgiveness instead of permission for more hardware.
But I've migrated plenty of companies off custom deployment setups to PAAS and told many ceo's simply what OP above has shared. Even a part time dev ops engineer is still $60000 a year, and that can buy us a LOT on PAAS. Using PAAS you can have effectively zero dev ops, I've also trained non technical people on how to scale their own servers if no devs are around because you just have a web based UI slider.
I consider myself a developer who cares more about the business, risk, profits and runway. A lot of developers don't share this mentality (which is fine btw always need engineers who like engineering for engineering sakes) but in meetings you will have a hard time beating me in an argument if you try to say that running servers ourselves would be "cheaper", and/or even faster, safer and definitely not more stable. (obviously not in all situations, but kind of most for modern crud web apps that don't require complicated compute setups)
I'm probably being overly antagonistic, forgive me for that, though highly recommend questioning the real cost of running your own setups.
Signed up with Contabo without incident and have been a happy customer ever since.
Look, whether or not Hetzner chose to block me out of blatant geo-racism is not at issue here, I wouldn't want to do business with them either way since they declined to give me a reasonable explanation and I wouldn't feel secure with a provider like that.
The draw of a docker-compose-like interface for deployment is so alluring that I have spent the last year or so working on a tool called Defang that takes a compose file and deploys it to the cloud. We don't support Hetzner (yet), but we do support AWS, GCP, and DO. We provision networking, IAM, compute, database, secrets, etc in your cloud account, so you maintain full control, but you also get the ergonomics of compose.
If you are on a PaaS and you want to reduce cost without losing ergonomics and scalability, it might be interesting.
Unless they plan to move prod and dev as well, and using staging now as a test platform.
Once few problems glitch when moving to prod, they may no longer think they are saving much money.
The key element here is the need to continuously exercise both processes (Heroku + your staging server), to work out both processes & maintain familiarity on both.
Depending on the amount of staff involved in the above, it might eclipse the compute savings, but only OP knows those details. I'm sure they are a smart bunch.
What's in it for Disco ?
What's the pricing ?
How many work hours per month does keeping this thing stable take.
If it takes over 15 Heroku is cheaper.
Hosting with bare metal is still expensive, you pay in other ways.
It's like juniors who did not recieve a proper training/education got hired into companies where someone told them to go serverless on some heroku or vercel, or use some incredibly expensive aws service because that's a "modern correct way" to do it, except now they were a developer for long enough to get a "senior" title in their job title now are in positions of actually modelling this architecture themselves
At a certain scale (or when you do more custom dev), you just don't prioritize it because it's a blip in your financials compared to salaries. And it's not worth spending what might be thousands (or tens of thousands) of dollars of engineering and testing effort to migrate off, when there are other priorities.
And one day you (in our case) realize that you're spending $600/month between Linode/AWS/Fly/Render (cuz we experimented a lot), claude code can help you with some of the stuff, and you want to consolidate it all back to the olden days where you just had 2-3 servers paying a total of $60-75 a month . But you still have to set aside probably a day or two to do it (no longer a couple sprints). So then the math works out where we make that money back within a few months.
So that's where I am at the moment.
https://community.fly.io/t/cpu-quotas-update/23473
https://render.com/docs/migrate-from-heroku
If you're running something that's too expensive for your taste and can share more information, happy to brainstorm some options.
At least, the "fear" factor (will the new system work? what bugs will it introduce? how much time will I spend, etc.) pushes a lot of folks to accept a very big price differential aka known knowns versus unknowns...
It's understandable really. It's just that once you've migrated, you almost definitely never want to go back :-)
But this is also one of those things where, almost invariably after you do it, you think "Oh man, we should've done this 3 years ago!!! It just ended up taking a day and everything's working beautifully"
...but this CX33 "server" being discussed - is a 6 bucks a month VPS [0]
Normally you build a prototype on laptop and move it out to fat hardware when it outgrows that. Here they started with 3k infra and then later realized it runs on toaster. Completely back to front.
Maybe they just never iterated on a local version and nobody developed an intuition for requirements. Switched straight to iterating on a nebulous cloud where you can't tell how much horsepower is behind the cloudfunctions etc.
Presumably there is a perfectly reasonably explanation and it's just not spelled out, it just seems weird based on given info
[0] https://www.hetzner.com/cloud
[0 ]https://www.hetzner.com/cloud
Almost VPS-sized! haha
(I really wish we could support phones as the ultimate self-hosting device)
Quick question: how long would it take to provision and set up another server if this one dies?
But to provision a new server, as these are "stateless" (per 12 Factor) servers, it's just 1) get a VPS 2) install Docker+Disco using our curl|sh install script 3) authorize github 4) deploy a "project" (what we call an app), setting the env vars.
All in all ~10 minutes for a new machine.
[0] https://github.com/gregsadetsky/example-flask-site/blob/main...
Re Load balancing for example, Disco is built on top of Docker Swarm, so you can add nodes (ie machines) to scale horizontally - `disco nodes:add root@<ip>`
For monitoring/alerting, we offer some real time cpu/memory metrics (ie `docker stats`) and integrate with external syslog services.
Do you have specific use cases in mind which current PaaS providers satisfy? Would you say that these kinds of concerns are what's holding you back from leaving Heroku or others (and are you considering leaving because of price, support, etc.)? Cheers
How do I harden the server, back it up, etc? Basically the layer below Disco, to go beyond running it as a "toy"
This is not a dig at Disco, I run into the same issue with virtually any other self-hosted PaaS I could find.
Our philosophy is built on the "cattle, not pets" [0] and 12-factor [1] app methodologies. To some extent, the Disco server itself should be treated as disposable.
Disco runs your applications, which are just deployments of your code (ie git pulls). There's nothing on the server itself to back up. If a server were to die, you'd spin up a new one, run the install.sh script, and redeploy your apps in about 15 minutes.
For application data, our stance is that we believe you should use a dedicated, managed database provider for prod workloads. While we can run a "good enough" postgres as noted, we treat that as a dev/staging tool. Disco handles the stateless application layer, you should entrust your critical stateful data to a service that specializes in that.
Finally, re: security, we recommend a fresh Ubuntu 24.04 LTS server, which handles its own OS security updates. Disco only exposes the necessary web and SSH ports, so the attack surface is minimal by default.
[0] https://cloudscaling.com/blog/cloud-computing/the-history-of...
[1] https://12factor.net/
Which means, that if they want to test what it will look like running in cloud for prod, they are going to either need a pre-prod environment or go yolo
AMD Ryzen™ 7 3700X CPU 8 cores / 16 threads @ 3.6 GHz Generation: Matisse (Zen2) RAM 64 GB DDR4 ECC
Drives 4 x 22 TB HDD 2 x 1 TB SSD
is only 104 euros a month on Hetzner.
The STORAGE alone would cost $1624 a month in most clouds
If your group pushes over 23TiB/Month, than most usually look for un-metered colo hosts, and or CDN services with your resource integrity Attribute tags set.
Cloud makes sense for small IT shops that can't afford a fully staffed team. The general rule is a 5:1 labor to hardware cost amortized over 3 years.
AWS was successful as IT had enormous upfront costs most people found difficult to justify, and data-centers could absorb a DDoS that would be expensive to mitigate.
Rule #17: "Only forward meaningful data at every layer of a design, as it often naturally ensures a feasible convergent behavior."
There are a number of great options, but depends on the use-cases. =3
If you can fit them all on a 4 cpu / 32gb machine, you can easily forgo them and run the stack locally on a dev machine. IME staging environments are generally snowflakes that are hard to stand up (no automation).
$500/month each is a gross overpayment.
Not if you're running with external resources of specific type, or want to share the ongoing work with others. Or need to setup 6 different projects with 3 different databases at the same time. It really depends on your setup and way of working. Sometimes you can do local staging easily, sometimes it's going to be a lot of pain.
> Even with all 6 environments and other projects running, the server's resource usage remained low. The average CPU load stayed under 10%, and memory usage sat at just ~14 GB of the available 32 GB.
$55 server
$550 aws server
$3000 aws based paas server
I mean something like a list of moving parts so I can understand how it works. Perhaps something like this:
https://caprover.com/#:~:text=CapRover%20Architecture%20at%2...
Once everything is installed/running, a very tldr diagram would be:
GitHub (webhook on git push) -> Docker swarm running Caddy -> Disco Daemon REST API which will ask Docker to build the image, and then does a blue-green zero-time deployment swap
But yeah, a clearer/better diagram would be great. Thanks for the push!
And your description is a great macro view of it. Thanks!
I even shown one customer that their elaborate cluster costing £10k a month could run on a £10 vps faster and with less headache (they set it up for "big data" thinking 50GB is massive. There was no expectation of the database growing substantially beyond that).
Their response? Investors said it must run on the cloud, because they don't want to lose their money if homegrown setup goes down.
So there is that.
(Except this backfires, because a service running on a RHEL or Debian machine might go on for 5-10 years untouched without any particular issue, security aside, while anything relying on kubernetes or the hyperscaler's million little services needs to be tweaked every 6 months and re-engineered every few years or it will completely stop working.)
Dokku can be an option if needed to maintain heroku endpoints.
Their advertised prices are not half of Hetzner, but you can find hidden deals if you do a Google search, or via sites like https://racknerdtracker.com (no affiliation).
How do you typically deploy this?
but glad we have new product offering for this
If you setup a server with the curl|sh install script on the homepage, you'll get a url at the end that directs you there. And you can use the CLI too of course.
But yeah, thanks for the reminder!
Bring back sanity to tech.
https://news.ycombinator.com/newsguidelines.html
(I've responded to the other commenter elsewhere in the thread.)
Just something to consider if you are in a professional environment before switching your entire infra: maintenance cost is expensive. I strongly suggest to throw man-days in your cost calculation.
To prevent security vulnerabilities, the team will need to write some playbooks to auto-update regularly your machine, hoping for no breaking changes. Or instead write a pipeline for immutable OS images updates. And it often mean testing on an additional canary VM first.
Scaling up the VM from a compute point of view is not that straightforward as well, and will require depending of the provider either downtime or to migrate the entire deployments to a new instance.
Scaling from a disk size point of view, you will need to play with filesystems.
And depending on the setup you are using, you might have to manage lets encrypt, authentication and authorization, secrets vaults, etc (here at least Disco manages the SSL certs for you)
If you are small enough, you are not going to be truly affected by downtime. If you are just a little bigger, a single hot spare is going to be sufficient.
The place where you get dinged is heavy growth in personnel and bandwidth. You end up needing to solve CPU bound activities quicker because it hurts the whole system. You need to start thinking about sticky round robin load balancing and other fun pieces.
This is where the cloud can allow you to trade money for velocity. Eventually, though, you will need to pay up.
That said, the average SaaS can go a long way with a single server per product.
Only if those man-days actually incur a marginal cost. If it's just employees you already have spending their time on things, then it's not worth factoring in because it's a cost you pay regardless.
It's precisely why we moved from a self-hosted demo environment server to heroku - the developers that had both the skills to manage a server and enough seniority to have access accross all the different projects could bring in more by building.
This part can be outsourced to a PaaS company, so that the company engineers can be focused on what is the company actually making money from.