bcachefs

Status update

 - Interior btree node updates are now journalled; removing the need for btree writes to be FUA

 - Interior btree node updates are now fully transactional, we no longer have to do a...

Towards snapshots

Just finished a major rework that gets us a step closer to snapshots: the btree code is incrementally being changed to handle extents like regular keys.

Previously, when reading in a btree no...

Status update

There is now a (very work-in-progress) fuse port!

The fuse port isn't intended to ever be for serious use - but I do expect it to be useful for debugging in the future; if someone is hitting ...

At long last - reflink is done

For those who aren't familiar with the idea - reflink means using shared, reference counted extents to do "shallow copies" - copies that share data transparently on disk, but are copy on write (unl...

Still hacking away at reflink

It's pretty close to done, but working through the last of the xfstests failures has been tedious.

But - I just pushed out a punch of prep work patches, and something else cool is now done - ...

Notes on Phoronix benchmarks

Phoronix posted some bcachefs benchmarks: https://www.phoronix.com/scan.php?page=arti...

Fully persistent allocation info is finally done

Finally! It was a huge effort, but it's done and pushed out.

This means that when mounting a filesystem - even after an unclean shutdown - we don't have to walk all the metadata anymore, bec...

Status update

5.0 rebase is up

And, more importantly - fully persistent allocation info is finally just about done! It's passing the tests, not much left before I can push it out...

Status update - persistent alloc info

So, first some background:

Fully persistent allocation info is going to require updating the alloc btree every time we update the extents btree - one key in the alloc btree for every pointer ...

More on fully persistent allocation information

So, to recap: bcachefs now persists allocation information on clean shutdown, so mounting after a clean shutdown doesn't require walking any metadata. However, we're not yet keeping allocation info...

Fast mounts update

Persistent alloc info for clean shutdowns is finally done - this means when mounting after a clean shutdown, we don't have to scan metadata anymore, and mounting should be just as fast or faster th...

bcachefs at FOSDEM

I'll be at FOSDEM. I'm not planning on giving a talk or anything, but if anyone else is interested and is going to be there, send a message and I'd love to meet up.

Status update - quotas and option handling

Option handling improvements: There's a single master list of option in opts.h, and that list is now used by bcachefs format as well, including for bcachefs format --help. This is a nice usability ...

Status update - fast mount times, reflink

So for now, I'm leaving off the remaining parts of erasure coding - the important part was getting everything done that impacts both the on disk format, and the rest of the design. There's some common...

Erasure coding has been pushed

It's not production ready yet - stripe level copygc isn't implemented yet, so disk fragmentation could lead to your filesystem getting filled with partially empty stripes and getting stuck. But, aside...

Erasure coding is coming!

First off, sorry for the slow progress lately - I've been dealing with some health issues that have been making it incredibly difficult to work. But, the good news is that we may have finally figured ...

Bcachefs extents - compression, checksumming

One topic that was asked about recently was compression in bcachefs, so I thought I'd write a bit about how extents are represented as a bunch of stuff falls out of that.

In bcachefs, checksumming a...

Vote for the next deep dive topic!

I've gotten a few comments that people have been enjoying my technical deep dives into things I'm working on.

There's a lot of other things I could write about as well, not just bcachefs ...

Filesystem metadata operations are now all fully atomic

In the last post, I wrote about some new transaction infrastructure I was working on that would make it practical to make all the high level filesystem operations (e.g. create, link, unlink) fully ato...

Progress towards faster mount times - new transaction infrastructure

I've talked a bit before about the new transaction infrastructure I've been working on, but to recap:

bcachefs has, for quite some time, had the ability to use multiple btree iterators simultaneousl...

Btree unit tests

Been spending a surprising amount of time lately on the core btree - in a good way, as in "oh, here's some good an useful improvements I can easily make", not "oh crap, this thing is broken and I have...

The bug squashing continues...

Been squashing quite a few bugs lately, but this latest one has been quite a trip down the rabbit hole...

Initial symptom was that on xfstest generic/475, very occasionally we'd see an extent past t...

Status update

definitely not drunk debugging right now

I know I've been shit at posting updates, so ask your questions now - about what's going on with upstreaming or anything else you can think of

New feature: specify a device's durability

Just pushed a new feature (only lightly tested so far): when formatting, you can specify a "durability" for each device: the effect of this is that data on that device will be counted as being replica...

Tiering is dead; long live disk groups

The new disk groups-based code for configuring data placement has been merged, and the notion of configuring disks into "tiers" has been removed. If you have an existing filesystem that uses tiering, ...

Just pushed support for zstd compression

Please test (and don't assume it won't eat all your data)

ktest

The test framework I use for bcachefs - ktest - has been getting various cleanups and fixes to make it easier for other people to use - in particular, it works on non debian distributions now.

For a...

Initramfs support for root on encrypted bcachefs

I just pushed initrams hooks/scripts for handling a bcachefs encrypted root filesystem - after you make install in bcachefs-tools, they'll be picked up next time you generate an initramfs, and if your...

New rereplicate tool; replication ready for testing

Replication support is finally feature complete; it should have everything implemented that's needed for handling and recoving from device failure.

If replication is enabled on a filesystem, a ...

Migrate tool

just fixed some bugs in the migrate tool, should be working again