Does Linux need a new file system? Ex-Google engineer thinks so: Bcache author says it's nearly read - Polidicks

[b]Does Linux need a new file system? Ex-Google engineer thinks so[/b] Source: [url=http://www.theregister.co.uk/2015/08/24/does_linux_need_a_new_file_system_exgoogle_engineer_thinks_so/]Register.co.uk[/url] _________________________ [quote]Former Googler Kent Overstreet has announced that a long-term project to craft a new Linux file system is at a point where he'd like other developers to pitch in. Since you're already asking why bother with a new file system, the answer Overstreet provides in this post to the kernel mailing list is that he wants to “match ext4 and xfs on performance and reliability, but with the features of btrfs/zfs”. He says “the bcache codebase has been evolving/metastasizing into a full blown, general purpose posix filesystem – a modern COW [copy-on-write – El Reg] filesystem with checksumming, compression, multiple devices, caching, and eventually snapshots and all kinds of other nifty features”. Regarding key features of the filesystem, Overstreet says multiple device support is “80 per cent” complete with the recovery code to be completed; caching and tiering are functional, at least; compression currently uses zlib only, and there's a choice between crc32c or a 64 bit checksum. “My main priority is getting the code sufficiently stable and tested for production use, probably the #2 priority is snapshots. Bcachefs won't be done in a month (or a year), but I do want to see it out there and getting used”, he writes. Regarding the need for other contributors. Overstreet remarks that he's been working on it as a full-time project while living on savings, so “this would be a wonderful time both for other developers to jump in and get involved, and for potential users to pony up some funding”. The project is at Github, here and here, for those that want to try it out.[/quote] Pretty cool, we'll see how this goes.

You are about to see how hard it is off paper to switch file systems. I mean, a lot of new devices still default to FAT.

Microsoft will, of course, refuse to provide native support for it in windows no matter how ubiquitous it becomes.

[QUOTE=froztshock;48529474]Microsoft will, of course, refuse to provide native support for it in windows no matter how ubiquitous it becomes.[/QUOTE] Linux doesnt have native ntfs support either

[QUOTE=Cmx;48529563]Linux doesnt have native ntfs support either[/QUOTE] That's because they legally can't given Microsoft's stance. Microsoft on the other hand could create and document a proper API that actually allows third-party filesystems to both properly integrate and give them low-level disk device access. Lack of that made adding third-party Extra 3&4 support a bitch.

[t]http://i.imgur.com/4npnCS1.png[/t] What do you call native?.

This isn't really a solve-all, but it does sure help. [URL]http://www.ext2fsd.com[/URL] It works pretty damn seamlessly, and have yet to run into any issues. I primarily use it (alongside OSFMount) for mounting my Android partition images.

[QUOTE=initrd;48529679][t]http://i.imgur.com/4npnCS1.png[/t] What do you call native?.[/QUOTE] As far as I know, that doesn't implement the entirety of NTFS.

[QUOTE=froztshock;48529474]Microsoft will, of course, refuse to provide native support for it in windows no matter how ubiquitous it becomes.[/QUOTE] Microsoft is trying to ditch NTFS as well ([URL="https://en.wikipedia.org/wiki/ReFS"]ReFS[/URL]), and with their recent crush on Linux, we may see proper kernel support. It would be in Microsoft's interest for the sake of the Azure platform.

Relevant [img]https://imgs.xkcd.com/comics/standards.png[/img] We probably won't see this one become stable for the next few years anyway. Isn't btrfs like 10 years old and still considered "unstable"?

[QUOTE=froztshock;48529474]Microsoft will, of course, refuse to provide native support for it in windows no matter how ubiquitous it becomes.[/QUOTE] if their customers start to demand it they will.

[QUOTE=Naelstrom;48530856]Relevant [img]https://imgs.xkcd.com/comics/standards.png[/img] We probably won't see this one become stable for the next few years anyway. Isn't btrfs like 10 years old and still considered "unstable"?[/QUOTE] I swear this is the most posted comic on this entire forum

[QUOTE=Naelstrom;48530856]Relevant [img]https://imgs.xkcd.com/comics/standards.png[/img] We probably won't see this one become stable for the next few years anyway. Isn't btrfs like 10 years old and still considered "unstable"?[/QUOTE] Irrelevant. It's a filesystem, not a standard.

IMO ExFAT should be the universal standard [editline]24th August 2015[/editline] [QUOTE=vercas;48530964]Irrelevant. It's a filesystem, not a standard.[/QUOTE] Not irrelevant. File systems are standards too.

[QUOTE=BFG9000;48530965]IMO ExFAT should be the universal standard [editline]24th August 2015[/editline] Not irrelevant. File systems are standards too.[/QUOTE] And obviously people should stop making filesystems because we have too many, amirite? Since [b]you[/b] interface with them all in the exact same way, it's just one standard. A filesystem is a specification, not a standard, and the driver is an implementation.

[QUOTE=vercas;48530999]And obviously people should stop making filesystems because we have too many, amirite? Since [b]you[/b] interface with them all in the exact same way, it's just one standard. A filesystem is a specification, not a standard, and the driver is an implementation.[/QUOTE] You don't interface with them all in the exact same way, btrfs has way more features and functionality than ext4 (system imaging, per-file RAID, etc). They're competing standards and I think it's ridiculous that you would go to argue about the specifics of an ambiguous word anyway.

[QUOTE=Naelstrom;48530856]Relevant [img]https://imgs.xkcd.com/comics/standards.png[/img] We probably won't see this one become stable for the next few years anyway. Isn't btrfs like 10 years old and still considered "unstable"?[/QUOTE] Btrfs is considered both stable and unstable. It depends who you ask, and what features you require. Is Btrfs currently (Linux 4.1.5) stable enough for use as an ext4 replacement? Absolutely. Is it stable enough for a mdraid replacement? Depends. On what exactly, I am not sure, as I've used it merely as a replacement for ext4, for a couple of years now, never touching anything else.

Can someone who knows this stuff explain the nitty gritty of file systems to me? Like why are new file systems needed? Why don't existing ones fulfill any and all needs when these things have been around for decades? I'm sure it's much more complex than I'm imagining but it seems like a relatively straightforward thing to design, and then could be left alone for a long time. I really only know the vague differences between fat32 and ntfs like how large files can be. If you've got resources to read I'd be interested in that too :)

[QUOTE=DOG-GY;48533831]Can someone who knows this stuff explain the nitty gritty of file systems to me? Like why are new file systems needed? Why don't existing ones fulfill any and all needs when these things have been around for decades? I'm sure it's much more complex than I'm imagining but it seems like a relatively straightforward thing to design, and then could be left alone for a long time. I really only know the vague differences between fat32 and ntfs like how large files can be. If you've got resources to read I'd be interested in that too :)[/QUOTE] Fragmentation, corruption, power loss during write, compression, speed, case sensitivity, file recovery, copy on write, and raid features are all things that file systems have to deal with. All filesystems handle all of these issues differently, someone wants one that combines the speed and stability of ext4 and the neat features of btrfs. You can watch this talk about why btrfs is cool: [url]http://youtube.com/watch?v=hxWuaozpe2I[/url] It includes a graphical comparison between ext4 and btrfs at some point.

[QUOTE=Naelstrom;48533990]Fragmentation, corruption, power loss during write, compression, speed, case sensitivity, file recovery, copy on write, and raid features are all things that file systems have to deal with. All filesystems handle all of these issues differently, someone wants one that combines the speed and stability of ext4 and the neat features of btrfs. You can watch this talk about why btrfs is cool: [url]http://youtube.com/watch?v=hxWuaozpe2I[/url] It includes a graphical comparison between ext4 and btrfs at some point.[/QUOTE] Thanks babe. So you think there's any kind of endgame for these things? It seems reasonable to me that there may be a point where the hardware and low level software is so standardized there's little need to do more than plug in the relevant codebase so to speak.

[QUOTE=mastersrp;48529994]As far as I know, that doesn't implement the entirety of NTFS.[/QUOTE] The kernel driver isn't complete but the userspace ntfs-3g driver is better(and slower iirc). On the matter of having any kind of software standarized, i don't think that's possible, you might want a filesystem that tries to save space as much as possible while someone else might want one that minimizes latency for certain kinds of operations or whatever. By the way exFAT as an standard? [URL="https://en.wikipedia.org/wiki/ExFAT#Restrictive_licensing_and_software_patents"]Terrible idea[/URL].

[QUOTE=DOG-GY;48534069]Thanks babe. So you think there's any kind of endgame for these things? It seems reasonable to me that there may be a point where the hardware and low level software is so standardized there's little need to do more than plug in the relevant codebase so to speak.[/QUOTE] You're trying to ask if we'll all be using one superior filesystem in the future? I don't think so, no. There will never be a standardization that everyone agrees on. No file-system can cover every use case without having drawbacks of some kind. Also with technology changing so fast the idea of perfection moves as well. Btrfs seems pretty cool with all this RAID stuff and CRC checksums, but they won't be so cool when we have harddrives that literally cannot be corrupted. If you get what I mean.

[QUOTE=initrd;48534246]The kernel driver isn't complete but the userspace ntfs-3g driver is better(and slower iirc). On the matter of having any kind of software standardized, i don't think that's possible, you might want a filesystem that tries to save space as much as possible while someone else might want one that minimizes latency for certain kinds of operations or whatever. By the way exFAT as an standard? [URL="https://en.wikipedia.org/wiki/ExFAT#Restrictive_licensing_and_software_patents"]Terrible idea[/URL].[/QUOTE] well, exFAT was chosen as the standard filesystem for SDXC primarily due to the fact that it is the only filesystem that is compatible across the majority of the consumer facing computers on the planet (windows vista onward plus macOS 10.6.5 onwards). Due to the adoption of it in the primary removable storage format in existence in 2009, it became defacto standard even in android. Something had to give with the larger sdxc format primarily due to the fact that fat32 is terrible for large file sizes and/or high numbers of files, and they needed to retain compatibility with windows PCs, while not having the cumbersome limits of the comparatively ancient FAT32. exFAT is a good file system when it comes to a flash memory oriented format. The actual grounds behind having MS be the creator of said filesystem is another issue altogether.