Commit Graph

73 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
4cdf6b9af9 fusefronted: log more details on WriteAt failures
Also log inode number, fd number, offset and length.

Maybe help debugging https://github.com/rfjakob/gocryptfs/issues/269 .
2018-10-17 22:18:07 +02:00
Jakob Unterwurzacher
f316f1b2df fusefronted: disallow writes running concurrently with reads
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,

  doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,

as the read could pick up a block that has not yet been completely written -
write() is not atomic!

Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.

This also simplifies the file header locking.
2018-07-22 22:29:22 +02:00
Jakob Unterwurzacher
c70df522d2 fusefrontend: doWrite: delete file header if first write fails
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:

  readFileID 2580: incomplete file, got 18 instead of 19 bytes

This could happen when writing the file header works, but writing
the actual data fails.

Now we kill the header again by truncating the file to zero.
2018-07-15 15:12:55 +02:00
Jakob Unterwurzacher
55bb22bad6 fusefrontend: doWrite: no need to take HeaderLock.RLock()
Other writers are blocked by ContentLock already.
2018-07-15 12:40:23 +02:00
Jakob Unterwurzacher
95b93db35f fusefrontend: log prealloc failures at Info level
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.

Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
2018-07-14 15:18:27 +02:00
Jakob Unterwurzacher
01a078e7c0 Fix golint warnings 2018-07-01 22:00:06 +02:00
Jakob Unterwurzacher
a2af1fb5da fusefrontend: export "File" type
"gocryptfs -fsck" will need access to helper functions,
and to get that, it will need to cast a gofuse.File to a
fusefrontend.File. Make fusefrontend.File exported to make
this work.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
1a18d8e609 fsck: rename "CorruptItems" channel to "MitigatedCorruptions"
Make it clear that this channel is only used to report corruptions
that are transparently mitigated and do not return an error to
the user.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
4e5783591f fsck: report skipped corrupt files
OpenDir and ListXAttr skip over corrupt entries,
readFileID treats files the are too small as empty.
This improves usability in the face of corruption,
but hides the problem in a log message instead of
putting it in the return code.

Create a channel to report these corruptions to fsck
so it can report them to the user.

Also update the manpage and the changelog with the -fsck option.

Closes https://github.com/rfjakob/gocryptfs/issues/191
2018-04-03 21:24:48 +02:00
Jakob Unterwurzacher
b6c8960b01 fsck: clean up log output
Make sure we get only 1 warning output per
problem.

Also, add new corruption types to broken_fs_v1.4.
2018-04-02 18:32:30 +02:00
Jakob Unterwurzacher
fb06c65ee9 fusefronted: reject oversized Read and Write requests
This should not happen via FUSE as the kernel caps the size,
but with fsck we have the first user that calls Read directly.
For symmetry, check it for Write as well.
2018-04-01 21:21:55 +02:00
Jakob Unterwurzacher
268e0484e2 Revert most of "fusefrontend: clamp oversized reads"
We cannot return less data than requested to the kernel!

From https://libfuse.github.io/doxygen/structfuse__operations.html:

  Read should return exactly the number of bytes
  requested except on EOF or error, otherwise the
  rest of the data will be substituted with
  zeroes.

Reverts commit 3009ec9852 minus
the formatting improvements we want to keep.

Fixes https://github.com/rfjakob/gocryptfs/issues/147
Reopens https://github.com/rfjakob/gocryptfs/issues/145
2017-10-21 17:43:21 +02:00
Jakob Unterwurzacher
3009ec9852 fusefrontend: clamp oversized reads
Our byte cache pools are sized acc. to MAX_KERNEL_WRITE, but the
running kernel may have a higher limit set. Clamp to what we can
handle.

Fixes a panic on a Synology NAS reported at
https://github.com/rfjakob/gocryptfs/issues/145
2017-10-17 21:48:29 +02:00
Jakob Unterwurzacher
52ab0462a4 fusefrontend: doRead: skip decryption for an empty read
Previously we ran through the decryption steps even for an empty
ciphertext slice. The functions handle it correctly, but returning
early skips all the extra calls.

Speeds up the tar extract benchmark by about 4%.
2017-07-02 16:02:13 +02:00
Jakob Unterwurzacher
12c0101a23 contentenc: add PReqPool and use it in DecryptBlocks
This gets us a massive speed boost in streaming reads.
2017-06-30 23:30:57 +02:00
Jakob Unterwurzacher
b2a23e94d1 fusefrontend: doRead: use CReqPool for ciphertext buffer
Easily saves lots of allocations.
2017-06-30 23:15:31 +02:00
Jakob Unterwurzacher
06398e82d9 fusefrontend: Read: use provided buffer
This will allow us to return internal buffers to a pool.
2017-06-30 23:11:38 +02:00
Jakob Unterwurzacher
80676c685f contentenc: add safer "bPool" pool variant; add pBlockPool
bPool verifies the lengths of slices going in and out.

Also, add a plaintext block pool - pBlockPool - and use
it for decryption.
2017-06-29 23:44:32 +02:00
Jakob Unterwurzacher
3c6fe98eb1 contentenc: use sync.Pool memory pools for encryption
We use two levels of buffers:

1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)

This commit adds a sync.Pool for both levels.

The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
2017-06-20 21:22:00 +02:00
Charles Duffy
da1bd74246 Fix missing Owner coercion for already-open files (#117) 2017-06-09 22:04:56 +02:00
Jakob Unterwurzacher
a24faa3ba5 fusefrontend: write: consolidate and move encryption to contentenc
Collect all the plaintext and pass everything to contentenc in
one call.

This will allow easier parallization of the encryption.

https://github.com/rfjakob/gocryptfs/issues/116
2017-06-01 22:19:27 +02:00
Jakob Unterwurzacher
fb3cc6ea40 openfiletable: rename WriteLock to ContentLock
...and IDLock to HeaderLock. This matches what the locks actually
protect.
2017-05-01 21:57:18 +02:00
Jakob Unterwurzacher
f322ee87e3 fusefrontend: rely on nodefs.defaultFile for no-op functions
Now that we embed nodefs.NewDefaultFile(), we can drop our own
no-ops.
2017-05-01 19:12:37 +02:00
Jakob Unterwurzacher
9ab11aa4d7 fusefrontend: drop writeOnly flag
We do not have to track the writeOnly status because the kernel
will not forward read requests on a write-only FD to us anyway.

I have verified this behavoir manually on a 4.10.8 kernel and also
added a testcase.
2017-05-01 17:49:37 +02:00
Jakob Unterwurzacher
514f515dd7 fusefronted, openfiletable: move the open file table to its own package
The open file table code needs some room to grow for the upcoming
FD multiplexing implementation.
2017-05-01 17:26:50 +02:00
Jakob Unterwurzacher
b66e03486a fusefronted: drop unused file.String() function
This is a very old leftover.
2017-04-29 18:20:39 +02:00
Jakob Unterwurzacher
3409ade272 forcedecode: tighten checks
...and fix a few golint issues and print a scary warning message on mount.

Also, force the fs to ro,noexec.
2017-04-24 00:25:02 +02:00
danim7
f1945c4daa Add -forcedecode
Force decode of encrypted files even if the integrity check fails, instead of
failing with an IO error. Warning messages are still printed to syslog if corrupted
files are encountered.
It can be useful to recover files from disks with bad sectors or other corrupted
media.

Closes https://github.com/rfjakob/gocryptfs/pull/102 .
2017-04-23 23:11:56 +02:00
Jakob Unterwurzacher
9777e4bf7e Fix Flock build breakage
go-fuse has added a new method to the nodefs.File interface that
caused this build error:

  internal/fusefrontend/file.go:75: cannot use file literal (type *file) as type nodefs.File in return argument:
  	*file does not implement nodefs.File (missing Flock method)

Fixes https://github.com/rfjakob/gocryptfs/issues/104 and
prevents the problem from happening again.
2017-04-23 00:06:56 +02:00
Jakob Unterwurzacher
cb47f65212 fusefrontend: get rid of leftover debug output 2017-03-18 16:48:28 +01:00
Jakob Unterwurzacher
00df0771e3 serialize_reads: add read serialization logic
Due to kernel readahead, we usually get multiple read requests
at the same time. These get submitted to the backing storage in
random order, which is a problem if seeking is very expensive.

Details: https://github.com/rfjakob/gocryptfs/issues/92
2017-03-18 16:18:00 +01:00
Jakob Unterwurzacher
14038a1644 fusefrontend: readFileID: reject files that consist only of a header
A header-only file will be considered empty (this is not supposed to happen).
This makes File ID poisoning more difficult.
2017-03-12 21:11:02 +01:00
Jakob Unterwurzacher
0f8d3318a3 main, fusefrontend: add "-noprealloc" option
Preallocation is very slow on hdds that run btrfs. Give the
user the option to disable it. This greatly speeds up small file
operations but reduces the robustness against out-of-space errors.

Also add the option to the man page.

More info: https://github.com/rfjakob/gocryptfs/issues/63
2016-11-25 09:19:14 +01:00
Jakob Unterwurzacher
024511d9c7 fusefrontend: coalesce 4kB writes
This improves performance on hdds running ext4, and improves
streaming write performance on hdds running btrfs. Tar extract
slows down on btrfs for some reason.

See https://github.com/rfjakob/gocryptfs/issues/63

Benchmarks:

encfs v1.9.1
============

$ ./benchmark.bash -encfs /mnt/hdd-ext4
Testing EncFS at /mnt/hdd-ext4/benchmark.bash.u0g
WRITE: 131072000 bytes (131 MB, 125 MiB) copied, 1,48354 s, 88,4 MB/s
UNTAR: 20.79
LS:    3.04
RM:    6.62

$ ./benchmark.bash -encfs /mnt/hdd-btrfs
Testing EncFS at /mnt/hdd-btrfs/benchmark.bash.h40
WRITE: 131072000 bytes (131 MB, 125 MiB) copied, 1,52552 s, 85,9 MB/s
UNTAR: 24.51
LS:    2.73
RM:    5.32

gocryptfs v1.1.1-26-g4a7f8ef
============================

$ ./benchmark.bash /mnt/hdd-ext4
Testing gocryptfs at /mnt/hdd-ext4/benchmark.bash.1KG
WRITE: 131072000 bytes (131 MB, 125 MiB) copied, 1,55782 s, 84,1 MB/s
UNTAR: 22.23
LS:    1.47
RM:    4.17

$ ./benchmark.bash /mnt/hdd-btrfs
Testing gocryptfs at /mnt/hdd-btrfs/benchmark.bash.2t8
WRITE: 131072000 bytes (131 MB, 125 MiB) copied, 6,87206 s, 19,1 MB/s
UNTAR: 69.87
LS:    1.52
RM:    5.33

gocryptfs v1.1.1-32
===================

$ ./benchmark.bash /mnt/hdd-ext4
Testing gocryptfs at /mnt/hdd-ext4/benchmark.bash.Qt3
WRITE: 131072000 bytes (131 MB, 125 MiB) copied, 1,22577 s, 107 MB/s
UNTAR: 23.46
LS:    1.46
RM:    4.67

$ ./benchmark.bash /mnt/hdd-btrfs/
Testing gocryptfs at /mnt/hdd-btrfs//benchmark.bash.XVk
WRITE: 131072000 bytes (131 MB, 125 MiB) copied, 3,68735 s, 35,5 MB/s
UNTAR: 116.87
LS:    1.84
RM:    6.34
2016-11-25 09:03:32 +01:00
Jakob Unterwurzacher
0489d08ae2 fusefrontend: get the file ID from the open files table
This fixes the problem that a truncate can reset the file
ID without the other open FDs noticing it.
2016-11-17 22:29:45 +01:00
Jakob Unterwurzacher
e04dc05012 fusefrontend: upgrade wlockMap to use device AND inode number
If there are multiple filesystems backing the gocryptfs filesystems
inode numbers are not guaranteed to be unique.
2016-11-17 20:32:19 +01:00
Jakob Unterwurzacher
c2192cfcad fusefrontend: drop atime workarounds
The fix at https://github.com/hanwen/go-fuse/pull/131 has been merged.
Drop the workarounds and re-enable the tests.
2016-10-30 16:29:36 +01:00
Jakob Unterwurzacher
85f1fd0b0f fusefronted: more concise corrupt block log message
Calculating the block offset is easy enough, even more now
that gocryptfs-xray exists.
2016-10-28 21:18:36 +02:00
Jakob Unterwurzacher
a08d55f42d fusefronted: optimize NFS streaming writes by saving one Stat()
Stat() calls are expensive on NFS as they need a full network
round-trip. We detect when a write immediately follows the
last one and skip the Stat in this case because the write
cannot create a file hole.

On my (slow) NAS, this takes the write speed from 24MB/s to
41MB/s.
2016-10-28 21:17:53 +02:00
Jakob Unterwurzacher
d64ccf7cf4 fusefrontend: move hole padding check out of Write()
The details of the hole handling don't have to be in
Write, so move it away.
2016-10-25 22:37:45 +02:00
Jakob Unterwurzacher
6538dc15af fusefrontend: rename "createsHole" to clearer "createsCiphertextHole"
...and add comments for what is happening.
2016-10-25 21:19:37 +02:00
Jakob Unterwurzacher
aeda9721d0 Fix misspellings
Close https://github.com/rfjakob/gocryptfs/issues/54
2016-10-24 19:18:13 +02:00
Jakob Unterwurzacher
589748548f tests: add 1980.tar.gz extract test
Test that we get the right timestamp when extracting a tarball.

Also simplify the workaround in doTestUtimesNano() and fix the
fact that it was running no test at all.
2016-10-19 22:25:54 +02:00
Jakob Unterwurzacher
600ceece35 lint fixes 2016-10-19 01:12:45 +02:00
Jakob Unterwurzacher
891a3b4c8a fusefrontend: Utimens: one more band-aid
Revert once https://github.com/hanwen/go-fuse/pull/131 is merged.
2016-10-16 20:20:00 +02:00
Jakob Unterwurzacher
5144470e3d fusefrontend: Utimens: ugly band-aid for nil pointer crash in go-fuse
Crash is described at https://github.com/rfjakob/gocryptfs/issues/48 .
Revert this once https://github.com/hanwen/go-fuse/pull/131 is merged.
2016-10-16 15:08:05 +02:00
Valient Gough
b764917cd5 lint fixes 2016-10-04 23:18:33 +02:00
Jakob Unterwurzacher
a2510efe12 reverse: use per-purpose nonce generation
Also pull all the deterministic nonce code into fusefrontend_reverse
to greatly simplify the normal code path.
2016-09-29 21:56:49 +02:00
Jakob Unterwurzacher
12808138ef contentenc: add "ExternalNonce" mode
This will be used for strong symlink encryption in reverse mode.
2016-09-25 17:44:19 +02:00
Jakob Unterwurzacher
5f726aaa9d contentenc: add GCM-SIV support
Also add ReverseDummyNonce nonce generation.
2016-09-25 16:43:17 +02:00