Commit Graph

322 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
72b975867a fusefronted: allow_other: close race between mknod and chown
If the user manages to replace the directory with
a symlink at just the right time, we could be tricked
into chown'ing the wrong file.

This change fixes the race by using fchownat, which
unfortunately is not available on darwin, hence a compat
wrapper is added.

Scenario, as described by @slackner at
https://github.com/rfjakob/gocryptfs/issues/177 :

1. Create a forward mount point with `plaintextnames` enabled
2. Mount as root user with `allow_other`
3. For testing purposes create a file `/tmp/file_owned_by_root`
   which is owned by the root user
4. As a regular user run inside of the GoCryptFS mount:

```
mkdir tempdir
mknod tempdir/file_owned_by_root p &
mv tempdir tempdir2
ln -s /tmp tempdir
```

When the steps are done fast enough and in the right order
(run in a loop!), the device file will be created in
`tempdir`, but the `lchown` will be executed by following
the symlink. As a result, the ownership of the file located
at `/tmp/file_owned_by_root` will be changed.
2017-11-27 21:04:45 +01:00
Jakob Unterwurzacher
1bb47b6796 reverse: reject too-long symlink target reads with ENAMETOOLONG
If the symlink target gets too long due to base64 encoding, we should
return ENAMETOOLONG instead of having the kernel reject the data and
returning an I/O error to the user.

Fixes https://github.com/rfjakob/gocryptfs/issues/167
2017-11-26 21:37:12 +01:00
Sebastian Lackner
90687215a4 fusefrontend_reverse: Do not mix up cache information for different directories
Fixes https://github.com/rfjakob/gocryptfs/issues/168

Steps to reproduce the problem:

* Create a regular reverse mount point
* Create files with the same very long name in multiple directories - so far
  everything works as expected, and it will appear with a different name each
  time, for example, gocryptfs.longname.A in directory A and
  gocryptfs.longname.B in directory B
* Try to access a path with A/gocryptfs.longname.B or B/gocryptfs.longname.A -
  this should fail, but it actually works.

The problem is that the longname cache only uses the path as key and not the
dir or divIV. Assume an attacker can directly interact with a reverse mount and
knows the relation longname path -> unencoded path in one directory, it allows
to test if the same unencoded filename appears in any other directory.
2017-11-25 16:20:48 +01:00
Sebastian Lackner
95870e841e fusefrontend: Skip gocryptfs.diriv handling when directory was deleted successfully
Fixes https://github.com/rfjakob/gocryptfs/issues/171

Steps to reproduce:

* Create a regular forward mount point
* Create a new directory in the mount point
* Manually delete the gocryptfs.diriv file from the corresponding ciphertext
  directory
* Attempt to delete the directory with 'rmdir <dirname>'

Although the code explicitly checks for empty directories, it will still attempt
to move the non-existent gocryptfs.diriv file and fails with:

    rmdir: failed to remove '<dirname>': No such file or directory
2017-11-25 16:20:00 +01:00
Sebastian Lackner
9f56b33e0c fusefrontend: Fix longname handling for renames with existing target
Fixes https://github.com/rfjakob/gocryptfs/issues/170

Steps to reproduce the problem:

* Create a regular forward mount point
* Create a file with a shortname and one with a long filename
* Try to run 'mv <shortname> <longname>'

This should actually work and replace the existing file, but instead it
fails with:

    mv: cannot move '<shortname>' to '<longname>': File exists

The problem is the creation of the .name file. If the target already exists
we can safely ignore the EEXIST error and just keep the existing .name file.
2017-11-25 16:19:09 +01:00
Sebastian Lackner
f80f19f589 fusefrontend_reverse: Add a missing Close() call 2017-11-22 23:42:49 +01:00
Sebastian Lackner
c547673529 nametransform: Return error if decrypted name is '.' or '..' 2017-11-22 23:42:08 +01:00
Sebastian Lackner
f3c777d5ea main: Add '-devrandom' commandline option
Allows to use /dev/random for generating the master key instead of the
default Go implementation. When the kernel random generator has been
properly initialized both are considered equally secure, however:

* Versions of Go prior to 1.9 just fall back to /dev/urandom if the
  getrandom() syscall would be blocking (Go Bug #19274)

* Kernel versions prior to 3.17 do not support getrandom(), and there
  is no check if the random generator has been properly initialized
  before reading from /dev/urandom

This is especially useful for embedded hardware with low-entroy. Please
note that generation of the master key might block indefinitely if the
kernel cannot harvest enough entropy.
2017-11-21 23:37:06 +01:00
Jakob Unterwurzacher
34547a6c39 tests: don't read /proc, the number of entries changes too quickly
This could lead to test failures like this:

  --- FAIL: TestGetdents (0.02s)
  	getdents_test.go:57: len(getdentsEntries)=362, len(readdirEntries)=360
  FAIL
2017-10-22 18:13:08 +02:00
Jakob Unterwurzacher
268e0484e2 Revert most of "fusefrontend: clamp oversized reads"
We cannot return less data than requested to the kernel!

From https://libfuse.github.io/doxygen/structfuse__operations.html:

  Read should return exactly the number of bytes
  requested except on EOF or error, otherwise the
  rest of the data will be substituted with
  zeroes.

Reverts commit 3009ec9852 minus
the formatting improvements we want to keep.

Fixes https://github.com/rfjakob/gocryptfs/issues/147
Reopens https://github.com/rfjakob/gocryptfs/issues/145
2017-10-21 17:43:21 +02:00
Jakob Unterwurzacher
29445c976d contentenc: reserve one additional block in CReqPool
...to account for unaligned reads.

I have not seen this happen in the wild because the kernel
always seems to issue 4k-aligned requests. But the cost
of the additional block in the pool is low and prevents
a buffer overrun panic when an unaligned read does happen.
2017-10-19 09:23:10 +02:00
Jakob Unterwurzacher
3009ec9852 fusefrontend: clamp oversized reads
Our byte cache pools are sized acc. to MAX_KERNEL_WRITE, but the
running kernel may have a higher limit set. Clamp to what we can
handle.

Fixes a panic on a Synology NAS reported at
https://github.com/rfjakob/gocryptfs/issues/145
2017-10-17 21:48:29 +02:00
Jakob Unterwurzacher
4da245c69d fusefrontend_reverse: fix 176-byte names
A file with a name of exactly 176 bytes length caused this error:

  ls: cannot access ./tmp/dsg/sXSGJLTuZuW1FarwIkJs0w/b6mGjdxIRpaeanTo0rbh0A/QjMRrQZC_4WLhmHI1UOBcA/gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY: No such file or directory
  ls: cannot access ./tmp/dsg/sXSGJLTuZuW1FarwIkJs0w/b6mGjdxIRpaeanTo0rbh0A/QjMRrQZC_4WLhmHI1UOBcA/gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY.name: No such file or directory
  -????????? ? ?     ?             ?            ? gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY
  -????????? ? ?     ?             ?            ? gocryptfs.longname.QV-UipdDXeUVdl05WruoEzBNPrQCfpu6OzJL0_QnDKY.name

Root cause was a wrong shortNameMax constant that failed to
account for the obligatory padding byte.

Fix the constant and also expand the TestLongnameStat test case
to test ALL file name lengths from 1-255 bytes.

Fixes https://github.com/rfjakob/gocryptfs/issues/143 .
2017-10-01 13:50:25 +02:00
Jakob Unterwurzacher
0072a96f20 siv_aead: fix trivial typo in comment 2017-09-17 11:42:46 +02:00
Jakob Unterwurzacher
885fdcabda contentenc: deduplicate AD packing into new concatAD() func
The encrypt and decrypt path both had a copy that were equivalent
but ordered differently, which was confusing.

Consolidate it in a new dedicated function.
2017-09-17 11:21:48 +02:00
Jakob Unterwurzacher
4bd2c6736a contentenc: DecryptBlocks: give block number counter a clearer name
Using firstBlockNo as the counter is confusing, create a
copy named "blockNo" and use that.
2017-09-17 10:59:04 +02:00
Jakob Unterwurzacher
604b0779d4 macos: automatically remove .DS_Store on Rmdir
MacOS sprinkles .DS_Store files everywhere. This is hard to avoid for
users, so handle it transparently in Rmdir().

Mitigates https://github.com/rfjakob/gocryptfs/issues/140
2017-09-05 22:47:15 +02:00
Jakob Unterwurzacher
6f3b65d924 fusefrontend: reorder logic in Rmdir to get rid of one indentation level
Handle the errors first so that the normal code path is not indented.

This should not cause any behavoir changes.
2017-09-05 22:10:08 +02:00
Jakob Unterwurzacher
3a5a783b54 macos: don't throw IO errors because of .DS_Store files
MacOS creates lots of these files, and if the directory is otherwise
empty, we would throw an IO error to the unsuspecting user.

With this patch, we log a warning, but otherwise pretend we did not
see it.

Mitigates https://github.com/rfjakob/gocryptfs/issues/140
2017-09-05 21:47:05 +02:00
Jakob Unterwurzacher
538cae610c syscallcompat: Getdents: warn once if we get DT_UNKNOWN
...and if Getdents is not available at all.

Due to this warning I now know that SSHFS always returns DT_UNKNOWN:

    gocryptfs[8129]: Getdents: convertDType: received DT_UNKNOWN, falling back to Lstat

This behavoir is confirmed at http://ahefner.livejournal.com/16875.html:

    "With sshfs, I finally found that obscure case. The dtype is always set to DT_UNKNOWN [...]"
2017-09-03 15:05:54 +02:00
Jakob Unterwurzacher
276567eb13 fusefrontend: use DirIVCache in OpenDir()
Previously, OpenDir() did not use the cache at all, missing
an opportunity to speed up repeated directory reads.
2017-09-03 13:59:53 +02:00
Jakob Unterwurzacher
7da0e97c8b dirivcache: add better function comments + a sanity check on Store()
The comments were unclear on whether relative or absolute paths
have to be passed.
2017-09-03 13:53:50 +02:00
Jakob Unterwurzacher
ed046aa359 Fix misspellings reported by goreportcard.com
https://goreportcard.com/report/github.com/rfjakob/gocryptfs#misspell
2017-08-21 21:06:05 +02:00
Jakob Unterwurzacher
312ea32bb7 cryptocore: add urandom + randprefetch benchmarks
The benchmark that supported the decision for 512-byte
prefetching previously lived outside the repo.

Let's add it where it belongs so it cannot get lost.
2017-08-16 18:33:00 +02:00
Jakob Unterwurzacher
989b880989 fusefrontend: use Getdents if available
Getdents avoids calling Lstat on each file.
2017-08-15 19:04:02 +02:00
Jakob Unterwurzacher
e50a6a57e5 syscallcompat: implement Getdents()
The Readdir function provided by os is inherently slow because
it calls Lstat on all files.

Getdents gives us all the information we need, but does not have
a proper wrapper in the stdlib.

Implement the "Getdents()" wrapper function that calls
syscall.Getdents() and parses the returned byte blob to a
fuse.DirEntry slice.
2017-08-15 19:03:57 +02:00
Jakob Unterwurzacher
0c520845f3 main: purge masterkey from memory as soon as possible
Remove the "Masterkey" field from fusefrontend.Args because it
should not be stored longer than neccessary. Instead pass the
masterkey as a separate argument to the filesystem initializers.

Then overwrite it with zeros immediately so we don't have
to wait for garbage collection.

Note that the crypto implementation still stores at least a
masterkey-derived value, so this change makes it harder, but not
impossible, to extract the encryption keys from memory.

Suggested at https://github.com/rfjakob/gocryptfs/issues/137
2017-08-11 19:02:26 +02:00
Jakob Unterwurzacher
e80b5f2049 nametransform: extend diriv cache to 100 entries
* extend the diriv cache to 100 entries
* add special handling for the immutable root diriv

The better cache allows to shed some complexity from the path
encryption logic (parent-of-parent check).

Mitigates https://github.com/rfjakob/gocryptfs/issues/127
2017-08-09 22:00:53 +02:00
Jakob Unterwurzacher
75ec94a87a nametransform: add Dir() function
Dir is like filepath.Dir but returns "" instead of ".".
This was already implemented in fusefrontend_reverse as saneDir().

We will need it in nametransform for the improved diriv caching.
2017-08-06 23:14:39 +02:00
Jakob Unterwurzacher
5190cc09bb nametransform: move diriv cache into it's own package
Needs some space to grow.

renamed:    internal/nametransform/diriv_cache.go -> internal/nametransform/dirivcache/dirivcache.go
2017-08-06 21:59:15 +02:00
Jakob Unterwurzacher
32611ff97a nametransform: deduplicate code to encryptAndHashName()
This operation has been done three time by identical
sections of code. Create a function for it.
2017-08-06 21:23:42 +02:00
Jakob Unterwurzacher
d12aa57715 fusefronted_reverse: fix ino collision between .name and .diriv files
A directory with a long name has two associated virtual files:
the .name file and the .diriv files.

These used to get the same inode number:

  $ ls -di1  * */*
             33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name

With this change we use another prefix (2 instead of 1) for .name files.

  $ ls -di1 * */*
             33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
  2000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
2017-07-29 16:15:49 +02:00
Jakob Unterwurzacher
d5133ca5ac fusefrontend_reverse: return ENOENT for undecryptable names
This was working until DecryptName switched to returning
EBADMSG instead of EINVAL.

Add a test to catch the regression next time.
2017-07-27 20:31:22 +02:00
Jakob Unterwurzacher
ccf1a84e41 macos: make testing without openssl work properly
On MacOS, building and testing without openssl is much easier.
The tests should skip tests that fail because of missing openssl
instead of aborting.

Fixes https://github.com/rfjakob/gocryptfs/issues/123
2017-07-14 23:22:15 +02:00
Jakob Unterwurzacher
61e964457d stupidgcm: fix openssl 1.1 build failure
Fixed by including the correct header. Should work on older openssl
versions as well.

Error was:
locking.go:21: undefined reference to `CRYPTO_set_locking_callback'
2017-07-14 20:44:07 +02:00
Jakob Unterwurzacher
3062de6187 fusefronted: enable writing to write-only files
Due to RMW, we always need read permissions on the backing file. This is a
problem if the file permissions do not allow reading (i.e. 0200 permissions).
This patch works around that problem by chmod'ing the file, obtaining a fd,
and chmod'ing it back.

Test included.

Issue reported at: https://github.com/rfjakob/gocryptfs/issues/125
2017-07-11 23:19:58 +02:00
Jakob Unterwurzacher
b6bda01c33 contentenc: MergeBlocks: short-circuit the trivial case
Saves 3% for the tar extract benchmark because we skip the allocation.
2017-07-02 16:23:24 +02:00
Jakob Unterwurzacher
52ab0462a4 fusefrontend: doRead: skip decryption for an empty read
Previously we ran through the decryption steps even for an empty
ciphertext slice. The functions handle it correctly, but returning
early skips all the extra calls.

Speeds up the tar extract benchmark by about 4%.
2017-07-02 16:02:13 +02:00
Jakob Unterwurzacher
9f4bd76576 stupidgcm: add test for in-place Open
Adds a test for the optimization introduced in:

	stupidgcm: Open: if "dst" is big enough, use it as the output buffer
2017-07-01 09:56:05 +02:00
Jakob Unterwurzacher
12c0101a23 contentenc: add PReqPool and use it in DecryptBlocks
This gets us a massive speed boost in streaming reads.
2017-06-30 23:30:57 +02:00
Jakob Unterwurzacher
e4b5005bcc stupidgcm: Open: if "dst" is big enough, use it as the output buffer
This means we won't need any allocation for the plaintext.
2017-06-30 23:24:12 +02:00
Jakob Unterwurzacher
b2a23e94d1 fusefrontend: doRead: use CReqPool for ciphertext buffer
Easily saves lots of allocations.
2017-06-30 23:15:31 +02:00
Jakob Unterwurzacher
06398e82d9 fusefrontend: Read: use provided buffer
This will allow us to return internal buffers to a pool.
2017-06-30 23:11:38 +02:00
Jakob Unterwurzacher
80676c685f contentenc: add safer "bPool" pool variant; add pBlockPool
bPool verifies the lengths of slices going in and out.

Also, add a plaintext block pool - pBlockPool - and use
it for decryption.
2017-06-29 23:44:32 +02:00
Jakob Unterwurzacher
0cc6f53496 stupidgcm: use "dst" as the output buffer it is big enough
This saves an allocation of the ciphertext block.
2017-06-29 18:52:33 +02:00
Jakob Unterwurzacher
3c6fe98eb1 contentenc: use sync.Pool memory pools for encryption
We use two levels of buffers:

1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)

This commit adds a sync.Pool for both levels.

The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
2017-06-20 21:22:00 +02:00
Jakob Unterwurzacher
a4563e21ec main, syscallcompat: use Dup3 instead of Dup2
Dup2 is not implemented on linux/arm64.

Fixes https://github.com/rfjakob/gocryptfs/issues/121 .

Also adds cross-compilation to CI.
2017-06-18 15:43:22 +02:00
Jakob Unterwurzacher
e52594dae6 contentenc: parallelize encryption for 128kiB writes
128kiB = 32 x 4kiB pages is the maximum we get from the kernel. Splitting
up smaller writes is probably not worth it.

Parallelism is limited to two for now.
2017-06-11 21:56:16 +02:00
Jakob Unterwurzacher
9837cb0ddc cryptocore: prefetch nonces in the background
Spawn a worker goroutine that reads the next 512-byte block
while the current one is being drained.

This should help reduce waiting times when /dev/urandom is very
slow (like on Linux 3.16 kernels).
2017-06-11 21:29:50 +02:00
Jakob Unterwurzacher
80516ed335 cryptocore: prefetch nonces in 512-byte blocks
On my machine, reading 512-byte blocks from /dev/urandom
(same via getentropy syscall) is a lot faster in terms of
throughput:

Blocksize    Throughput
 16          28.18 MB/s
512          83.75 MB/s

For a single-threaded streaming write, this drops the CPU usage of
nonceGenerator.Get to almost 1/3:

        flat  flat%   sum%        cum   cum%
Before     0     0% 95.08%      0.35s  2.92%  github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get
After  0.01s 0.092% 92.34%      0.13s  1.20%  github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get

This change makes the nonce reading single-threaded, which may
hurt massively-parallel writes.
2017-06-09 22:05:14 +02:00