Commit Graph

279 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
80676c685f contentenc: add safer "bPool" pool variant; add pBlockPool
bPool verifies the lengths of slices going in and out.

Also, add a plaintext block pool - pBlockPool - and use
it for decryption.
2017-06-29 23:44:32 +02:00
Jakob Unterwurzacher
0cc6f53496 stupidgcm: use "dst" as the output buffer it is big enough
This saves an allocation of the ciphertext block.
2017-06-29 18:52:33 +02:00
Jakob Unterwurzacher
3c6fe98eb1 contentenc: use sync.Pool memory pools for encryption
We use two levels of buffers:

1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)

This commit adds a sync.Pool for both levels.

The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
2017-06-20 21:22:00 +02:00
Jakob Unterwurzacher
a4563e21ec main, syscallcompat: use Dup3 instead of Dup2
Dup2 is not implemented on linux/arm64.

Fixes https://github.com/rfjakob/gocryptfs/issues/121 .

Also adds cross-compilation to CI.
2017-06-18 15:43:22 +02:00
Jakob Unterwurzacher
e52594dae6 contentenc: parallelize encryption for 128kiB writes
128kiB = 32 x 4kiB pages is the maximum we get from the kernel. Splitting
up smaller writes is probably not worth it.

Parallelism is limited to two for now.
2017-06-11 21:56:16 +02:00
Jakob Unterwurzacher
9837cb0ddc cryptocore: prefetch nonces in the background
Spawn a worker goroutine that reads the next 512-byte block
while the current one is being drained.

This should help reduce waiting times when /dev/urandom is very
slow (like on Linux 3.16 kernels).
2017-06-11 21:29:50 +02:00
Jakob Unterwurzacher
80516ed335 cryptocore: prefetch nonces in 512-byte blocks
On my machine, reading 512-byte blocks from /dev/urandom
(same via getentropy syscall) is a lot faster in terms of
throughput:

Blocksize    Throughput
 16          28.18 MB/s
512          83.75 MB/s

For a single-threaded streaming write, this drops the CPU usage of
nonceGenerator.Get to almost 1/3:

        flat  flat%   sum%        cum   cum%
Before     0     0% 95.08%      0.35s  2.92%  github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get
After  0.01s 0.092% 92.34%      0.13s  1.20%  github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get

This change makes the nonce reading single-threaded, which may
hurt massively-parallel writes.
2017-06-09 22:05:14 +02:00
Charles Duffy
da1bd74246 Fix missing Owner coercion for already-open files (#117) 2017-06-09 22:04:56 +02:00
Jakob Unterwurzacher
d2be22a07f cryptocore: remove lastNonce check
This check would need locking to be multithreading-safe.
But as it is in the fastpath, just remove it.
rand.Read() already guarantees that the value is random.
2017-06-07 23:08:43 +02:00
Jakob Unterwurzacher
294628b384 contentenc: move EncryptBlocks() loop into its own functions
This allows easy parallelization in the future.
2017-06-07 22:09:15 +02:00
Jakob Unterwurzacher
71978ec88a Add "-trace" flag (record execution trace)
Uses the runtime/trace functionality.

TODO: add to man page.
2017-06-07 22:09:06 +02:00
Jakob Unterwurzacher
a24faa3ba5 fusefrontend: write: consolidate and move encryption to contentenc
Collect all the plaintext and pass everything to contentenc in
one call.

This will allow easier parallization of the encryption.

https://github.com/rfjakob/gocryptfs/issues/116
2017-06-01 22:19:27 +02:00
Jakob Unterwurzacher
f44902aaae Fix two comments
One out-of-date and the other with a typo.
2017-06-01 18:53:57 +02:00
Charles Duffy
cf1ded5236 Implement force_owner option to display ownership as a specific user. 2017-06-01 00:26:17 +02:00
Jakob Unterwurzacher
fc2a5f5ab0 pathiv: fix test failure on Go 1.6
Travis failed on Go 1.6.3 with this error:

	internal/pathiv/pathiv_test.go:20: no args in Error call

This change should solve the problem and provides a better error
message on (real) test failure.
2017-05-31 08:21:36 +02:00
Jakob Unterwurzacher
9a217ce786 pathiv: move block IV algorithm into this package
This was implemented in fusefrontend_reverse, but we need it
in fusefrontend as well. Move the algorithm into pathiv.BlockIV().
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
d202a456f5 pathiv: move derivedIVContainer into the package
...under the new name "FileIVs".

This will also be used by forward mode.
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
857507e8b1 fusefrontend_reverse: move pathiv to its own package
We will also need it in forward mode.
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
d6ef283c3f cryptocore: improve comments and add tests for hkdfDerive
These should make it easier to re-implement the key derivation
that was enabled with the "HKDF" feature flag.
2017-05-27 14:41:20 +02:00
Jakob Unterwurzacher
9ecf2d1a3f fusefrontend_reverse: store derived values for hard-linked files
With hard links, the path to a file is not unique. This means
that the ciphertext data depends on the path that is used to access
the files.

Fix that by storing the derived values when we encounter a hard-linked
file. This means that the first path wins.
2017-05-25 21:33:16 +02:00
Jakob Unterwurzacher
9a3f9350fe nametransform: reject all-zero dirIV
This should never happen in normal operation and is a sign of
data corruption. Catch it early.
2017-05-25 14:21:55 +02:00
Jakob Unterwurzacher
2ce269ec63 contenenc: reject all-zero file ID
This should never happen in normal operation and is a sign of
data corruption. Catch it early.
2017-05-25 14:20:27 +02:00
Jakob Unterwurzacher
c0e411f81d contentenc: better error reporting in ParseHeader
Log the message ourselves and return EINVAL.

Before:

	gocryptfs[26962]: go-fuse: can't convert error type: ParseHeader: invalid version: got 0, want 2

After:

	gocryptfs[617]: ParseHeader: invalid version: want 2, got 0. Returning EINVAL.
2017-05-25 14:18:44 +02:00
Jakob Unterwurzacher
e827763f2e nametransform: harden name decryption against invalid input
This fixes a few issues I have found reviewing the code:

1) Limit the amount of data ReadLongName() will read. Previously,
you could send gocryptfs into out-of-memory by symlinking
gocryptfs.diriv to /dev/zero.

2) Handle the empty input case in unPad16() by returning an
error. Previously, it would panic with an out-of-bounds array
read. It is unclear to me if this could actually be triggered.

3) Reject empty names after base64-decoding in DecryptName().
An empty name crashes emeCipher.Decrypt().
It is unclear to me if B64.DecodeString() can actually return
a non-error empty result, but let's guard against it anyway.
2017-05-23 21:26:38 +02:00
Jakob Unterwurzacher
508fd9e1d6 main: downgrade panic log create failure from fatal error to warning
Exiting with a fatal error just pushes users to use "-nosyslog",
which is even worse than not having a paniclog.
2017-05-23 18:01:21 +02:00
Jakob Unterwurzacher
245b84c887 nametransform: diriv cache: fall back to the grandparent
When a user calls into a deep directory hierarchy, we often
get a sequence like this from the kernel:

LOOKUP a
LOOKUP a/b
LOOKUP a/b/c
LOOKUP a/b/c/d

The diriv cache was not effective for this pattern, because it
was designed for this:

LOOKUP a/a
LOOKUP a/b
LOOKUP a/c
LOOKUP a/d

By also using the cached entry of the grandparent we can avoid lots
of diriv reads.

This benchmark is against a large encrypted directory hosted on NFS:

Before:

  $ time ls -R nfs-backed-mount > /dev/null
  real	1m35.976s
  user	0m0.248s
  sys	0m0.281s

After:

  $ time ls -R nfs-backed-mount > /dev/null
  real	1m3.670s
  user	0m0.217s
  sys 	0m0.403s
2017-05-22 22:36:54 +02:00
Jakob Unterwurzacher
c44389d942 exitcodes: specific codes for failure to read or write gocryptfs.conf
New codes:
* OpenConf = 23
* WriteConf = 24
2017-05-14 14:30:50 +02:00
Jakob Unterwurzacher
2aea2d3d62 exitcodes: add code 22 for "password is empty"
Empty passwords are not allowed. Let's give the error
it's own exit code.
2017-05-14 14:02:08 +02:00
Jakob Unterwurzacher
8aabc54276 exitcodes: get rid of generic "Mount" exit code
Instead, create three new specific exit codes:
* FuseNewServer = 19
* CtlSock = 20
* PanicLogCreate = 21
2017-05-14 13:51:26 +02:00
Jakob Unterwurzacher
d5adde1eeb exitcodes: pull all exit code definitions into the package
This commit defines all exit codes in one place in the exitcodes
package.

Also, it adds a test to verify the exit code on incorrect
password, which is what SiriKali cares about the most.

Fixes https://github.com/rfjakob/gocryptfs/issues/77 .
2017-05-07 22:16:22 +02:00
Jakob Unterwurzacher
ad7942f434 fusefrontend: implement path decryption via ctlsock
Closes https://github.com/rfjakob/gocryptfs/issues/84 .
2017-05-07 21:01:39 +02:00
Jakob Unterwurzacher
26881538e1 nametranform, fusefrontend: better errors on invalid names
nametransform.DecryptName() now always returns syscall.EBADMSG if
the name was invalid.

fusefrontend.OpenDir error messages have been normalized.
2017-05-07 20:58:27 +02:00
Jakob Unterwurzacher
68387b470c Fix typos found by Misspell
Misspell Finds commonly misspelled English words
gocryptfs/internal/configfile/scrypt.go
Line 41: warning: "paramter" is a misspelling of "parameter" (misspell)
gocryptfs/internal/ctlsock/ctlsock_serve.go
Line 1: warning: "implementes" is a misspelling of "implements" (misspell)
gocryptfs/tests/test_helpers/helpers.go
Line 27: warning: "compatability" is a misspelling of "compatibility" (misspell)
2017-05-07 12:22:15 +02:00
Jakob Unterwurzacher
c52e1abc58 fusefrontend: log "too many open files" errors
This usually indicates that the open file limit for gocryptfs is
too low. We should report this to the user.
2017-05-03 23:46:52 +02:00
Jakob Unterwurzacher
fb3cc6ea40 openfiletable: rename WriteLock to ContentLock
...and IDLock to HeaderLock. This matches what the locks actually
protect.
2017-05-01 21:57:18 +02:00
Jakob Unterwurzacher
f322ee87e3 fusefrontend: rely on nodefs.defaultFile for no-op functions
Now that we embed nodefs.NewDefaultFile(), we can drop our own
no-ops.
2017-05-01 19:12:37 +02:00
Jakob Unterwurzacher
1a89919d80 contentenc: downgrade "interrupted write?" warning to debug
This can happen during normal operation, and is harmless since

14038a1644
"fusefrontend: readFileID: reject files that consist only of a header"

causes dormant header-only files to be rewritten on the next write.
2017-05-01 18:44:18 +02:00
Jakob Unterwurzacher
9ab11aa4d7 fusefrontend: drop writeOnly flag
We do not have to track the writeOnly status because the kernel
will not forward read requests on a write-only FD to us anyway.

I have verified this behavoir manually on a 4.10.8 kernel and also
added a testcase.
2017-05-01 17:49:37 +02:00
Jakob Unterwurzacher
514f515dd7 fusefronted, openfiletable: move the open file table to its own package
The open file table code needs some room to grow for the upcoming
FD multiplexing implementation.
2017-05-01 17:26:50 +02:00
Jakob Unterwurzacher
863c3ca36f fusefrontend: rename write_lock.go -> open_file_table.go
The data structure was originally called write lock table, but
is now simply called the open file table. Rename the file to
reflect that.
2017-04-29 22:24:38 +02:00
Jakob Unterwurzacher
b66e03486a fusefronted: drop unused file.String() function
This is a very old leftover.
2017-04-29 18:20:39 +02:00
Jakob Unterwurzacher
6e029a3799 readpassword: increase max password size to 2048
This is the value EncFS uses, so let's follow suit.
Suggested at https://github.com/rfjakob/gocryptfs/issues/77 .
2017-04-29 15:15:11 +02:00
Jakob Unterwurzacher
edb3e19cb5 fix golint complaints 2017-04-29 14:50:58 +02:00
Jakob Unterwurzacher
7d38f80a78 nametransform: WriteDirIV: replace ioutil.WriteFile
As reported at https://github.com/rfjakob/gocryptfs/issues/105 ,
the "ioutil.WriteFile(file, iv, 0400)" call causes "permissions denied"
errors on an NFSv4 setup.

"strace"ing diriv creation and gocryptfs.conf creation shows this:

conf (works on the user's NFSv4 mount):
openat(AT_FDCWD, "/tmp/a/gocryptfs.conf.tmp", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0400) = 3

diriv (fails):
openat(AT_FDCWD, "/tmp/a/gocryptfs.diriv", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0400) = 3

This patch creates the diriv file with the same flags that are used for
creating the conf:
openat(AT_FDCWD, "/tmp/a/gocryptfs.diriv", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0400) = 3

Closes https://github.com/rfjakob/gocryptfs/issues/105
2017-04-29 14:15:13 +02:00
Jakob Unterwurzacher
3409ade272 forcedecode: tighten checks
...and fix a few golint issues and print a scary warning message on mount.

Also, force the fs to ro,noexec.
2017-04-24 00:25:02 +02:00
danim7
f1945c4daa Add -forcedecode
Force decode of encrypted files even if the integrity check fails, instead of
failing with an IO error. Warning messages are still printed to syslog if corrupted
files are encountered.
It can be useful to recover files from disks with bad sectors or other corrupted
media.

Closes https://github.com/rfjakob/gocryptfs/pull/102 .
2017-04-23 23:11:56 +02:00
Jakob Unterwurzacher
9777e4bf7e Fix Flock build breakage
go-fuse has added a new method to the nodefs.File interface that
caused this build error:

  internal/fusefrontend/file.go:75: cannot use file literal (type *file) as type nodefs.File in return argument:
  	*file does not implement nodefs.File (missing Flock method)

Fixes https://github.com/rfjakob/gocryptfs/issues/104 and
prevents the problem from happening again.
2017-04-23 00:06:56 +02:00
Jakob Unterwurzacher
778c955eea fusefrontend_reverse: switch to stable inode numbers
The volatile inode numbers that we used before cause "find" to complain and error out.
Virtual inode numbers are derived from their parent file inode number by adding 10^19,
which is hopefully large enough no never cause problems in practice.

If the backing directory contains inode numbers higher than that, stat() on these files
will return EOVERFLOW.

Example directory lising after this change:

  $ ls -i
               926473 gocryptfs.conf
  1000000000000926466 gocryptfs.diriv
               944878 gocryptfs.longname.hmZojMqC6ns47eyVxLlH2ailKjN9bxfosi3C-FR8mjA
  1000000000000944878 gocryptfs.longname.hmZojMqC6ns47eyVxLlH2ailKjN9bxfosi3C-FR8mjA.name
               934408 Tdfbf02CKsTaGVYnAsSypA
2017-04-01 17:19:15 +02:00
Jakob Unterwurzacher
e87aebb835 fusefrontend_reverse: drop unused dirIVAttr function
This has long been replaced by virtualFile.GetAttr().
2017-04-01 17:05:55 +02:00
Jakob Unterwurzacher
acb73ca436 fusefrontend_reverse: convert fmt.Printf calls to tlog
The fmt.Printfs output would end up in the paniclog.
2017-04-01 15:49:53 +02:00