We use two levels of buffers:
1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)
This commit adds a sync.Pool for both levels.
The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
128kiB = 32 x 4kiB pages is the maximum we get from the kernel. Splitting
up smaller writes is probably not worth it.
Parallelism is limited to two for now.
Spawn a worker goroutine that reads the next 512-byte block
while the current one is being drained.
This should help reduce waiting times when /dev/urandom is very
slow (like on Linux 3.16 kernels).
On my machine, reading 512-byte blocks from /dev/urandom
(same via getentropy syscall) is a lot faster in terms of
throughput:
Blocksize Throughput
16 28.18 MB/s
512 83.75 MB/s
For a single-threaded streaming write, this drops the CPU usage of
nonceGenerator.Get to almost 1/3:
flat flat% sum% cum cum%
Before 0 0% 95.08% 0.35s 2.92% github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get
After 0.01s 0.092% 92.34% 0.13s 1.20% github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get
This change makes the nonce reading single-threaded, which may
hurt massively-parallel writes.
This check would need locking to be multithreading-safe.
But as it is in the fastpath, just remove it.
rand.Read() already guarantees that the value is random.
Travis failed on Go 1.6.3 with this error:
internal/pathiv/pathiv_test.go:20: no args in Error call
This change should solve the problem and provides a better error
message on (real) test failure.
With hard links, the path to a file is not unique. This means
that the ciphertext data depends on the path that is used to access
the files.
Fix that by storing the derived values when we encounter a hard-linked
file. This means that the first path wins.
This fixes a few issues I have found reviewing the code:
1) Limit the amount of data ReadLongName() will read. Previously,
you could send gocryptfs into out-of-memory by symlinking
gocryptfs.diriv to /dev/zero.
2) Handle the empty input case in unPad16() by returning an
error. Previously, it would panic with an out-of-bounds array
read. It is unclear to me if this could actually be triggered.
3) Reject empty names after base64-decoding in DecryptName().
An empty name crashes emeCipher.Decrypt().
It is unclear to me if B64.DecodeString() can actually return
a non-error empty result, but let's guard against it anyway.
When a user calls into a deep directory hierarchy, we often
get a sequence like this from the kernel:
LOOKUP a
LOOKUP a/b
LOOKUP a/b/c
LOOKUP a/b/c/d
The diriv cache was not effective for this pattern, because it
was designed for this:
LOOKUP a/a
LOOKUP a/b
LOOKUP a/c
LOOKUP a/d
By also using the cached entry of the grandparent we can avoid lots
of diriv reads.
This benchmark is against a large encrypted directory hosted on NFS:
Before:
$ time ls -R nfs-backed-mount > /dev/null
real 1m35.976s
user 0m0.248s
sys 0m0.281s
After:
$ time ls -R nfs-backed-mount > /dev/null
real 1m3.670s
user 0m0.217s
sys 0m0.403s
This commit defines all exit codes in one place in the exitcodes
package.
Also, it adds a test to verify the exit code on incorrect
password, which is what SiriKali cares about the most.
Fixes https://github.com/rfjakob/gocryptfs/issues/77 .
Misspell Finds commonly misspelled English words
gocryptfs/internal/configfile/scrypt.go
Line 41: warning: "paramter" is a misspelling of "parameter" (misspell)
gocryptfs/internal/ctlsock/ctlsock_serve.go
Line 1: warning: "implementes" is a misspelling of "implements" (misspell)
gocryptfs/tests/test_helpers/helpers.go
Line 27: warning: "compatability" is a misspelling of "compatibility" (misspell)
This can happen during normal operation, and is harmless since
14038a1644
"fusefrontend: readFileID: reject files that consist only of a header"
causes dormant header-only files to be rewritten on the next write.
We do not have to track the writeOnly status because the kernel
will not forward read requests on a write-only FD to us anyway.
I have verified this behavoir manually on a 4.10.8 kernel and also
added a testcase.
As reported at https://github.com/rfjakob/gocryptfs/issues/105 ,
the "ioutil.WriteFile(file, iv, 0400)" call causes "permissions denied"
errors on an NFSv4 setup.
"strace"ing diriv creation and gocryptfs.conf creation shows this:
conf (works on the user's NFSv4 mount):
openat(AT_FDCWD, "/tmp/a/gocryptfs.conf.tmp", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0400) = 3
diriv (fails):
openat(AT_FDCWD, "/tmp/a/gocryptfs.diriv", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0400) = 3
This patch creates the diriv file with the same flags that are used for
creating the conf:
openat(AT_FDCWD, "/tmp/a/gocryptfs.diriv", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0400) = 3
Closes https://github.com/rfjakob/gocryptfs/issues/105
Force decode of encrypted files even if the integrity check fails, instead of
failing with an IO error. Warning messages are still printed to syslog if corrupted
files are encountered.
It can be useful to recover files from disks with bad sectors or other corrupted
media.
Closes https://github.com/rfjakob/gocryptfs/pull/102 .
go-fuse has added a new method to the nodefs.File interface that
caused this build error:
internal/fusefrontend/file.go:75: cannot use file literal (type *file) as type nodefs.File in return argument:
*file does not implement nodefs.File (missing Flock method)
Fixes https://github.com/rfjakob/gocryptfs/issues/104 and
prevents the problem from happening again.
The volatile inode numbers that we used before cause "find" to complain and error out.
Virtual inode numbers are derived from their parent file inode number by adding 10^19,
which is hopefully large enough no never cause problems in practice.
If the backing directory contains inode numbers higher than that, stat() on these files
will return EOVERFLOW.
Example directory lising after this change:
$ ls -i
926473 gocryptfs.conf
1000000000000926466 gocryptfs.diriv
944878 gocryptfs.longname.hmZojMqC6ns47eyVxLlH2ailKjN9bxfosi3C-FR8mjA
1000000000000944878 gocryptfs.longname.hmZojMqC6ns47eyVxLlH2ailKjN9bxfosi3C-FR8mjA.name
934408 Tdfbf02CKsTaGVYnAsSypA
This PR addresses the Issue #95, about "Confusing file owner for
longname files in reverse mode".
It affects only the reverse mode, and introduces two
modifications:
1) The "gocryptfs.longname.XXXX.name" files are assigned the owner and
group of the underlying plaintext file. Therefore it is consistent
with the file "gocryptfs.longname.XXXX" that has the encrypted
contents of the plaintext file.
2) The two virtual files mentioned above are given -r--r--r--
permissions. This is consistent with the behavior described in
function Access in internal/fusefrontend_reverse/rfs.go where all
virtual files are always readable. Behavior also observed in point
c) in #95 .
Issue #95 URL: https://github.com/rfjakob/gocryptfs/issues/95
Pull request URL: https://github.com/rfjakob/gocryptfs/pull/97
Due to kernel readahead, we usually get multiple read requests
at the same time. These get submitted to the backing storage in
random order, which is a problem if seeking is very expensive.
Details: https://github.com/rfjakob/gocryptfs/issues/92
...if doWrite() can do it for us. This avoids the situation
that the file only consists of a file header when calling
doWrite.
A later patch will check for this condition and warn about it,
as with this change it should no longer occour in normal operation.
If you truncate a ciphertext file to 19 bytes, you could get the
impression that the plaintext is 18446744073709551585 bytes long,
as reported by "ls -l".
Fix it by clamping the value to zero.
Prior to this commit, gocryptfs's reverse mode did not report correct
directory entry sizes for symbolic links, where the dentry size needs to
be the same as the length of a string containing the target path.
This commit corrects this issue and adds a test case to verify the
correctness of the implementation.
This issue was discovered during the use of a strict file copying program
on a reverse-mounted gocryptfs file system.
Raw64 is supported (but was disabled by default) since gocryptfs
v1.2. However, the implementation was buggy because it forgot
about long names and symlinks.
Disable it for now by default and enable it later, together
with HKDF.
...but keep it disabled by default for new filesystems.
We are still missing an example filesystem and CLI arguments
to explicitely enable and disable it.
As we have dropped Go 1.4 compatibility already, and will add
a new feature flag for gocryptfs v1.3 anyway, this is a good
time to enable Raw64 as well.
There is no security reason for doing this, but it will allow
to consolidate the code once we drop compatibility with gocryptfs v1.2
(and earlier) filesystems.
There are two independent backends, one for name encryption,
the other one, AEAD, for file content.
"BackendTypeEnum" only applies to AEAD (file content), so make that
clear in the name.
Version 1.1 of the EME package (github.com/rfjakob/eme) added
a more convenient interface. Use it.
Note that you have to upgrade your EME package (go get -u)!
This really only handles scrypt and no other key-derivation functions.
Renaming the files prevents confusion once we introduce HKDF.
renamed: internal/configfile/kdf.go -> internal/configfile/scrypt.go
renamed: internal/configfile/kdf_test.go -> internal/configfile/scrypt_test.go
A crypto benchmark mode like "openssl speed".
Example run:
$ ./gocryptfs -speed
AES-GCM-256-OpenSSL 180.89 MB/s (selected in auto mode)
AES-GCM-256-Go 48.19 MB/s
AES-SIV-512-Go 37.40 MB/s
This used to hang at 100% CPU:
cat /dev/zero | gocryptfs -init a
...and would ultimately send the box into out-of-memory.
The number 1000 is chosen arbitrarily and seems big enough
given that the password must be one line.
Suggested by @mhogomchungu in https://github.com/rfjakob/gocryptfs/issues/77 .
From the comment:
// CheckTrailingGarbage tries to read one byte from stdin and exits with a
// fatal error if the read returns any data.
// This is meant to be called after reading the password, when there is no more
// data expected. This helps to catch problems with third-party tools that
// interface with gocryptfs.
Preallocation is very slow on hdds that run btrfs. Give the
user the option to disable it. This greatly speeds up small file
operations but reduces the robustness against out-of-space errors.
Also add the option to the man page.
More info: https://github.com/rfjakob/gocryptfs/issues/63
This could have caused spurious ENOENT errors.
That it did not cause these errors all the time is interesting
and probably because an earlier readdir would place the entry
in the cache. This masks the bug.
$ golint ./... | grep -v underscore | grep -v ALL_CAPS
internal/fusefrontend_reverse/rfs.go:52:36: exported func NewFS returns unexported type *fusefrontend_reverse.reverseFS, which can be annoying to use
internal/nametransform/raw64_go1.5.go:10:2: exported const HaveRaw64 should have comment (or a comment on this block) or be unexported
The Back In Time backup tool (https://github.com/bit-team/backintime)
wants to write directly into the ciphertext dir.
This may cause the cached directory IV to become out-of-date.
Having an expiry time limits the inconstency to one second, like
attr_timeout does for the kernel getattr cache.
Running xfstests generic/075 on tmpfs often triggered a panic
for what seems to be a tmpfs bug.
Quoting from the email to lkml,
http://www.spinics.net/lists/kernel/msg2370127.html :
tmpfs seems to be incorrectly returning 0-bytes when reading from
a file that is concurrently being truncated.
Stat() calls are expensive on NFS as they need a full network
round-trip. We detect when a write immediately follows the
last one and skip the Stat in this case because the write
cannot create a file hole.
On my (slow) NAS, this takes the write speed from 24MB/s to
41MB/s.
Test that we get the right timestamp when extracting a tarball.
Also simplify the workaround in doTestUtimesNano() and fix the
fact that it was running no test at all.
This can happen during normal operation when the directory has
been deleted concurrently. But it can also mean that the
gocryptfs.diriv is missing due to an error, so log the event
at "info" level.
AES-SIV uses 1/2 of the key for authentication, 1/2 for
encryption, so we need a 64-byte key for AES-256. Derive
it from the master key by hashing it with SHA-512.
On a CPU without AES-NI:
$ go test -bench .
Benchmark4kEncStupidGCM-2 50000 24155 ns/op 169.57 MB/s
Benchmark4kEncGoGCM-2 20000 93965 ns/op 43.59 MB/s
Benchmark4kEncGCMSIV-2 500 2576193 ns/op 1.59 MB/s
The last patch added functionality for generating gocryptfs.longname.*
files, this patch adds support for mapping them back to the full
filenames.
Note that resolving a long name needs a full readdir. A cache
will be implemented later on to improve performance.
As ReadDirIV operates on a path anyway, opening the directory
has no clear safety advantage w.r.t. concurrent renames.
If the backing directory is a reverse-mounted gocryptfs filesystem,
each directory open is an OPENDIR, and this causes a full directory
read!
This patch improves the "ls -lR" performance of an
DIR --> gocryptfs-reverse --> gocryptfs
chain by a factor of ~10.
OPENDIR counts for ls -lR:
Before 15570
After 2745
Commit af5441dcd9 has caused a
regression ( https://github.com/rfjakob/gocryptfs/issues/35 )
that is fixed by this commit.
The go-fuse library by now has all the syscall wrappers in
place to correctly handle Utimens, also for symlinks.
Instead of duplicating the effort here just call into go-fuse.
Closes#35
This fixes a build problem on 32-bit hosts:
internal/fusefrontend/file.go:400: cannot use a.Unix() (type int64) as
type int32 in assignment
internal/fusefrontend/file.go:406: cannot use m.Unix() (type int64) as
type int32 in assignment
It also enables full nanosecond timestamps for dates
after 1970.
[...]/stupidgcm/locking.go:16:2:
warning: indirection of non-volatile null pointer will
be deleted, not trap [-Wnull-dereference]
[...]/stupidgcm/locking.go:16:2:
note: consider using __builtin_trap() or qualifying
pointer with 'volatile'
https://github.com/rfjakob/gocryptfs/issues/15
...and convert all calls to syscall.{Fallocate,Openat}
to syscallcompat .
Both syscalls are not available on OSX. We emulate Openat and just
return EOPNOTSUPP for Fallocate.
unPad16 returns detailed errors including the position of the
incorrect bytes. Kill a possible padding oracle by lumping
everything into a generic error.
The detailed error is only logged if debug is active.
We were growing the file block-by-block which was pretty
inefficient. We now coalesce all the grows into a single
Ftruncate. Also simplifies the code!
Simplistic benchmark: Before:
$ time truncate -s 1000M foo
real 0m0.568s
After:
$ time truncate -s 1000M foo
real 0m0.205s
XFS returns a different error code if you try to overwrite
a non-empty directory with a directory:
XFS: mv: cannot move ‘foo’ to ‘bar/foo’: File exists
ext4: mv: cannot move 'foo' to 'bar/foo': Directory not empty
So have EEXIST trigger the Rmdir logic as well.
Fixes issue #20
Link: https://github.com/rfjakob/gocryptfs/issues/20
Drop the date and add the "go-fuse: " prefix so you can see
where the message is coming from.
Before:
Jun 27 09:03:15 brikett gocryptfs[4150]: 2016/06/27 09:03:15 Unimplemented opcode INTERRUPT
After:
Jun 27 09:10:58 brikett gocryptfs[4961]: go-fuse: Unimplemented opcode INTERRUPT
The "!fs.args.DirIV" special case was removed by b17f0465c7
but that, by accident, also removed the handling for
PlaintextNames.
Re-add it as an explicit PlaintextNames special case.
Also adds support for removing directories that miss their
gocryptfs.diriv file for some reason.
...unless "-nosyslog" is passed.
All gocryptfs messages already go to syslog, but the messages
that the go-fuse lib emits were still printed to stdout.
Fixes issue #13 ( https://github.com/rfjakob/gocryptfs/issues/13 )
FUSE filesystems are mounted with "nosuid" by default. If we run as root,
we can use device files by passing the opposite mount option, "suid".
Also we have to use syscall.Chmod instead of os.Chmod because the
portability translation layer "syscallMode" messes up the sgid
and suid bits.
Fixes 70% of the failures in xfstests generic/193. The remaining are
related to truncate, but we err on the safe side:
$ diff -u tests/generic/193.out /home/jakob/src/fuse-xfstests/results//generic/193.out.bad
[...]
check that suid/sgid bits are cleared after successful truncate...
with no exec perm
before: -rwSr-Sr--
-after: -rw-r-Sr--
+after: -rw-r--r--
Support truncate(2) by opening the file and calling ftruncate(2)
While the glibc "truncate" wrapper seems to always use ftruncate, fsstress from
xfstests uses this a lot by calling "truncate64" directly.
tlog is used heavily everywhere and deserves a shorter name.
Renamed using sed magic, without any manual rework:
find * -type f -exec sed -i 's/toggledlog/tlog/g' {} +
Warnings were:
main.go:234: declaration of err shadows declaration at main.go:163:
internal/fusefrontend/file.go:401: declaration of err shadows declaration at internal/fusefrontend/file.go:379:
internal/fusefrontend/file.go:419: declaration of err shadows declaration at internal/fusefrontend/file.go:379:
internal/fusefrontend/fs_dir.go:140: declaration of err shadows declaration at internal/fusefrontend/fs_dir.go:97:
If /proc/self/fd/X did not exist, the actual error is that the file
descriptor was invalid.
go-fuse's pathfs prefers using an open fd even for path-based operations
but does not take any locks to prevent the fd from being closed.
Instead, it retries the operation by path if it get EBADF. So this
change allows the retry logic to work correctly.
This fixes the error
rsync: failed to set times on "/tmp/ping.Kgw.mnt/linux-3.0/[...]/.dvb_demux.c.N7YlEM":
No such file or directory (2)
that was triggered by pingpong-rsync.bash.
We (actually, go-fuse) used to call Chown() instead of Lchown()
which meant that the operation would fail on dangling symlinks.
Fix this by calling os.Lchown() ourself. Also add a test case
for this.
Several fatal errors were just printed to stdout, which
meant they were invisible when running the test suite.
Fix this by introducing toggledlog.Fatal and convert as
follows:
Fatal errors -> toggledlog.Fatal
Warnings -> toggledlog.Warn
Password prompts -> fmt.Fprintf
This should make things saner and more extensible. It prepares
the infrastructure for "required feature flags" that will be used
to deprecate old gocryptfs version.
This field is added for the convenience of users and
may help them to identify which gocryptfs version
they need to mount a filesystem.
The same information is essentially contained in FeatureFlags,
but this is more difficult to decode for humans.
It is completely ignored programmatically (also by older gocryptfs
versions).
Just presenting an empty directory means that the user does not know
that things went wrong unless he checks the syslog or tries to delete
the directory.
It would be nice to report the error even if only some files were
invalid. However, go-fuse does not allow returning the valid
directory entries AND an error.
... with the "released" boolean.
For some reason, the "f.fd.Fd() < 0" check did not work reliably,
leading to nil pointer panics on the following wlock.lock().
The problem was discovered during fsstress testing and is unlikely
to happen in normal operations.
With this change, we passed 1700+ fsstress iterations.
Paths in statfs() calls were not encrypted resulting in
an Function not implemented error, when the unencrypted
path didn't exist in the underlying (encrypted)
filesystem.
$ df plain/existingdir
df: ‘plain/existingdir’: Function not implemented
Go GCM is faster than OpenSSL if the CPU has AES instructions
and you are running Go 1.6+.
The "-openssl" option now defaults to "auto".
"gocryptfs -debug -version" displays the result of the autodetection.
See https://github.com/rfjakob/gocryptfs/issues/23 for details and
benchmarks.
Commit 730291feab properly freed wlock when the file descriptor is
closed. However, concurrently running Write and Truncates may
still want to lock it. Check if the fd has been closed first.
In general, OpenSSL is only threadsafe if you provide a locking function
through CRYPTO_set_locking_callback. However, the GCM operations that
stupidgcm uses never call that function.
To guard against that ever changing, set a dummy locking callback
that crashes the app.
Quoting from the patch:
We compare against Go's built-in GCM implementation. Since stupidgcm only
supports 128-bit IVs and Go only supports that from 1.5 onward, we cannot
run these tests on older Go versions.
This fixes the test failures on Travis CI.
Quoting from 07a4ff79d2
/* Set expected tag value. A restriction in OpenSSL 1.0.1c and earlier
* required the tag before any AAD or ciphertext */
...complete with tests and benchmark.
This will allow us to get rid of the dependency to spacemonkeygo/openssl
that causes problems on Arch Linux
( https://github.com/rfjakob/gocryptfs/issues/21 )
