Commit Graph

1110 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
ed046aa359 Fix misspellings reported by goreportcard.com
https://goreportcard.com/report/github.com/rfjakob/gocryptfs#misspell
2017-08-21 21:06:05 +02:00
Jakob Unterwurzacher
26a6e61a46 MANPAGE: update exit codes
The exit codes have been documented in CLI_ABI.md for a while,
but they should also be listed in the man page.

Also fix the rendering of "[-o COMMA-SEPARATED-OPTIONS]", where
the square brackets where interpreted as something. Escape all
square brackets to be safe.
2017-08-21 20:53:25 +02:00
Jakob Unterwurzacher
a605e8fed0 README: update changelog for v1.4.1 release 2017-08-21 20:39:25 +02:00
Jakob Unterwurzacher
07f57314af package[-static].bash: stop leaking the local user id in the tarball
The local user id of the packager is not interesting for users who
download the tarball.

Also it will cause the gocryptfs binary to have an unintended owner
when the tarball is extraced as root.

Fix the issue by using "tar --owner=root --group=root" which
overwrites user and group id with zero.
2017-08-21 14:10:05 +02:00
Jakob Unterwurzacher
312ea32bb7 cryptocore: add urandom + randprefetch benchmarks
The benchmark that supported the decision for 512-byte
prefetching previously lived outside the repo.

Let's add it where it belongs so it cannot get lost.
2017-08-16 18:33:00 +02:00
Jakob Unterwurzacher
838bf883df Update performance.txt for Linux kernel upgrade
For some reason, writing became a lot faster in Linux 4.11
(scheduler improvements?).
2017-08-15 19:13:00 +02:00
Jakob Unterwurzacher
069647842c Update performance.txt for to Getdents change 2017-08-15 19:07:08 +02:00
Jakob Unterwurzacher
d5671b785a docs: label "ENV CHANGE" column 2017-08-15 19:04:02 +02:00
Jakob Unterwurzacher
989b880989 fusefrontend: use Getdents if available
Getdents avoids calling Lstat on each file.
2017-08-15 19:04:02 +02:00
Jakob Unterwurzacher
e50a6a57e5 syscallcompat: implement Getdents()
The Readdir function provided by os is inherently slow because
it calls Lstat on all files.

Getdents gives us all the information we need, but does not have
a proper wrapper in the stdlib.

Implement the "Getdents()" wrapper function that calls
syscall.Getdents() and parses the returned byte blob to a
fuse.DirEntry slice.
2017-08-15 19:03:57 +02:00
Jakob Unterwurzacher
affb1c2f66 main: rework "you need Go 1.5" lockout to make "go vet" happy
When you run "go vet" explicitely against go1.4.go, it ignores
the "+build !go1.5" tag and, of course, throws a syntax error:

  $ go vet go1.4.go
  can't load package: package main:
  go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"

Unfortunatey, this is how https://goreportcard.com/ seems to call
"go vet", and means we get 0% on the "go vet" test and see this
error:

  An error occurred while running this test (strconv.Atoi: parsing " go1.4.go": invalid syntax)

By reworking the logic to use a non-existant package we get an
uglier error

  $ GOROOT=/opt/go1.4.3 /opt/go1.4.3/bin/go build
  go1.4.go:7:8: cannot find package "You_need_Go_1.5_or_higher_to_compile_gocryptfs" in any of:
  	/opt/go1.4.3/src/You_need_Go_1.5_or_higher_to_compile_gocryptfs (from $GOROOT)
  	/home/jakob/go/src/You_need_Go_1.5_or_higher_to_compile_gocryptfs (from $GOPATH)
  profiling.go:6:2: cannot find package "runtime/trace" in any of:
  	/opt/go1.4.3/src/runtime/trace (from $GOROOT)
  	/home/jakob/go/src/runtime/trace (from $GOPATH)

but make "go vet" happy.
2017-08-15 12:52:40 +02:00
Jakob Unterwurzacher
0c520845f3 main: purge masterkey from memory as soon as possible
Remove the "Masterkey" field from fusefrontend.Args because it
should not be stored longer than neccessary. Instead pass the
masterkey as a separate argument to the filesystem initializers.

Then overwrite it with zeros immediately so we don't have
to wait for garbage collection.

Note that the crypto implementation still stores at least a
masterkey-derived value, so this change makes it harder, but not
impossible, to extract the encryption keys from memory.

Suggested at https://github.com/rfjakob/gocryptfs/issues/137
2017-08-11 19:02:26 +02:00
Jakob Unterwurzacher
f59479736b tests: add diriv cache race test
Passes.
2017-08-10 21:01:19 +02:00
Jakob Unterwurzacher
e80b5f2049 nametransform: extend diriv cache to 100 entries
* extend the diriv cache to 100 entries
* add special handling for the immutable root diriv

The better cache allows to shed some complexity from the path
encryption logic (parent-of-parent check).

Mitigates https://github.com/rfjakob/gocryptfs/issues/127
2017-08-09 22:00:53 +02:00
Jakob Unterwurzacher
75ec94a87a nametransform: add Dir() function
Dir is like filepath.Dir but returns "" instead of ".".
This was already implemented in fusefrontend_reverse as saneDir().

We will need it in nametransform for the improved diriv caching.
2017-08-06 23:14:39 +02:00
Jakob Unterwurzacher
5190cc09bb nametransform: move diriv cache into it's own package
Needs some space to grow.

renamed:    internal/nametransform/diriv_cache.go -> internal/nametransform/dirivcache/dirivcache.go
2017-08-06 21:59:15 +02:00
Jakob Unterwurzacher
32611ff97a nametransform: deduplicate code to encryptAndHashName()
This operation has been done three time by identical
sections of code. Create a function for it.
2017-08-06 21:23:42 +02:00
Jakob Unterwurzacher
e4fdb42496 build.bash: implement "you need Go 1.5" lockout in pure Go
As noticed by @riking, the logic in the bash script will break
when Go 1 version numbers reach double-digits.

Instead, use a build tag "!go1.5" to cause a syntax error:

  $ /opt/go1.4.3/bin/go build
  can't load package: package github.com/rfjakob/gocryptfs:
  go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"

Fixes https://github.com/rfjakob/gocryptfs/issues/133
2017-08-02 23:41:20 +02:00
Jakob Unterwurzacher
1f39ede4b4 main: save memory profile every 60 seconds
...and move all profiling functionality to its own file, as
the main function is already long enough.

Periodically saving the memory profile allows capturing the used
memory during normal operation, as opposed to on exit, where the
kernel has already issued FORGETs for all inodes.

This functionality has been used to create the memory profile shown
in https://github.com/rfjakob/gocryptfs/issues/132 .
2017-07-30 16:07:00 +02:00
Jakob Unterwurzacher
86253b7567 main: doMount: call FreeOSMemory() before jumping into server loop
scrypt (used during masterkey decryption) allocates a lot of memory.
Go only returns memory to the OS after 5 minutes, which looks like
a waste. Call FreeOSMemory() to return it immediately.

Looking a fresh mount:

before: VmRSS:	   73556 kB
after:  VmRSS:	    8568 kB
2017-07-29 17:17:12 +02:00
Jakob Unterwurzacher
d12aa57715 fusefronted_reverse: fix ino collision between .name and .diriv files
A directory with a long name has two associated virtual files:
the .name file and the .diriv files.

These used to get the same inode number:

  $ ls -di1  * */*
             33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name

With this change we use another prefix (2 instead of 1) for .name files.

  $ ls -di1 * */*
             33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
  2000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
2017-07-29 16:15:49 +02:00
Jakob Unterwurzacher
d5133ca5ac fusefrontend_reverse: return ENOENT for undecryptable names
This was working until DecryptName switched to returning
EBADMSG instead of EINVAL.

Add a test to catch the regression next time.
2017-07-27 20:31:22 +02:00
Jakob Unterwurzacher
496968e06c main: redirectStdFds: keep logger from holding stdout open
We passed our stdout and stderr to the new logger instance,
which makes sense to see any error message, but also means that
the fd is kept open even when we close it.

Fixes the new TestMountBackground test and
https://github.com/rfjakob/gocryptfs/issues/130 .
2017-07-24 00:05:39 +02:00
Jakob Unterwurzacher
318c41827c tests: check if we close stderr and stdout correctly on mount
Currently fails, as reported at
https://github.com/rfjakob/gocryptfs/issues/130 .
2017-07-24 00:05:29 +02:00
Jakob Unterwurzacher
0b6e3ce635 main: move redirectStdFds() to daemonize.go
This really is a part of daemonization.

No code changes.
2017-07-23 19:21:23 +02:00
Jakob Unterwurzacher
74e58ae8e6 tests: fsstress-gocryptfs.bash: sync up with EncFS
I have added a subset of fsstress-gocryptfs.bash to EncFS as
fsstress-encfs.sh, improving the code a bit.

This change forward-ports these improvements to
fsstress-gocryptfs.bash.
2017-07-21 23:34:44 +02:00
Jakob Unterwurzacher
ccf1a84e41 macos: make testing without openssl work properly
On MacOS, building and testing without openssl is much easier.
The tests should skip tests that fail because of missing openssl
instead of aborting.

Fixes https://github.com/rfjakob/gocryptfs/issues/123
2017-07-14 23:22:15 +02:00
Jakob Unterwurzacher
61e964457d stupidgcm: fix openssl 1.1 build failure
Fixed by including the correct header. Should work on older openssl
versions as well.

Error was:
locking.go:21: undefined reference to `CRYPTO_set_locking_callback'
2017-07-14 20:44:07 +02:00
Jakob Unterwurzacher
3062de6187 fusefronted: enable writing to write-only files
Due to RMW, we always need read permissions on the backing file. This is a
problem if the file permissions do not allow reading (i.e. 0200 permissions).
This patch works around that problem by chmod'ing the file, obtaining a fd,
and chmod'ing it back.

Test included.

Issue reported at: https://github.com/rfjakob/gocryptfs/issues/125
2017-07-11 23:19:58 +02:00
Jakob Unterwurzacher
849ec10081 macos: print load_osxfuse hint if fuse.NewServer fails
Currently neither gocryptfs nor go-fuse automatically call load_osxfuse
if the /dev/osxfuse* device(s) do not exist. At least tell the user
what to do.

See https://github.com/rfjakob/gocryptfs/issues/124 for user pain.
2017-07-10 23:33:41 +02:00
Jeff Kriske
9f8e19b856 Specify a volname for osxfuse
If I use gocryptfs cypher plain then the resulting volume
should be named 'plain' just as it would be on Linux.
2017-07-10 23:19:11 +02:00
Jakob Unterwurzacher
b6bda01c33 contentenc: MergeBlocks: short-circuit the trivial case
Saves 3% for the tar extract benchmark because we skip the allocation.
2017-07-02 16:23:24 +02:00
Jakob Unterwurzacher
52ab0462a4 fusefrontend: doRead: skip decryption for an empty read
Previously we ran through the decryption steps even for an empty
ciphertext slice. The functions handle it correctly, but returning
early skips all the extra calls.

Speeds up the tar extract benchmark by about 4%.
2017-07-02 16:02:13 +02:00
Jakob Unterwurzacher
ab787e18f0 README: update mac os x support status to "beta" 2017-07-01 11:51:02 +02:00
Jakob Unterwurzacher
b256c39271 profiling: add tar-extract helper
Extracts the linux-3.0.tar.gz tarball while capturing memory
and cpu profiles.
2017-07-01 11:42:52 +02:00
Jakob Unterwurzacher
52a8e14332 Travis CI: update to latest stable Go point releases 2017-07-01 11:39:33 +02:00
Jakob Unterwurzacher
9d10dcbd1c main: get rid of magic "MaxWrite: 1048576" constant
go-fuse caps MaxWrite at MAX_KERNEL_WRITE anyway, and we
actually depend on this behavoir now as the byte pools
are sized according to MAX_KERNEL_WRITE.

So let's use MAX_KERNEL_WRITE explicitely.
2017-07-01 11:32:17 +02:00
Jakob Unterwurzacher
b3e554acc2 Update performance.txt with new results
Massive speed boost for streaming reads.
2017-07-01 10:00:50 +02:00
Jakob Unterwurzacher
9f4bd76576 stupidgcm: add test for in-place Open
Adds a test for the optimization introduced in:

	stupidgcm: Open: if "dst" is big enough, use it as the output buffer
2017-07-01 09:56:05 +02:00
Jakob Unterwurzacher
12c0101a23 contentenc: add PReqPool and use it in DecryptBlocks
This gets us a massive speed boost in streaming reads.
2017-06-30 23:30:57 +02:00
Jakob Unterwurzacher
e4b5005bcc stupidgcm: Open: if "dst" is big enough, use it as the output buffer
This means we won't need any allocation for the plaintext.
2017-06-30 23:24:12 +02:00
Jakob Unterwurzacher
b2a23e94d1 fusefrontend: doRead: use CReqPool for ciphertext buffer
Easily saves lots of allocations.
2017-06-30 23:15:31 +02:00
Jakob Unterwurzacher
06398e82d9 fusefrontend: Read: use provided buffer
This will allow us to return internal buffers to a pool.
2017-06-30 23:11:38 +02:00
Jakob Unterwurzacher
2932a285aa Update performance.txt with new numbers 2017-06-29 23:45:39 +02:00
Jakob Unterwurzacher
80676c685f contentenc: add safer "bPool" pool variant; add pBlockPool
bPool verifies the lengths of slices going in and out.

Also, add a plaintext block pool - pBlockPool - and use
it for decryption.
2017-06-29 23:44:32 +02:00
Jakob Unterwurzacher
3d32bcd37b profiling: fix hardcoded path in hint 2017-06-29 19:20:34 +02:00
Jakob Unterwurzacher
dee88f3c4d Update performance.txt with new numbers 2017-06-29 19:00:16 +02:00
Jakob Unterwurzacher
0cc6f53496 stupidgcm: use "dst" as the output buffer it is big enough
This saves an allocation of the ciphertext block.
2017-06-29 18:52:33 +02:00
Jakob Unterwurzacher
c12a50d4c8 benchmarks: add streaming read benchmark 2017-06-27 00:04:58 +02:00
Jakob Unterwurzacher
b9b52854c3 profiling: add streaming read profiling helper
Reads 1GB of zeros while collecting memory and cpu profiles.
2017-06-24 15:52:24 +02:00