Commit Graph

897 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
e80b5f2049 nametransform: extend diriv cache to 100 entries
* extend the diriv cache to 100 entries
* add special handling for the immutable root diriv

The better cache allows to shed some complexity from the path
encryption logic (parent-of-parent check).

Mitigates https://github.com/rfjakob/gocryptfs/issues/127
2017-08-09 22:00:53 +02:00
Jakob Unterwurzacher
75ec94a87a nametransform: add Dir() function
Dir is like filepath.Dir but returns "" instead of ".".
This was already implemented in fusefrontend_reverse as saneDir().

We will need it in nametransform for the improved diriv caching.
2017-08-06 23:14:39 +02:00
Jakob Unterwurzacher
5190cc09bb nametransform: move diriv cache into it's own package
Needs some space to grow.

renamed:    internal/nametransform/diriv_cache.go -> internal/nametransform/dirivcache/dirivcache.go
2017-08-06 21:59:15 +02:00
Jakob Unterwurzacher
32611ff97a nametransform: deduplicate code to encryptAndHashName()
This operation has been done three time by identical
sections of code. Create a function for it.
2017-08-06 21:23:42 +02:00
Jakob Unterwurzacher
e4fdb42496 build.bash: implement "you need Go 1.5" lockout in pure Go
As noticed by @riking, the logic in the bash script will break
when Go 1 version numbers reach double-digits.

Instead, use a build tag "!go1.5" to cause a syntax error:

  $ /opt/go1.4.3/bin/go build
  can't load package: package github.com/rfjakob/gocryptfs:
  go1.4.go:5:1: expected 'package', found 'STRING' "You need Go 1.5 or higher to compile gocryptfs!"

Fixes https://github.com/rfjakob/gocryptfs/issues/133
2017-08-02 23:41:20 +02:00
Jakob Unterwurzacher
1f39ede4b4 main: save memory profile every 60 seconds
...and move all profiling functionality to its own file, as
the main function is already long enough.

Periodically saving the memory profile allows capturing the used
memory during normal operation, as opposed to on exit, where the
kernel has already issued FORGETs for all inodes.

This functionality has been used to create the memory profile shown
in https://github.com/rfjakob/gocryptfs/issues/132 .
2017-07-30 16:07:00 +02:00
Jakob Unterwurzacher
86253b7567 main: doMount: call FreeOSMemory() before jumping into server loop
scrypt (used during masterkey decryption) allocates a lot of memory.
Go only returns memory to the OS after 5 minutes, which looks like
a waste. Call FreeOSMemory() to return it immediately.

Looking a fresh mount:

before: VmRSS:	   73556 kB
after:  VmRSS:	    8568 kB
2017-07-29 17:17:12 +02:00
Jakob Unterwurzacher
d12aa57715 fusefronted_reverse: fix ino collision between .name and .diriv files
A directory with a long name has two associated virtual files:
the .name file and the .diriv files.

These used to get the same inode number:

  $ ls -di1  * */*
             33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name

With this change we use another prefix (2 instead of 1) for .name files.

  $ ls -di1 * */*
             33313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw
  1000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw/gocryptfs.diriv
  2000000000033313535 gocryptfs.longname.2togDFouca9mrTwtfF1RNW5DZRAQY8alaR7wO_Xd5Zw.name
2017-07-29 16:15:49 +02:00
Jakob Unterwurzacher
d5133ca5ac fusefrontend_reverse: return ENOENT for undecryptable names
This was working until DecryptName switched to returning
EBADMSG instead of EINVAL.

Add a test to catch the regression next time.
2017-07-27 20:31:22 +02:00
Jakob Unterwurzacher
496968e06c main: redirectStdFds: keep logger from holding stdout open
We passed our stdout and stderr to the new logger instance,
which makes sense to see any error message, but also means that
the fd is kept open even when we close it.

Fixes the new TestMountBackground test and
https://github.com/rfjakob/gocryptfs/issues/130 .
2017-07-24 00:05:39 +02:00
Jakob Unterwurzacher
318c41827c tests: check if we close stderr and stdout correctly on mount
Currently fails, as reported at
https://github.com/rfjakob/gocryptfs/issues/130 .
2017-07-24 00:05:29 +02:00
Jakob Unterwurzacher
0b6e3ce635 main: move redirectStdFds() to daemonize.go
This really is a part of daemonization.

No code changes.
2017-07-23 19:21:23 +02:00
Jakob Unterwurzacher
74e58ae8e6 tests: fsstress-gocryptfs.bash: sync up with EncFS
I have added a subset of fsstress-gocryptfs.bash to EncFS as
fsstress-encfs.sh, improving the code a bit.

This change forward-ports these improvements to
fsstress-gocryptfs.bash.
2017-07-21 23:34:44 +02:00
Jakob Unterwurzacher
ccf1a84e41 macos: make testing without openssl work properly
On MacOS, building and testing without openssl is much easier.
The tests should skip tests that fail because of missing openssl
instead of aborting.

Fixes https://github.com/rfjakob/gocryptfs/issues/123
2017-07-14 23:22:15 +02:00
Jakob Unterwurzacher
61e964457d stupidgcm: fix openssl 1.1 build failure
Fixed by including the correct header. Should work on older openssl
versions as well.

Error was:
locking.go:21: undefined reference to `CRYPTO_set_locking_callback'
2017-07-14 20:44:07 +02:00
Jakob Unterwurzacher
3062de6187 fusefronted: enable writing to write-only files
Due to RMW, we always need read permissions on the backing file. This is a
problem if the file permissions do not allow reading (i.e. 0200 permissions).
This patch works around that problem by chmod'ing the file, obtaining a fd,
and chmod'ing it back.

Test included.

Issue reported at: https://github.com/rfjakob/gocryptfs/issues/125
2017-07-11 23:19:58 +02:00
Jakob Unterwurzacher
849ec10081 macos: print load_osxfuse hint if fuse.NewServer fails
Currently neither gocryptfs nor go-fuse automatically call load_osxfuse
if the /dev/osxfuse* device(s) do not exist. At least tell the user
what to do.

See https://github.com/rfjakob/gocryptfs/issues/124 for user pain.
2017-07-10 23:33:41 +02:00
Jeff Kriske
9f8e19b856 Specify a volname for osxfuse
If I use gocryptfs cypher plain then the resulting volume
should be named 'plain' just as it would be on Linux.
2017-07-10 23:19:11 +02:00
Jakob Unterwurzacher
b6bda01c33 contentenc: MergeBlocks: short-circuit the trivial case
Saves 3% for the tar extract benchmark because we skip the allocation.
2017-07-02 16:23:24 +02:00
Jakob Unterwurzacher
52ab0462a4 fusefrontend: doRead: skip decryption for an empty read
Previously we ran through the decryption steps even for an empty
ciphertext slice. The functions handle it correctly, but returning
early skips all the extra calls.

Speeds up the tar extract benchmark by about 4%.
2017-07-02 16:02:13 +02:00
Jakob Unterwurzacher
ab787e18f0 README: update mac os x support status to "beta" 2017-07-01 11:51:02 +02:00
Jakob Unterwurzacher
b256c39271 profiling: add tar-extract helper
Extracts the linux-3.0.tar.gz tarball while capturing memory
and cpu profiles.
2017-07-01 11:42:52 +02:00
Jakob Unterwurzacher
52a8e14332 Travis CI: update to latest stable Go point releases 2017-07-01 11:39:33 +02:00
Jakob Unterwurzacher
9d10dcbd1c main: get rid of magic "MaxWrite: 1048576" constant
go-fuse caps MaxWrite at MAX_KERNEL_WRITE anyway, and we
actually depend on this behavoir now as the byte pools
are sized according to MAX_KERNEL_WRITE.

So let's use MAX_KERNEL_WRITE explicitely.
2017-07-01 11:32:17 +02:00
Jakob Unterwurzacher
b3e554acc2 Update performance.txt with new results
Massive speed boost for streaming reads.
2017-07-01 10:00:50 +02:00
Jakob Unterwurzacher
9f4bd76576 stupidgcm: add test for in-place Open
Adds a test for the optimization introduced in:

	stupidgcm: Open: if "dst" is big enough, use it as the output buffer
2017-07-01 09:56:05 +02:00
Jakob Unterwurzacher
12c0101a23 contentenc: add PReqPool and use it in DecryptBlocks
This gets us a massive speed boost in streaming reads.
2017-06-30 23:30:57 +02:00
Jakob Unterwurzacher
e4b5005bcc stupidgcm: Open: if "dst" is big enough, use it as the output buffer
This means we won't need any allocation for the plaintext.
2017-06-30 23:24:12 +02:00
Jakob Unterwurzacher
b2a23e94d1 fusefrontend: doRead: use CReqPool for ciphertext buffer
Easily saves lots of allocations.
2017-06-30 23:15:31 +02:00
Jakob Unterwurzacher
06398e82d9 fusefrontend: Read: use provided buffer
This will allow us to return internal buffers to a pool.
2017-06-30 23:11:38 +02:00
Jakob Unterwurzacher
2932a285aa Update performance.txt with new numbers 2017-06-29 23:45:39 +02:00
Jakob Unterwurzacher
80676c685f contentenc: add safer "bPool" pool variant; add pBlockPool
bPool verifies the lengths of slices going in and out.

Also, add a plaintext block pool - pBlockPool - and use
it for decryption.
2017-06-29 23:44:32 +02:00
Jakob Unterwurzacher
3d32bcd37b profiling: fix hardcoded path in hint 2017-06-29 19:20:34 +02:00
Jakob Unterwurzacher
dee88f3c4d Update performance.txt with new numbers 2017-06-29 19:00:16 +02:00
Jakob Unterwurzacher
0cc6f53496 stupidgcm: use "dst" as the output buffer it is big enough
This saves an allocation of the ciphertext block.
2017-06-29 18:52:33 +02:00
Jakob Unterwurzacher
c12a50d4c8 benchmarks: add streaming read benchmark 2017-06-27 00:04:58 +02:00
Jakob Unterwurzacher
b9b52854c3 profiling: add streaming read profiling helper
Reads 1GB of zeros while collecting memory and cpu profiles.
2017-06-24 15:52:24 +02:00
Jakob Unterwurzacher
5c7b5770ce Update performance numbers 2017-06-20 21:46:27 +02:00
Jakob Unterwurzacher
3c6fe98eb1 contentenc: use sync.Pool memory pools for encryption
We use two levels of buffers:

1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)

This commit adds a sync.Pool for both levels.

The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
2017-06-20 21:22:00 +02:00
Jakob Unterwurzacher
609343accf README: update changelog for v1.4 2017-06-20 19:56:38 +02:00
Jakob Unterwurzacher
bfe421b327 MANPAGE: reorder options to match "-hh" output; add "-hkdf", "-trace" 2017-06-20 19:49:18 +02:00
Jakob Unterwurzacher
f3965a4e4c build.bash: use plain "git describe" for go-fuse
go-fuse recently added a git tag - let's use it.
2017-06-20 18:59:48 +02:00
Jakob Unterwurzacher
c9c4bc0141 profiling: add streaming-write profiling helper
Writes 1GB of zeros to a gocryptfs mount while collecting
cpu and memory profiles.
2017-06-18 22:56:50 +02:00
Jakob Unterwurzacher
a4563e21ec main, syscallcompat: use Dup3 instead of Dup2
Dup2 is not implemented on linux/arm64.

Fixes https://github.com/rfjakob/gocryptfs/issues/121 .

Also adds cross-compilation to CI.
2017-06-18 15:43:22 +02:00
Jakob Unterwurzacher
afc3a8252b Add performance numbers for v1.3-69-ge52594d 2017-06-11 21:58:01 +02:00
Jakob Unterwurzacher
e52594dae6 contentenc: parallelize encryption for 128kiB writes
128kiB = 32 x 4kiB pages is the maximum we get from the kernel. Splitting
up smaller writes is probably not worth it.

Parallelism is limited to two for now.
2017-06-11 21:56:16 +02:00
Jakob Unterwurzacher
24a7b1b7b8 Add performance numbers for last change
Slight streaming write improvement.
2017-06-11 21:44:24 +02:00
Jakob Unterwurzacher
9837cb0ddc cryptocore: prefetch nonces in the background
Spawn a worker goroutine that reads the next 512-byte block
while the current one is being drained.

This should help reduce waiting times when /dev/urandom is very
slow (like on Linux 3.16 kernels).
2017-06-11 21:29:50 +02:00
Jakob Unterwurzacher
f351c3c1ec benchmark.bash: add dd-only mode, enable via "-dd"
Allows for quickly testing the streaming write throughput.
2017-06-11 12:05:59 +02:00
Jakob Unterwurzacher
b5358ea623 performance.txt: add numbers for latest change
Also, get rid of the half-empty line.
2017-06-09 22:13:23 +02:00