Commit Graph

862 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
c12a50d4c8 benchmarks: add streaming read benchmark 2017-06-27 00:04:58 +02:00
Jakob Unterwurzacher
b9b52854c3 profiling: add streaming read profiling helper
Reads 1GB of zeros while collecting memory and cpu profiles.
2017-06-24 15:52:24 +02:00
Jakob Unterwurzacher
5c7b5770ce Update performance numbers 2017-06-20 21:46:27 +02:00
Jakob Unterwurzacher
3c6fe98eb1 contentenc: use sync.Pool memory pools for encryption
We use two levels of buffers:

1) 4kiB+overhead for each ciphertext block
2) 128kiB+overhead for each FUSE write (32 ciphertext blocks)

This commit adds a sync.Pool for both levels.

The memory-efficiency for small writes could be improved,
as we now always use a 128kiB buffer.
2017-06-20 21:22:00 +02:00
Jakob Unterwurzacher
609343accf README: update changelog for v1.4 2017-06-20 19:56:38 +02:00
Jakob Unterwurzacher
bfe421b327 MANPAGE: reorder options to match "-hh" output; add "-hkdf", "-trace" 2017-06-20 19:49:18 +02:00
Jakob Unterwurzacher
f3965a4e4c build.bash: use plain "git describe" for go-fuse
go-fuse recently added a git tag - let's use it.
2017-06-20 18:59:48 +02:00
Jakob Unterwurzacher
c9c4bc0141 profiling: add streaming-write profiling helper
Writes 1GB of zeros to a gocryptfs mount while collecting
cpu and memory profiles.
2017-06-18 22:56:50 +02:00
Jakob Unterwurzacher
a4563e21ec main, syscallcompat: use Dup3 instead of Dup2
Dup2 is not implemented on linux/arm64.

Fixes https://github.com/rfjakob/gocryptfs/issues/121 .

Also adds cross-compilation to CI.
2017-06-18 15:43:22 +02:00
Jakob Unterwurzacher
afc3a8252b Add performance numbers for v1.3-69-ge52594d 2017-06-11 21:58:01 +02:00
Jakob Unterwurzacher
e52594dae6 contentenc: parallelize encryption for 128kiB writes
128kiB = 32 x 4kiB pages is the maximum we get from the kernel. Splitting
up smaller writes is probably not worth it.

Parallelism is limited to two for now.
2017-06-11 21:56:16 +02:00
Jakob Unterwurzacher
24a7b1b7b8 Add performance numbers for last change
Slight streaming write improvement.
2017-06-11 21:44:24 +02:00
Jakob Unterwurzacher
9837cb0ddc cryptocore: prefetch nonces in the background
Spawn a worker goroutine that reads the next 512-byte block
while the current one is being drained.

This should help reduce waiting times when /dev/urandom is very
slow (like on Linux 3.16 kernels).
2017-06-11 21:29:50 +02:00
Jakob Unterwurzacher
f351c3c1ec benchmark.bash: add dd-only mode, enable via "-dd"
Allows for quickly testing the streaming write throughput.
2017-06-11 12:05:59 +02:00
Jakob Unterwurzacher
b5358ea623 performance.txt: add numbers for latest change
Also, get rid of the half-empty line.
2017-06-09 22:13:23 +02:00
Jakob Unterwurzacher
80516ed335 cryptocore: prefetch nonces in 512-byte blocks
On my machine, reading 512-byte blocks from /dev/urandom
(same via getentropy syscall) is a lot faster in terms of
throughput:

Blocksize    Throughput
 16          28.18 MB/s
512          83.75 MB/s

For a single-threaded streaming write, this drops the CPU usage of
nonceGenerator.Get to almost 1/3:

        flat  flat%   sum%        cum   cum%
Before     0     0% 95.08%      0.35s  2.92%  github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get
After  0.01s 0.092% 92.34%      0.13s  1.20%  github.com/rfjakob/gocryptfs/internal/cryptocore.(*nonceGenerator).Get

This change makes the nonce reading single-threaded, which may
hurt massively-parallel writes.
2017-06-09 22:05:14 +02:00
Charles Duffy
da1bd74246 Fix missing Owner coercion for already-open files (#117) 2017-06-09 22:04:56 +02:00
Jakob Unterwurzacher
d2be22a07f cryptocore: remove lastNonce check
This check would need locking to be multithreading-safe.
But as it is in the fastpath, just remove it.
rand.Read() already guarantees that the value is random.
2017-06-07 23:08:43 +02:00
Jakob Unterwurzacher
294628b384 contentenc: move EncryptBlocks() loop into its own functions
This allows easy parallelization in the future.
2017-06-07 22:09:15 +02:00
Jakob Unterwurzacher
71978ec88a Add "-trace" flag (record execution trace)
Uses the runtime/trace functionality.

TODO: add to man page.
2017-06-07 22:09:06 +02:00
Jakob Unterwurzacher
22820bcd76 main: reorder force_owner flag parsing
No functional changes, just keeping the profiling-related flags
together.
2017-06-07 22:07:56 +02:00
Jakob Unterwurzacher
0ac5e44137 Add "package-static.bash" helper script
Creates a tar.gz with a static build of gocryptfs and the man page.
2017-06-04 19:25:08 +02:00
Jakob Unterwurzacher
d8d7c5c4fa Remove obsolete TODO.md
This file has been obsolete for some time,
TODOs are no longer tracked here but on github.
2017-06-04 19:04:23 +02:00
Jakob Unterwurzacher
b3b10273b5 golint: get rid of underscore warnings for good.
We got another warning for force_other:

	cli_args.go:26:45: don't use underscores in Go names; struct field force_owner should be forceOwner

Use a broader grep.
2017-06-01 22:19:46 +02:00
Jakob Unterwurzacher
a24faa3ba5 fusefrontend: write: consolidate and move encryption to contentenc
Collect all the plaintext and pass everything to contentenc in
one call.

This will allow easier parallization of the encryption.

https://github.com/rfjakob/gocryptfs/issues/116
2017-06-01 22:19:27 +02:00
Jakob Unterwurzacher
53b7c17261 Don't cap GOMAXPROCS at 4.
Before Go 1.5, GOMAXPROCS defaulted to 1, hence it made
sense to unconditionally increase it to 4.

But since Go 1.5, GOMAXPROCS defaults to the number of cores,
so don't keep it from increasing above 4.

Also, update the performance numbers.
2017-06-01 20:55:13 +02:00
Jakob Unterwurzacher
f44902aaae Fix two comments
One out-of-date and the other with a typo.
2017-06-01 18:53:57 +02:00
Jakob Unterwurzacher
f91ce0b004 main: increase max write size to maximum
Previously, it was at the go-fuse default of 64KiB. Getting
bigger writes should increase throughput somewhat.

Testing on tmpfs shows an improvement from 112MiB/s to 120MiB/s.
2017-06-01 18:52:02 +02:00
Charles Duffy
cf1ded5236 Implement force_owner option to display ownership as a specific user. 2017-06-01 00:26:17 +02:00
Jakob Unterwurzacher
fc2a5f5ab0 pathiv: fix test failure on Go 1.6
Travis failed on Go 1.6.3 with this error:

	internal/pathiv/pathiv_test.go:20: no args in Error call

This change should solve the problem and provides a better error
message on (real) test failure.
2017-05-31 08:21:36 +02:00
Jakob Unterwurzacher
1e598e96fc main: add "-info" option
Pretty-prints the config while stripping out sensitive
(and uninteresting) data

https://github.com/rfjakob/gocryptfs/issues/111
2017-05-30 19:01:32 +02:00
Jakob Unterwurzacher
ca5f469ea3 README: update changelog with short help text 2017-05-30 18:01:03 +02:00
Jakob Unterwurzacher
df2f4b1c40 main: add short help text
We have accumulated so many options over time that they
no longer fit on the screen.

Display only a useful subset of options to the user unless
they pass "-hh".
2017-05-30 17:59:13 +02:00
Jakob Unterwurzacher
9a217ce786 pathiv: move block IV algorithm into this package
This was implemented in fusefrontend_reverse, but we need it
in fusefrontend as well. Move the algorithm into pathiv.BlockIV().
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
e43eb36da3 tests: add v1.3-reverse example filesystem
We check the md5 sum of the encrypted version of a file to make sure we don't
accidentially change the ciphertext generation.
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
d202a456f5 pathiv: move derivedIVContainer into the package
...under the new name "FileIVs".

This will also be used by forward mode.
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
857507e8b1 fusefrontend_reverse: move pathiv to its own package
We will also need it in forward mode.
2017-05-30 17:04:46 +02:00
Jakob Unterwurzacher
4d2cc551cf package.bash: RHEL: strip brackets
We now convert "CentOS release 5.11 (Final)"
to             "CentOS_release_5.11_Final"

https://github.com/rfjakob/gocryptfs/issues/113
2017-05-29 08:42:01 +02:00
Jakob Unterwurzacher
d59e7da6a6 gocryptfs-xray: dumpmasterkey: disable "Reading password from stdin"
...and also exit with the proper exit code when we get an error.
2017-05-28 17:43:09 +02:00
Jakob Unterwurzacher
7f5ae5f843 gocryptfs-xray: add function to dump the master key
Fixes https://github.com/rfjakob/gocryptfs/issues/83
2017-05-28 17:11:50 +02:00
Jakob Unterwurzacher
c36a55e985 package.bash: make it work somewhat on RHEL
RHEL and CentOS do not have /etc/os-release yet. Read from
/etc/redhat-release instead.

Fixes https://github.com/rfjakob/gocryptfs/issues/113
2017-05-27 15:22:27 +02:00
Jakob Unterwurzacher
d6ef283c3f cryptocore: improve comments and add tests for hkdfDerive
These should make it easier to re-implement the key derivation
that was enabled with the "HKDF" feature flag.
2017-05-27 14:41:20 +02:00
Jakob Unterwurzacher
ce4aaf16d8 README: update changelog 2017-05-25 21:36:49 +02:00
Jakob Unterwurzacher
9ecf2d1a3f fusefrontend_reverse: store derived values for hard-linked files
With hard links, the path to a file is not unique. This means
that the ciphertext data depends on the path that is used to access
the files.

Fix that by storing the derived values when we encounter a hard-linked
file. This means that the first path wins.
2017-05-25 21:33:16 +02:00
Jakob Unterwurzacher
bfc8d47747 doc: add performance numbers for 1.3-27 2017-05-25 21:30:58 +02:00
Jakob Unterwurzacher
9a3f9350fe nametransform: reject all-zero dirIV
This should never happen in normal operation and is a sign of
data corruption. Catch it early.
2017-05-25 14:21:55 +02:00
Jakob Unterwurzacher
2ce269ec63 contenenc: reject all-zero file ID
This should never happen in normal operation and is a sign of
data corruption. Catch it early.
2017-05-25 14:20:27 +02:00
Jakob Unterwurzacher
c0e411f81d contentenc: better error reporting in ParseHeader
Log the message ourselves and return EINVAL.

Before:

	gocryptfs[26962]: go-fuse: can't convert error type: ParseHeader: invalid version: got 0, want 2

After:

	gocryptfs[617]: ParseHeader: invalid version: want 2, got 0. Returning EINVAL.
2017-05-25 14:18:44 +02:00
Jakob Unterwurzacher
70c16fde4c main: replace paniclog with logger(1)
Instead of redirecting stdout and stderr to /tmp/gocryptfs_paniclog,
where it is hard to find, redirect them to a newly spawned logger(1)
instance that forwards the messages to syslog.

See https://github.com/rfjakob/gocryptfs/issues/109 for an example
where the paniclog was lost due to a reboot.

Also, instead of closing stdin, redirect it to /dev/null, like most
daemons seem to do.
2017-05-24 19:57:45 +02:00
Jakob Unterwurzacher
e827763f2e nametransform: harden name decryption against invalid input
This fixes a few issues I have found reviewing the code:

1) Limit the amount of data ReadLongName() will read. Previously,
you could send gocryptfs into out-of-memory by symlinking
gocryptfs.diriv to /dev/zero.

2) Handle the empty input case in unPad16() by returning an
error. Previously, it would panic with an out-of-bounds array
read. It is unclear to me if this could actually be triggered.

3) Reject empty names after base64-decoding in DecryptName().
An empty name crashes emeCipher.Decrypt().
It is unclear to me if B64.DecodeString() can actually return
a non-error empty result, but let's guard against it anyway.
2017-05-23 21:26:38 +02:00