Commit Graph

1200 Commits

Author SHA1 Message Date
Jakob Unterwurzacher 930c37e03d syscallcompat: use O_PATH in OpenDirNofollow
This fixes the "0100 directory" problem in reverse mode,
and should be slightly faster.
2018-09-08 18:06:33 +02:00
Jakob Unterwurzacher 9ec9d0c49c syscallcompat: untangle OpenNofollow and rename to OpenDirNofollow
The function used to do two things:

1) Walk the directory tree in a manner safe from symlink attacks
2) Open the final component in the mode requested by the caller

This change drops (2), which was only used once, and lets the caller
handle it. This simplifies the function and makes it fit for reuse in
forward mode in openBackingPath(), and for using O_PATH on Linux.
2018-09-08 17:41:17 +02:00
Jakob Unterwurzacher bc14f8dcb6 tests: add chmod test
Makes sure we don't add regressions when fixing
https://github.com/rfjakob/gocryptfs/issues/259
2018-09-08 17:00:23 +02:00
Jakob Unterwurzacher 21b5fae0e6 tests: restore TestUtimesNanoSymlink
The function seems to have been renamed by 426b9536 by mistake.
Rename it back so the test is run again.
2018-09-08 16:36:01 +02:00
Jakob Unterwurzacher ca24c20694 main: don't read the config file twice (fix pipe bug)
Instead, first Load() the file, then DecryptMasterKey().

Fixes https://github.com/rfjakob/gocryptfs/issues/258
2018-09-08 13:04:33 +02:00
Jakob Unterwurzacher 2bdf7d5172 configfile: add LoadAndDecrypt wrapper
Callers that do not want to decrypt the masterkey should
call plain Load().

https://github.com/rfjakob/gocryptfs/issues/258
2018-09-08 12:40:29 +02:00
Jakob Unterwurzacher 09d28c293e configfile: split off masterkey decryption
Preparation for fixing https://github.com/rfjakob/gocryptfs/issues/258
2018-09-08 12:19:19 +02:00
Jakob Unterwurzacher 21eaa8f164 configfile: return specific error on empty input
Report the actual problem instead of a generic
"unexpected end of JSON input".

https://github.com/rfjakob/gocryptfs/issues/258
2018-09-08 12:18:26 +02:00
Jakob Unterwurzacher 22906cc5fa travis: add Go 1.11
Released two days ago
2018-08-26 13:04:18 +02:00
Jakob Unterwurzacher 658cc4aebb syscallcompat: drop Fchmodat flags
These were silently ignored until now (!) but
are rejected by Go 1.11 stdlib.

Drop the flags so the tests work again, until
we figure out a better solution.

https://github.com/golang/go/issues/20130
2018-08-26 13:04:01 +02:00
Jakob Unterwurzacher 91dc44c8b4 travis: show fs type for /tmp 2018-08-26 12:39:51 +02:00
Jakob Unterwurzacher 39f39cd63c README: update benchmark output 2018-08-18 18:38:14 +02:00
Jakob Unterwurzacher 21f32b1ba7 Update reverse benchmarks 2018-08-18 12:25:03 +02:00
Jakob Unterwurzacher 30ba8140f2 Update README for v1.6 release 2018-08-18 12:19:47 +02:00
Jakob Unterwurzacher f2a0c6566e README: list trezor support in changelog 2018-08-15 23:44:46 +02:00
Jakob Unterwurzacher 7e111ea94c trezor: document -trezor in man page
Also fix "--", which was shown as just "-" in the man pager.
2018-08-15 23:34:23 +02:00
Jakob Unterwurzacher bd054e70ef trezor: show support in version string
Show enable_trezor in the version string if we were compiled
with `-tags enable_trezor`. And hide the `-trezor` flag from
the help output if we were not.
2018-08-15 23:31:37 +02:00
Jakob Unterwurzacher 7771a33f65 dep: update dependencies 2018-08-15 23:24:03 +02:00
Jakob Unterwurzacher 182f4833ee build.bash: handle args containing spaces
Fixes ./build.bash -tags "enable_trezor without_openssl"
2018-08-15 23:17:44 +02:00
Jakob Unterwurzacher dbd400d930 fusefrontend: truncateGrowFile: pass zeroPad error to caller
Errors from zeroPad were ignored until now, as discovered
using xfstests generic/083.
2018-08-15 17:25:22 +02:00
Jakob Unterwurzacher 0e1cbb75fe fsck: skip already-seen hard-linked files
We may hit files several times due to hard links.
Only check the contents once.
2018-08-15 14:02:05 +02:00
Jakob Unterwurzacher db4accf3a3 fsck: don't print corse corruption offset
The exact ciphertext block number (4KiB granularity) is
already printed in the doRead message. Don't cause
confusion by printing the 128KiB-granularity offset as
well.

  doRead 767: corrupt block #4: stupidgcm: message authentication failed
  fsck: error reading file "pa/d7/d14/f10c" (inum 767): 5=input/output error
2018-08-15 13:20:40 +02:00
Jakob Unterwurzacher 5acfbc1b2f main: add -e as an alias for -exclude 2018-08-15 13:12:13 +02:00
Jakob Unterwurzacher 7a02f71fc2 fusefrontend_reverse: reject excludes for the root directory ""
This is most likely a mistake by the user. Reject it.
2018-08-15 12:28:29 +02:00
Jakob Unterwurzacher 8989905333 Update changelog 2018-08-11 23:39:49 +02:00
Jakob Unterwurzacher ec2fdc19cf reverse mode: add --exclude option
https://github.com/rfjakob/gocryptfs/issues/235
2018-08-11 23:26:49 +02:00
Jakob Unterwurzacher eaa5aecd42 cli: add multipleStrings type
Will be used for --exclude.
2018-08-11 23:25:17 +02:00
Jakob Unterwurzacher 06f1ea951b tests: VerifyExistence: panic on inconsistent results
In the past we did not check whether all methods of checking
the file return the same result. We should.
2018-08-11 22:37:22 +02:00
Jakob Unterwurzacher f48b731e76 travis ci: drop go 1.5 and go 1.6
Trying to build with these versions now throws this error:

  # golang.org/x/sys/unix
  ../../../golang.org/x/sys/unix/ioctl.go:18: undefined: runtime.KeepAlive

It looks like x/sys/unix has dropped support for older Go
versions.
2018-08-02 20:06:34 +02:00
Jakob Unterwurzacher d620054caf fsck: print inode number on file read error
This makes it possible to find the file without mounting the fs.
2018-07-23 22:34:41 +02:00
Jakob Unterwurzacher f4a972ddf1 configfile: drop superflous Printf
Before:

  $ gocryptfs -fsck .
  LoadConfFile: ReadFile: &os.PathError{Op:"open", Path:"/var/tmp/check-gocryptfs/scratchdev/gocryptfs.conf", Err:0xd}
  Cannot open config file: open /var/tmp/check-gocryptfs/scratchdev/gocryptfs.conf: permission denied

After:

  $ gocryptfs -fsck .
  Cannot open config file: open /var/tmp/check-gocryptfs/scratchdev/gocryptfs.conf: permission denied
2018-07-23 22:25:40 +02:00
Jakob Unterwurzacher f316f1b2df fusefronted: disallow writes running concurrently with reads
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,

  doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,

as the read could pick up a block that has not yet been completely written -
write() is not atomic!

Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.

This also simplifies the file header locking.
2018-07-22 22:29:22 +02:00
Jakob Unterwurzacher c70df522d2 fusefrontend: doWrite: delete file header if first write fails
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:

  readFileID 2580: incomplete file, got 18 instead of 19 bytes

This could happen when writing the file header works, but writing
the actual data fails.

Now we kill the header again by truncating the file to zero.
2018-07-15 15:12:55 +02:00
Jakob Unterwurzacher 55bb22bad6 fusefrontend: doWrite: no need to take HeaderLock.RLock()
Other writers are blocked by ContentLock already.
2018-07-15 12:40:23 +02:00
Jakob Unterwurzacher bbf5b72fff WriteDirIV: delete incomplete gocryptfs.diriv file if write fails
If the underlying filesystem is full, writing to gocryptfs.diriv may
fail, and later fsck show this:

	OpenDir "xyz": could not read gocryptfs.diriv: wanted 16 bytes, got 0

Uncovered by xfstests generic/083.

Also fixes a fd leak in the error path.
2018-07-15 12:02:39 +02:00
Jakob Unterwurzacher bcca323cb7 contentenc: reserve one extra block in pool plaintext buffers
File holes and -fsck can cause unaligned read accesses, which means
we have to decrypt one extra plaintext block.

xfstests generic/083 manage to crash -fsck like this:

generic/083	2018/07/14 15:25:21 wrong len=266240, want=131072
panic: wrong len=266240, want=131072

goroutine 1 [running]:
log.Panicf(0x67fc00, 0x15, 0xc4204fec90, 0x2, 0x2)
	/usr/local/go/src/log/log.go:333 +0xda
github.com/rfjakob/gocryptfs/internal/contentenc.(*bPool).Put(0xc4200d4800, 0xc4202f2000, 0x21000, 0x41000)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/internal/contentenc/bpool.go:27 +0x15d
github.com/rfjakob/gocryptfs/internal/fusefrontend.(*File).doRead(0xc4200b4500, 0xc42019e000, 0x0, 0x20000, 0x28400, 0x20000, 0xc42019e000, 0xc4204ff008, 0x435164, 0xc420000180)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file.go:227 +0xba9
github.com/rfjakob/gocryptfs/internal/fusefrontend.(*File).Read(0xc4200b4500, 0xc42019e000, 0x20000, 0x20000, 0x28400, 0x0, 0x0, 0x0)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file.go:246 +0x23e
main.(*fsckObj).file(0xc420069320, 0xc42001a630, 0x21)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/fsck.go:126 +0x21f
main.(*fsckObj).dir(0xc420069320, 0xc420014dc0, 0x1d)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/fsck.go:76 +0x387
main.(*fsckObj).dir(0xc420069320, 0xc42021dae0, 0x19)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/fsck.go:74 +0x347
2018-07-15 11:39:19 +02:00
Jakob Unterwurzacher 38f79a1abc fsck: add debug output (enabled via -debug)
Turn the commented-out fmt.Printf into debug output
via the tlog infrastructure.
2018-07-15 11:34:30 +02:00
rfjakob 2ed3f128dd
file-format.md: list full block overhead
For large files, the overhead will be 0.78125 %.
2018-07-15 11:24:54 +02:00
Jakob Unterwurzacher 95b93db35f fusefrontend: log prealloc failures at Info level
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.

Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
2018-07-14 15:18:27 +02:00
Jakob Unterwurzacher 770c23db7c travis: run dep ensure in verbose mode
But make the output fold to reduce clutter.
Move wget to the install section, where output is folded by default.
2018-07-08 14:49:57 +02:00
Jakob Unterwurzacher 7f5a97e843 MANPAGE: add proper description text
We only had an "options" section which was
mislabeled as "description". Add a proper description text.
2018-07-07 23:51:16 +02:00
Jakob Unterwurzacher 6544328f96 README: it's not a mirror 2018-07-04 09:10:36 +02:00
Jakob Unterwurzacher 794f4fcde5 README: call the Markdown source "mirror" 2018-07-04 09:08:26 +02:00
Jakob Unterwurzacher 53f7e1a0f0 macos: fix O_DIRECT build failure
O_DIRECT has no direct equivalent on MacOS
(check out https://github.com/libuv/libuv/issues/1600 for details).

Just define it to zero there.
2018-07-04 09:04:00 +02:00
Jakob Unterwurzacher f6a3e2c7d1 README: mention O_DIRECT change 2018-07-02 23:58:55 +02:00
Jakob Unterwurzacher 893e41149e fusefrontend: disallow O_DIRECT and fall back to buffered IO
O_DIRECT accesses must be aligned in both offset and length. Due to our
crypto header, alignment will be off, even if userspace makes aligned
accesses. Running xfstests generic/013 on ext4 used to trigger lots of
EINVAL errors due to missing alignment. Just fall back to buffered IO.
2018-07-02 23:54:37 +02:00
Jakob Unterwurzacher c51fc9e07d fusefronted: downgrade fallocate message severity
The message causes output mismatches in xfstests generic/112.
Downgrade the severity to Info so it gets disabled when using "-q".
2018-07-02 23:03:43 +02:00
Jakob Unterwurzacher 01a078e7c0 Fix golint warnings 2018-07-01 22:00:06 +02:00
Jakob Unterwurzacher 5243cd0e0d trezor: hide behind compile tag
The trezor libraries are not yet stable enough to build
gocryptfs with trezor support by default.

It does not even compile at the moment:

  $ ./build.bash -tags enable_trezor
  # github.com/conejoninja/tesoro/vendor/github.com/trezor/usbhid
  ../../conejoninja/tesoro/vendor/github.com/trezor/usbhid/hid.go:32:11: fatal error: os/threads_posix.c: No such file or directory
    #include "os/threads_posix.c"
           ^~~~~~~~~~~~~~~~~~~~
  compilation terminated.

https://github.com/conejoninja/tesoro/issues/9
2018-07-01 21:48:51 +02:00
Jakob Unterwurzacher 2d68b06f9d fsck: handle sparse files efficiently, fix xfstests generic/285
If we encounter a 128KB block of zeros, try to skip to the next
data section by calling File.SeekData().

This fixes xfstests generic/285, which creates a 17TB sparse file,
and runs fsck afterwards. Without this optimization, fsck would
take ages.
2018-07-01 21:48:51 +02:00