Commit Graph

1800 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
0c1ceed1fa fusefrontend: make GetAttr() symlink-safe
Use openBackingDir() and Fstatat().

High performance impact, though part of it should be
mitigated by adding DirIV caching to the new code paths.

$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.Eou: gocryptfs v1.6-37-ge3914b3-dirty; go-fuse v20170619-66-g6df8ddc; 2018-10-14 go1.11
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.2289 s, 213 MB/s
READ:  262144000 bytes (262 MB, 250 MiB) copied, 1.02616 s, 255 MB/s
UNTAR: 24.490
MD5:   13.120
LS:    3.368
RM:    9.232
2019-01-01 16:24:09 +01:00
Jakob Unterwurzacher
932efbd459 fusefrontend: make openBackingDir() symlink-safe
openBackingDir() used encryptPath(), which is not symlink-safe
itself. Drop encryptPath() and implement our own directory walk.

Adds three seconds to untar and two seconds to rm:

$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.MzG: gocryptfs v1.6-36-g8fb3c2f-dirty; go-fuse v20170619-66-g6df8ddc; 2018-10-14 go1.11
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.25078 s, 210 MB/s
READ:  262144000 bytes (262 MB, 250 MiB) copied, 1.0318 s, 254 MB/s
UNTAR: 20.941
MD5:   11.568
LS:    1.638
RM:    5.337
2019-01-01 16:24:09 +01:00
Jakob Unterwurzacher
0e2e7c13cf fusefrontend: mark symlink-safe FUSE calls
Document which FUSE calls are already symlink-safe in
the function comment.
2019-01-01 16:24:09 +01:00
Jakob Unterwurzacher
c09bf1f228 fusefrontend: make DecryptPath() symlink-safe
DecryptPath is now symlink-safe through the use of *at()
functions.
2019-01-01 16:24:09 +01:00
Jakob Unterwurzacher
ed6ed513d7 fusefrontend: make Access() symlink-safe.
Make Access() symlink-safe through use of faccessat.
2019-01-01 16:24:09 +01:00
Jakob Unterwurzacher
545a03da24 nametransform: comments: directly link to ioutil.WriteFile fix
So the reader does not have to read through the whole ticket.
The commit message has a nice summary of the problem.
2019-01-01 16:23:28 +01:00
Sebastian Lackner
5713154468 fusefrontend: Fix debug message in doWrite() method. 2019-01-01 16:12:42 +01:00
Sebastian Lackner
9ed60678e5 fusefrontend: Fix order of arguments in debug message for Read() FUSE call. 2019-01-01 16:12:05 +01:00
Sebastian Lackner
87ced5f95d nametransform: Delete incomplete longname files on error. 2019-01-01 16:09:57 +01:00
Sebastian Lackner
3dd5a6c069 main: Remove a duplicate word in a comment. 2018-12-30 08:33:24 +01:00
Sebastian Lackner
24594d99bf configfile: Fix a copy&paste error in validateParams method. 2018-12-28 09:58:46 +01:00
Sebastian Lackner
07c486603c configfile: Explicitly wipe scrypt derived key after decrypting/encrypting master key.
Further raises the bar for recovering keys from memory.
2018-12-27 18:47:14 +01:00
Sebastian Lackner
874eaf9734 Assorted spelling fixes.
Mostly detected with the 'codespell' utility, but also includes some
manual grammar fixes.
2018-12-27 15:19:55 +01:00
Sebastian Lackner
4c2ff26457 fusefrontend: Remove unnecessary check in doRead function.
The same condition is already checked a few lines above, and 'err' is not
changed inbetween.
2018-12-27 15:18:03 +01:00
Sebastian Lackner
1ced0b192e fusefrontend: Don't treat Fchownat error as failure in Mkdir.
The directory was already created, so return success even if Fchownat fails.
The same error handling is already used if fs.args.PlaintextNames is false.
2018-12-27 15:16:00 +01:00
Sebastian Lackner
5918884926 fusefrontend: Check the correct 'err' variable. 2018-12-27 15:11:23 +01:00
Sebastian Lackner
2a010263f6 build.bash: Escape LDFLAGS before passing them to 'go build'.
This ensures that ./build.bash still works when the LDFLAGS environment
variable contains multiple options, e.g., LDFLAGS="-lpthread -lm". The
correct way of passing multiple options is discussed here:
https://github.com/golang/go/issues/6234

For some unknown reason, the method only works when -extldflags is the
last argument - is this a bug in Go?
2018-12-27 15:08:35 +01:00
Jakob Unterwurzacher
3cba378ad5 build.bash: support user-set LDFLAGS
As requested at https://github.com/rfjakob/gocryptfs/pull/280
2018-12-16 21:34:41 +01:00
Jakob Unterwurzacher
bf241ce77e build.bash: use -trimpath for reproducible builds
Support both Go 1.7...1.9 and Go 1.10 by checking the
version and using the appropropriate syntax.

We trim GOPATH/src and use both -gcflags and -asmflags like Debian does in
ab2bbcfc00/lib/Debian/Debhelper/Buildsystem/golang.pm (L465) .
2018-12-16 20:39:01 +01:00
Jakob Unterwurzacher
d32c149266 build.bash: respect SOURCE_DATE_EPOCH
SOURCE_DATE_EPOCH seems to be the standard env variable
for faking a build date for reproducible builds.
2018-12-16 20:16:12 +01:00
Jakob Unterwurzacher
9daa205c10 README: replace openssl-gcm.md with link to wiki
The CPU-Benchmarks wiki page has a lot more info
than openssl-gcm.md had.
2018-12-16 12:49:13 +01:00
Jakob Unterwurzacher
a55e53c196 tests: fix TestPassfileNewline
Due to a copy-paste error, we ran the wrong test in the
subprocess.

Thanks @slackner for noticing at
295d432175 (r31690478) !
2018-12-16 12:33:25 +01:00
Jakob Unterwurzacher
295d432175 passfile: directly read file instead of invoking cat
Allows better error handling, gets rid of the call to an
external program, and fixes https://github.com/rfjakob/gocryptfs/issues/278 .
2018-12-15 17:09:38 +01:00
Jakob Unterwurzacher
b29ee62749 Add v1.6.1 to changelog 2018-12-12 21:41:05 +01:00
Jakob Unterwurzacher
e665df7179 syscallcompat: downgrade DT_UNKNOWN message level on XFS
Old XFS filesystems always return DT_UNKNOWN. Downgrade the message
to "info" level if we are on XFS.

Using the "warning" level means that users on old XFS filesystems
cannot run the test suite as it intentionally aborts on any
warnings.

Fixes https://github.com/rfjakob/gocryptfs/issues/267
2018-11-17 17:44:21 +01:00
Jakob Unterwurzacher
d882ed45da main: drop hardcoded /usr/bin/logger path
The hardcoded full paths were introduced to handle the
case of an empty PATH environment variable. However,
since commit 10212d791a we set PATH to a default
value if empty. The hardcoded paths are no longer neccessary,
and cause problems on some distros:

User voobscout on
https://github.com/rfjakob/gocryptfs/issues/225#issuecomment-438682034 :

  just to chime in - please don't hardcode paths, for example I'm on
  NixOS and logger lives in /run/current-system/sw/bin/logger

Drop the hardcoded paths.
2018-11-17 17:03:11 +01:00
Jakob Unterwurzacher
1ed08c7384 tlog: disable color codes when switching to syslog
When gocryptfs was started on a terminal and later
daemonized, the color codes stayed active in the syslog
output.

The codes are not visible in "journalctl -f", which is why
I have not noticed it yet, but they do show up in normal
syslog as the usual "#033[33m" crap.
2018-10-17 22:34:30 +02:00
Jakob Unterwurzacher
4cdf6b9af9 fusefronted: log more details on WriteAt failures
Also log inode number, fd number, offset and length.

Maybe help debugging https://github.com/rfjakob/gocryptfs/issues/269 .
2018-10-17 22:18:07 +02:00
Jakob Unterwurzacher
3cd892ccde main: also redirect Fatal logger to syslog on daemonization
The messages would still be collected via gocryptfs-logger,
but let's do it right.

Before:

  Oct 17 21:58:12 brikett gocryptfs[9926]: testing info
  Oct 17 21:58:12 brikett gocryptfs[9926]: testing warn
  Oct 17 21:58:12 brikett gocryptfs-9926-logger[9935]: testing fatal

After:

  Oct 17 22:00:53 brikett gocryptfs[10314]: testing info
  Oct 17 22:00:53 brikett gocryptfs[10314]: testing warn
  Oct 17 22:00:53 brikett gocryptfs[10314]: testing fatal
2018-10-17 22:18:03 +02:00
Jesse Dunietz
0751b4e00a Updated manpage for -idle flag 2018-10-11 22:31:45 +02:00
Jakob Unterwurzacher
e537ecbe2e tests: fix golint error
Error was:

  tests/cli/cli_test.go:552: declaration of "err" shadows declaration at tests/cli/cli_test.go:544
2018-10-11 20:57:52 +02:00
Jakob Unterwurzacher
04241455a2 tests: add idle timeout test
Mount with idle timeout 10ms and check that the process exits by itself
within 5 seconds.
2018-10-11 20:43:28 +02:00
Jesse Dunietz
87d3ed9187 Add option for autounmount
Even though filesystem notifications aren't implemented for FUSE, I decided to
try my hand at implementing the autounmount feature (#128). I based it on the
EncFS autounmount code, which records filesystem accesses and checks every X
seconds whether it's idled long enough to unmount.

I've tested the feature locally, but I haven't added any tests for this flag.
I also haven't worked with Go before. So please let me know if there's
anything that should be done differently.

One particular concern: I worked from the assumption that the open files table
is unique per-filesystem. If that's not true, I'll need to add an open file
count and associated lock to the Filesystem type instead.

https://github.com/rfjakob/gocryptfs/pull/265
2018-10-11 20:16:45 +02:00
Jakob Unterwurzacher
57a5a8791f tests: syscallcompat: allow failure for symlinks > 1000
MacOS and old XFS versions do not support very long symlinks,
but let's not make the tests fail because of that.

https://github.com/rfjakob/gocryptfs/issues/267
2018-10-11 19:45:47 +02:00
Jakob Unterwurzacher
4f2feb1be7 tests: catch "name too long" symlink failure on XFS
Retry with length 1000 if length 4000 fails, which
should work on all filesystems.

Failure was:

  --- FAIL: TestTooLongSymlink (0.00s)
      correctness_test.go:198: symlink xxx[...]xxxx /tmp/xfs.mnt/gocryptfs-test-parent/549823072/365091391/TooLongSymlink: file name too long

https://github.com/rfjakob/gocryptfs/issues/267
2018-10-10 22:40:55 +02:00
Jakob Unterwurzacher
5a1ebdb4f7 tests: respect TMPDIR if set
Setting TMPDIR now allows to run the tests against
a directory of your choice, making it easier to test
different filesystems.
2018-10-10 22:24:20 +02:00
Jakob Unterwurzacher
e4f1a32a9a fusefrontend: Fix uint16 build failure on Darwin
Error was:

  # github.com/rfjakob/gocryptfs/internal/fusefrontend
  internal/fusefrontend/fs.go:179: cannot use perms | 256 (type uint16) as type uint32 in argument to syscall.Fchmod
  internal/fusefrontend/fs.go:185: cannot use perms (type uint16) as type uint32 in argument to syscall.Fchmod
2018-09-23 12:17:59 +02:00
Jakob Unterwurzacher
a1fb456618 fusefrontend: make Rename() symlink-safe
Use Openat() and the openBackingDir() helper so we
never follow symlinks.
2018-09-23 12:17:59 +02:00
Jakob Unterwurzacher
897bb8924f fusefrontend: make Create() symlink-safe
Use Openat() and the openBackingDir() helper so we
never follow symlinks.
2018-09-23 12:17:59 +02:00
Jakob Unterwurzacher
63762b33af fusefrontend: Open(): fix dirfd leak
Close was missing.
2018-09-23 12:17:59 +02:00
Jakob Unterwurzacher
bead82c9fb fusefrontend: add named parameters to openBackingDir
Named parameters make using the function easier.
2018-09-23 12:17:59 +02:00
Jakob Unterwurzacher
c270b21efc fusefrontend: get rid of os.File* wrapping
Directly use int file descriptors for the dirfd
and get rid of one level of indirection.
2018-09-23 12:17:26 +02:00
Jakob Unterwurzacher
22fba4ac3e fusefrontent: make Open() symlink-safe 2018-09-23 12:17:26 +02:00
Jakob Unterwurzacher
05c8d4a1c4 tests: add symlink_race tool
Help uncover symlink races.
2018-09-23 12:17:26 +02:00
Jakob Unterwurzacher
2d01d5f2d4 tlog: always trim trailing newlines
The messages we print through tlog sometimes do, sometimes do
not contain a trailing newline. The stdlib logger usually drops
trailing newlines automatically, but tlog postfixes ColorReset to
the caller's message, so the logger logic does not work when we
print colored output.

Drop the newlines on our own, and add a test.

Fixes the blank lines in fsck output:

~/go/src/github.com/rfjakob/gocryptfs/tests/fsck$ ./run_fsck.bash
Reading password from extpass program
Decrypting master key
OpenDir "": invalid entry "invalid_file_name.3": illegal base64 data at input byte 17
OpenDir "": invalid entry "invalid_file_name_2": bad message
fsck: corrupt entry in dir "": "invalid_file_name.3"
fsck: corrupt entry in dir "": "invalid_file_name_2"
OpenDir "": invalid entry "invalid_file_name____1": bad message
fsck: corrupt entry in dir "": "invalid_file_name____1"
doRead 4327225: corrupt block #0: stupidgcm: message authentication failed
fsck: error reading file "corrupt_file" (inum 4327225): 5=input/output error
cipherSize 40 < overhead 50: corrupt file

doRead 4327074: corrupt header: ParseHeader: invalid version, want=2 have=22616
cipherSize 40 < overhead 50: corrupt file

fsck: error reading file "corrupt_file_2" (inum 4327074): 5=input/output error
Readlink "s-P7PcQDUcVkoeMDnC3EYA": decrypting target failed: stupidgcm: message authentication failed
fsck: error reading symlink "corrupt_symlink": 5=input/output error
Readlink "iI0MtUdzELPeOAZYwYZFee169hpGgd3l2PXQBcc9sl4": decrypting target failed: illegal base64 data at input byte 0
fsck: error reading symlink "corrupt_symlink_2": 5=input/output error
OpenDir "yrwcjj2qoC4IYvhw9sbfRg": could not read gocryptfs.diriv: wanted 16 bytes, got 17
fsck: error opening dir "diriv_too_long": 5=input/output error
OpenDir "trqecbMNXdzLqzpk7fSfKw": could not read gocryptfs.diriv: wanted 16 bytes, got 3
fsck: error opening dir "diriv_too_short": 5=input/output error
cipherSize 8 < header size 18: corrupt file

readFileID 4327049: incomplete file, got 8 instead of 19 bytes
fsck: corrupt file "incomplete_file_1" (inode 4327049)
readFileID 4327038: incomplete file, got 18 instead of 19 bytes
fsck: corrupt file "incomplete_file_2" (inode 4327038)
cipherSize 1 < header size 18: corrupt file

readFileID 4327063: incomplete file, got 1 instead of 19 bytes
fsck: corrupt file "incomplete_file_3" (inode 4327063)
fsck: error opening dir "missing_diriv": 2=no such file or directory
ListXAttr: invalid xattr name "user.gocryptfs.0a5e7yWl0SGUGeWB0Sy2K0": bad message
fsck: corrupt xattr name on file "xattr_corrupt_name": "user.gocryptfs.0a5e7yWl0SGUGeWB0Sy2K0"
GetXAttr: stupidgcm: message authentication failed
fsck: error reading xattr "user.foo" from "xattr_corrupt_value": 5=input/output error
fsck summary: 15 corrupt files
2018-09-23 11:28:49 +02:00
Jakob Unterwurzacher
737a2f2012 syscallcompat: untangle Openat flag check
Check for O_NWFOLLOW and O_EXCL separately to
make the logic clearer.
2018-09-22 19:38:47 +02:00
Jakob Unterwurzacher
9e6ee47bc9 tests: detect and report chmod failures earlier
Instead of reporting the consequence:

    matrix_test.go:906: modeHave 0664 != modeWant 0777

Report it if chmod itself fails, and also report the old file mode:

    matrix_test.go:901: chmod 000 -> 777 failed: bad file descriptor
2018-09-22 13:39:17 +02:00
Jakob Unterwurzacher
5ca6243eeb main: sanitize commas from fsname
The cipherdir path is used as the fsname, as displayed
in "df -T". Now, having a comma in fsname triggers a sanity check
in go-fuse, aborting the mount with:

  /bin/fusermount: mount failed: Invalid argument
  fuse.NewServer failed: fusermount exited with code 256

Sanitize fsname by replacing any commas with underscores.

https://github.com/rfjakob/gocryptfs/issues/262
2018-09-18 19:39:52 +02:00
Jakob Unterwurzacher
e8d8ae54d3 fusefrontend: use OpenDirNofollow in openBackingDir
Rename openBackingPath to openBackingDir and use OpenDirNofollow
to be safe against symlink races. Note that openBackingDir is
not used in several important code paths like Create().

But it is used in Unlink, and the performance impact in the RM benchmark
to be acceptable:

Before

	$ ./benchmark.bash
	Testing gocryptfs at /tmp/benchmark.bash.bYO: gocryptfs v1.6-12-g930c37e-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
	WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.07979 s, 243 MB/s
	READ:  262144000 bytes (262 MB, 250 MiB) copied, 0.882413 s, 297 MB/s
	UNTAR: 16.703
	MD5:   7.606
	LS:    1.349
	RM:    3.237

After

	$ ./benchmark.bash
	Testing gocryptfs at /tmp/benchmark.bash.jK3: gocryptfs v1.6-13-g84d6faf-dirty; go-fuse v20170619-49-gb11e293; 2018-09-08 go1.10.3
	WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 1.06261 s, 247 MB/s
	READ:  262144000 bytes (262 MB, 250 MiB) copied, 0.947228 s, 277 MB/s
	UNTAR: 17.197
	MD5:   7.540
	LS:    1.364
	RM:    3.410
2018-09-08 19:27:33 +02:00
Jakob Unterwurzacher
84d6fafeca tests: restore Go 1.7 compatibility
Go 1.7 does not have t.Name() yet.
2018-09-08 18:19:53 +02:00