Commit Graph

1374 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
eaa5aecd42 cli: add multipleStrings type
Will be used for --exclude.
2018-08-11 23:25:17 +02:00
Jakob Unterwurzacher
06f1ea951b tests: VerifyExistence: panic on inconsistent results
In the past we did not check whether all methods of checking
the file return the same result. We should.
2018-08-11 22:37:22 +02:00
Jakob Unterwurzacher
f48b731e76 travis ci: drop go 1.5 and go 1.6
Trying to build with these versions now throws this error:

  # golang.org/x/sys/unix
  ../../../golang.org/x/sys/unix/ioctl.go:18: undefined: runtime.KeepAlive

It looks like x/sys/unix has dropped support for older Go
versions.
2018-08-02 20:06:34 +02:00
Jakob Unterwurzacher
d620054caf fsck: print inode number on file read error
This makes it possible to find the file without mounting the fs.
2018-07-23 22:34:41 +02:00
Jakob Unterwurzacher
f4a972ddf1 configfile: drop superflous Printf
Before:

  $ gocryptfs -fsck .
  LoadConfFile: ReadFile: &os.PathError{Op:"open", Path:"/var/tmp/check-gocryptfs/scratchdev/gocryptfs.conf", Err:0xd}
  Cannot open config file: open /var/tmp/check-gocryptfs/scratchdev/gocryptfs.conf: permission denied

After:

  $ gocryptfs -fsck .
  Cannot open config file: open /var/tmp/check-gocryptfs/scratchdev/gocryptfs.conf: permission denied
2018-07-23 22:25:40 +02:00
Jakob Unterwurzacher
f316f1b2df fusefronted: disallow writes running concurrently with reads
As uncovered by xfstests generic/465, concurrent reads and writes
could lead to this,

  doRead 3015532: corrupt block #1039: stupidgcm: message authentication failed,

as the read could pick up a block that has not yet been completely written -
write() is not atomic!

Now writes take ContentLock exclusively, while reads take it shared,
meaning that multiple reads can run in parallel with each other, but
not with a write.

This also simplifies the file header locking.
2018-07-22 22:29:22 +02:00
Jakob Unterwurzacher
c70df522d2 fusefrontend: doWrite: delete file header if first write fails
xfstests generic/083 fills the filesystem almost completely while
running fsstress in parallel. In fsck, these would show up:

  readFileID 2580: incomplete file, got 18 instead of 19 bytes

This could happen when writing the file header works, but writing
the actual data fails.

Now we kill the header again by truncating the file to zero.
2018-07-15 15:12:55 +02:00
Jakob Unterwurzacher
55bb22bad6 fusefrontend: doWrite: no need to take HeaderLock.RLock()
Other writers are blocked by ContentLock already.
2018-07-15 12:40:23 +02:00
Jakob Unterwurzacher
bbf5b72fff WriteDirIV: delete incomplete gocryptfs.diriv file if write fails
If the underlying filesystem is full, writing to gocryptfs.diriv may
fail, and later fsck show this:

	OpenDir "xyz": could not read gocryptfs.diriv: wanted 16 bytes, got 0

Uncovered by xfstests generic/083.

Also fixes a fd leak in the error path.
2018-07-15 12:02:39 +02:00
Jakob Unterwurzacher
bcca323cb7 contentenc: reserve one extra block in pool plaintext buffers
File holes and -fsck can cause unaligned read accesses, which means
we have to decrypt one extra plaintext block.

xfstests generic/083 manage to crash -fsck like this:

generic/083	2018/07/14 15:25:21 wrong len=266240, want=131072
panic: wrong len=266240, want=131072

goroutine 1 [running]:
log.Panicf(0x67fc00, 0x15, 0xc4204fec90, 0x2, 0x2)
	/usr/local/go/src/log/log.go:333 +0xda
github.com/rfjakob/gocryptfs/internal/contentenc.(*bPool).Put(0xc4200d4800, 0xc4202f2000, 0x21000, 0x41000)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/internal/contentenc/bpool.go:27 +0x15d
github.com/rfjakob/gocryptfs/internal/fusefrontend.(*File).doRead(0xc4200b4500, 0xc42019e000, 0x0, 0x20000, 0x28400, 0x20000, 0xc42019e000, 0xc4204ff008, 0x435164, 0xc420000180)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file.go:227 +0xba9
github.com/rfjakob/gocryptfs/internal/fusefrontend.(*File).Read(0xc4200b4500, 0xc42019e000, 0x20000, 0x20000, 0x28400, 0x0, 0x0, 0x0)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend/file.go:246 +0x23e
main.(*fsckObj).file(0xc420069320, 0xc42001a630, 0x21)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/fsck.go:126 +0x21f
main.(*fsckObj).dir(0xc420069320, 0xc420014dc0, 0x1d)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/fsck.go:76 +0x387
main.(*fsckObj).dir(0xc420069320, 0xc42021dae0, 0x19)
	/home/jakob/go/src/github.com/rfjakob/gocryptfs/fsck.go:74 +0x347
2018-07-15 11:39:19 +02:00
Jakob Unterwurzacher
38f79a1abc fsck: add debug output (enabled via -debug)
Turn the commented-out fmt.Printf into debug output
via the tlog infrastructure.
2018-07-15 11:34:30 +02:00
rfjakob
2ed3f128dd
file-format.md: list full block overhead
For large files, the overhead will be 0.78125 %.
2018-07-15 11:24:54 +02:00
Jakob Unterwurzacher
95b93db35f fusefrontend: log prealloc failures at Info level
If the underlying filesystem is full, it is normal get ENOSPC here.
Log at Info level instead of Warning.

Fixes xfstests generic/015 and generic/027, which complained about
the extra output.
2018-07-14 15:18:27 +02:00
Jakob Unterwurzacher
770c23db7c travis: run dep ensure in verbose mode
But make the output fold to reduce clutter.
Move wget to the install section, where output is folded by default.
2018-07-08 14:49:57 +02:00
Jakob Unterwurzacher
7f5a97e843 MANPAGE: add proper description text
We only had an "options" section which was
mislabeled as "description". Add a proper description text.
2018-07-07 23:51:16 +02:00
Jakob Unterwurzacher
6544328f96 README: it's not a mirror 2018-07-04 09:10:36 +02:00
Jakob Unterwurzacher
794f4fcde5 README: call the Markdown source "mirror" 2018-07-04 09:08:26 +02:00
Jakob Unterwurzacher
53f7e1a0f0 macos: fix O_DIRECT build failure
O_DIRECT has no direct equivalent on MacOS
(check out https://github.com/libuv/libuv/issues/1600 for details).

Just define it to zero there.
2018-07-04 09:04:00 +02:00
Jakob Unterwurzacher
f6a3e2c7d1 README: mention O_DIRECT change 2018-07-02 23:58:55 +02:00
Jakob Unterwurzacher
893e41149e fusefrontend: disallow O_DIRECT and fall back to buffered IO
O_DIRECT accesses must be aligned in both offset and length. Due to our
crypto header, alignment will be off, even if userspace makes aligned
accesses. Running xfstests generic/013 on ext4 used to trigger lots of
EINVAL errors due to missing alignment. Just fall back to buffered IO.
2018-07-02 23:54:37 +02:00
Jakob Unterwurzacher
c51fc9e07d fusefronted: downgrade fallocate message severity
The message causes output mismatches in xfstests generic/112.
Downgrade the severity to Info so it gets disabled when using "-q".
2018-07-02 23:03:43 +02:00
Jakob Unterwurzacher
01a078e7c0 Fix golint warnings 2018-07-01 22:00:06 +02:00
Jakob Unterwurzacher
5243cd0e0d trezor: hide behind compile tag
The trezor libraries are not yet stable enough to build
gocryptfs with trezor support by default.

It does not even compile at the moment:

  $ ./build.bash -tags enable_trezor
  # github.com/conejoninja/tesoro/vendor/github.com/trezor/usbhid
  ../../conejoninja/tesoro/vendor/github.com/trezor/usbhid/hid.go:32:11: fatal error: os/threads_posix.c: No such file or directory
    #include "os/threads_posix.c"
           ^~~~~~~~~~~~~~~~~~~~
  compilation terminated.

https://github.com/conejoninja/tesoro/issues/9
2018-07-01 21:48:51 +02:00
Jakob Unterwurzacher
2d68b06f9d fsck: handle sparse files efficiently, fix xfstests generic/285
If we encounter a 128KB block of zeros, try to skip to the next
data section by calling File.SeekData().

This fixes xfstests generic/285, which creates a 17TB sparse file,
and runs fsck afterwards. Without this optimization, fsck would
take ages.
2018-07-01 21:48:51 +02:00
Jakob Unterwurzacher
e951043084 fusefrontend: add File.SeekData() function
This function will enable "gocryptfs -fsck" to handle
sparse files efficiently.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
a2af1fb5da fusefrontend: export "File" type
"gocryptfs -fsck" will need access to helper functions,
and to get that, it will need to cast a gofuse.File to a
fusefrontend.File. Make fusefrontend.File exported to make
this work.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
4a73637782 fsck: get rid of channel read closures
Create proper functions instead to declutter the logic.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
1a18d8e609 fsck: rename "CorruptItems" channel to "MitigatedCorruptions"
Make it clear that this channel is only used to report corruptions
that are transparently mitigated and do not return an error to
the user.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
283184dd5a fsck: suppress "no problems found" when running quiet
Passing "-q" now disables the "fsck summary: no problems found\n"
message.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
52c9ff1ded main: improve fuse.NewServer error message
Before: fuse.NewServer failed: "fusermount exited with code 256\n"
After:  fuse.NewServer failed: fusermount exited with code 256
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
6d64dfe8f7 Only print masterkey once on -init
It is no longer printed at all when mounting a filesystem,
printing on -init can be disabled with -q.

https://github.com/rfjakob/gocryptfs/issues/76
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
991891a5c4 trezor: add sanity checks for decrypted value
Check that the value has changed, is not all-zero
and has the right length.
2018-07-01 20:56:22 +02:00
Dmitry Yu Okunev
978f1f3f6d Implemented the support of Trezor devices. 2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
3f9a4bbcaa main: replace naked panic with log.Panic
Fix the check that failed to find the case as well.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
1a5e9cfb1e trezor: reject password change on Trezor-enabled fs
This is not supported (yet? does it make sense?).
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
9a15dfa494 trezor: add TrezorPayload
TrezorPayload stores 32 random bytes used for unlocking
the master key using a Trezor security module. The randomness makes sure
that a unique unlock value is used for each gocryptfs filesystem.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
91de77943f configfile: reduce function name stutter
configfile.LoadConfFile()   -> configfile.Load()
configfile.CreateConfFile() -> configfile.Create()
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
8e5ca7299a trezor: exit with usage error if used together with -extpass
Using an external program for the Trezor PIN is not supported
at the moment.
2018-07-01 20:56:22 +02:00
Jakob Unterwurzacher
4bf02f4760 trezor: add basic "-init -trezor" test
Verify that the Trezor feature flag is set after "-init -trezor".
2018-07-01 20:56:09 +02:00
Jakob Unterwurzacher
c6f6e8ec4d trezor: add skeleton for Trezor support
readpassword.Trezor() is not implemented yet and returns
a hardcoded dummy key.
2018-07-01 20:56:04 +02:00
Jakob Unterwurzacher
02ab358451 README: Add folders side-by-side animation gif
10 second animation that shows what gocryptfs does.
2018-07-01 15:28:22 +02:00
Jakob Unterwurzacher
743c7705b2 configfile: use tlog.ColorYellow instead of hardcoded color code 2018-06-24 19:52:47 +02:00
Jakob Unterwurzacher
1bab400fca Fix three golint warnings
We are clean again.

Warnings were:

internal/fusefrontend/fs.go:443:14: should omit type string from declaration
of var cTarget; it will be inferred from the right-hand side
internal/fusefrontend/xattr.go:26:1: comment on exported method FS.GetXAttr
should be of the form "GetXAttr ..."
internal/syscallcompat/sys_common.go:9:7: exported const PATH_MAX should have
comment or be unexported
2018-06-19 20:16:21 +02:00
Jakob Unterwurzacher
009cc0ae8b Update README and MANPAGE for v1.5
Also update the performance numbers. I see some slowdown, reason
is not yet clear, but nothing to block the release.
2018-06-12 23:07:42 +02:00
Jakob Unterwurzacher
e925e1b78e dep: fix constraints and update dependencies
Gopkg.toml was emptied out by commit c3e12b5e68 which
seemed to work fine at the time. It turns out that, in
absence of a

  branch = "master"

constraint, dep will use the last tag. We want latest
master, as this is what "go get" fetches, and hence
what Travis uses for testing.
2018-06-12 23:05:53 +02:00
Jakob Unterwurzacher
bfa50517e9 xattr: return EOPNOTSUPP instead of ENODATA in GetXattr
Reading system.posix_acl_access and system.posix_acl_default
should return EOPNOTSUPP to inform user-space that we do not
support ACLs.

xftestest essientially does

	chacl -l | grep "Operation not supported"

to determine if the filesystem supports ACLs, and used to
wrongly believe that gocryptfs does.
2018-06-12 23:05:53 +02:00
Jakob Unterwurzacher
bde7ba57b0 darwin does not have PATH_MAX
Define our own, with the value from Linux.
2018-06-08 00:47:48 +02:00
Jakob Unterwurzacher
5ad26495fc main: cut down "flag provided but not defined" error message
Don't dump the help text on the unsuspecting user, but
give a short error message:

  $ gocryptfs -foobar
  flag provided but not defined: -foobar
  Invalid command line: gocryptfs -foobar. Try 'gocryptfs -help'.

For comparison: This is what cp does:

  $ cp --foo
  cp: unrecognized option '--foo'
  Try 'cp --help' for more information.

And this what we used to do:

  $ gocryptfs -foobar
  flag provided but not defined: -foobar
  gocryptfs v1.4.4-45-gfb772da; go-fuse v20170619-35-gb16719c; 2018-06-08 go1.10.2

  Usage: gocryptfs -init|-passwd|-info [OPTIONS] CIPHERDIR
    or   gocryptfs [OPTIONS] CIPHERDIR MOUNTPOINT

  Common Options (use -hh to show all):
    -aessiv            Use AES-SIV encryption (with -init)
    -allow_other       Allow other users to access the mount
    -config            Custom path to config file
    -ctlsock           Create control socket at location
    -extpass           Call external program to prompt for the password
    -fg                Stay in the foreground
    -fusedebug         Debug FUSE calls
    -h, -help          This short help text
    -hh                Long help text with all options
    -init              Initialize encrypted directory
    -info              Display information about encrypted directory
    -masterkey         Mount with explicit master key instead of password
    -nonempty          Allow mounting over non-empty directory
    -nosyslog          Do not redirect log messages to syslog
    -passfile          Read password from file
    -passwd            Change password
    -plaintextnames    Do not encrypt file names (with -init)
    -q, -quiet         Silence informational messages
    -reverse           Enable reverse mode
    -ro                Mount read-only
    -speed             Run crypto speed test
    -version           Print version information
    --                 Stop option parsing
  You passed: "-foobar"
  flag provided but not defined: -foobar
2018-06-08 00:03:23 +02:00
Jakob Unterwurzacher
fb772da697 main: forkChild: try to read /proc/self/exe
On Linux, where /proc exists, this makes sure that we are
executing ourselves again, and not some other copy of the
gocryptfs executable.

This usually does not matter, but mount(1) unsets $PATH
and sets argv[0] to just "gocryptfs".
2018-06-07 23:09:27 +02:00
Julian Orth
10212d791a Set a default PATH if PATH is empty or unset
mount(1) unsets PATH before calling mount.fuse. Therefore it's not set
in gocrpytfs either and daemonization fails if gocryptfs was not
executed via an absolute path.

mount.fuse handles this by leaving the execution of the helper to
/bin/sh. /bin/sh handles an empty PATH by searching a few default
locations.

This patch sets the PATH to a sane default if it's empty or unset.
2018-06-07 22:57:57 +02:00