Commit Graph

1949 Commits

Author SHA1 Message Date
Jakob Unterwurzacher cbd5e8ba01 tests/default: add maxlen.bash test 2021-06-26 16:09:04 +02:00
Jakob Unterwurzacher 389aba6a6b maxlen.bash: suppress progress output if not on a terminal 2021-06-26 16:08:29 +02:00
Jakob Unterwurzacher 84e702126a fusefrontend: implement recursive diriv caching
The new contrib/maxlen.bash showed that we have exponential
runtime with respect to directory depth.

The new recursive diriv caching is a lot smarter as it caches
intermediate lookups. maxlen.bash now completes in a few seconds.

xfstests results same as
2d158e4c82/screenlog.0 :

  Failures: generic/035 generic/062 generic/080 generic/093 generic/099 generic/215 generic/285 generic/319 generic/426 generic/444 generic/467 generic/477 generic/523
  Failed 13 of 580 tests

benchmark.bash results are identical:

  $ ./benchmark.bash
  Testing gocryptfs at /tmp/benchmark.bash.BdQ: gocryptfs v2.0.1-17-g6b09bc0; go-fuse v2.1.1-0.20210611132105-24a1dfe6b4f8; 2021-06-25 go1.16.5 linux/amd64
  /tmp/benchmark.bash.BdQ.mnt is a mountpoint
  WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 0,4821 s, 544 MB/s
  READ:  262144000 bytes (262 MB, 250 MiB) copied, 0,266061 s, 985 MB/s
  UNTAR: 8,280
  MD5:   4,564
  LS:    1,745
  RM:    2,244
2021-06-25 13:56:53 +02:00
Jakob Unterwurzacher 05b813f202 nametransform: rename BadNameFlag to BadnameSuffix 2021-06-21 12:12:44 +02:00
Jakob Unterwurzacher 689b74835b nametransform: gather badname functions in badname.go 2021-06-21 12:10:04 +02:00
Jakob Unterwurzacher 2efef1e270 nametransform: delete NameTransformer interface
Useless layer of indirection.
2021-06-21 11:53:33 +02:00
Jakob Unterwurzacher e244b51491 tests: cli: add TestZerokey
TestZerokey verifies that `gocryptfs -zerokey` uses the same options as
`gocryptfs -init`.
2021-06-21 11:48:16 +02:00
Jakob Unterwurzacher 6b0e63c1a8 Improve startup debug output
The startup debug output was very verbose but still missing some
effective crypto settings.
2021-06-21 11:32:04 +02:00
Jakob Unterwurzacher c5d8fa83ae nametransform: pass badname patterns via New
This means we can unexport the field.
2021-06-20 19:09:46 +02:00
Jakob Unterwurzacher 203e65066f main: use JSONDump helper for debug output 2021-06-20 18:25:07 +02:00
Jakob Unterwurzacher 50630e9f3d fido2: hide "FIDO2" in gocryptfs.conf if not used
Result of:

$ gocryptfs -init foo
$ cat foo/gocryptfs.conf

Before:

{
	"Creator": "gocryptfs v2.0.1",
	"EncryptedKey": "FodEdNHD/cCwv1n5BuyAkbIOnJ/O5gfdCh3YssUCJ2DUr0A8DrQ5NH2SLhREeWRL3V8EMiPO2Ncr5IVwE4SSxQ==",
	"ScryptObject": {
		"Salt": "brGaw9Jg1kbPuSXFiwoxqK2oXFTgbniSgpiB+cu+67Y=",
		"N": 65536,
		"R": 8,
		"P": 1,
		"KeyLen": 32
	},
	"Version": 2,
	"FeatureFlags": [
		"GCMIV128",
		"HKDF",
		"DirIV",
		"EMENames",
		"LongNames",
		"Raw64"
	],
	"FIDO2": {
		"CredentialID": null,
		"HMACSalt": null
	}
}

After:

{
	"Creator": "gocryptfs v2.0.1-5-gf9718eb-dirty.DerDonut-badnamecontent",
	"EncryptedKey": "oFMj1lS1ZsM/vEfanNMeCTPw3PZr5VWeL7ap8Jd8YQm6evy2BAhtQ/pd6RzDx84wlCz9TpxqHRihuwSEMnOWWg==",
	"ScryptObject": {
		"Salt": "JZ/5mhy4a8EAQ/wDF1POIEe4/Ss38cfJgXgj26DuA4M=",
		"N": 65536,
		"R": 8,
		"P": 1,
		"KeyLen": 32
	},
	"Version": 2,
	"FeatureFlags": [
		"GCMIV128",
		"HKDF",
		"DirIV",
		"EMENames",
		"LongNames",
		"Raw64"
	]
}
2021-06-20 18:09:21 +02:00
DerDonut a611810ff4 Badname file content access
This proposal is the counterpart of the modifications from the `-badname`
parameter. It modifies the plain -> cipher mapping for filenames when using
`-badname` parameter. The new function `EncryptAndHashBadName` tries to find a
cipher filename for the given plain name with the following steps:

1. If `badname` is disabled or direct mapping is successful: Map directly
(default and current behaviour)

2. If a file with badname flag has a valid cipher file, this is returned
(=File just ends with the badname flag)

3. If a file with a badname flag exists where only the badname flag was added,
this is returned (=File cipher name could not be decrypted by function
`DecryptName` and just the badname flag was added)

4. Search for all files which cipher file name extists when cropping more and
more characters from the end. If only 1 file is found, return this

5. Return an error otherwise

This allows file access in the file browsers but most important it allows that
you rename files with undecryptable cipher names in the plain directories.
Renaming those files will then generate a proper cipher filename One
backdraft: When mounting the cipher dir with -badname parameter, you can never
create (or rename to) files whose file name ends with the badname file flag
(at the moment this is " GOCRYPTFS_BAD_NAME"). This will cause an error.

I modified the CLI test function to cover additional test cases. Test [Case
7](https://github.com/DerDonut/gocryptfs/blob/badnamecontent/tests/cli/cli_test.go#L712)
cannot be performed since the cli tests are executed in panic mode. The
testing is stopped on error. Since the function`DecryptName` produces internal
errors when hitting non-decryptable file names, this test was omitted.

This implementation is a proposal where I tried to change the minimum amount
of existing code. Another possibility would be instead of creating the new
function `EncryptAndHashBadName` to modify the signature of the existing
function `EncryptAndHashName(name string, iv []byte)` to
`EncryptAndHashName(name string, iv []byte, dirfd int)` and integrate the
functionality into this function directly. You may allow calling with dirfd=-1
or other invalid values an then performing the current functionality.
2021-06-20 18:09:21 +02:00
Jakob Unterwurzacher cdddd1d711 MANPAGE: describe -badname 2021-06-20 18:09:21 +02:00
Jakob Unterwurzacher 9ac77d42d1 contrib/maxlen.bash: also test dir and path length
Move the script from tests to contrib as it may now
be useful to somebody else.

https://github.com/rfjakob/gocryptfs/issues/552
2021-06-20 18:09:21 +02:00
Marcel Bochtler 2e738efe86 README: Add MacPorts install instructions
See [1] for the Portfile.

[1]: https://github.com/macports/macports-ports/blob/master/fuse/gocryptfs/Portfile.
2021-06-19 20:22:20 +02:00
Marcel Bochtler 7889346947 README: Rename Mac OS X to its latest name
See: https://www.apple.com/macos.
2021-06-19 20:22:20 +02:00
Jakob Unterwurzacher 98ab59c96e go.mod: update go-fuse
Memory compaction was merged
( 24a1dfe6b4 )

Fixes https://github.com/rfjakob/gocryptfs/issues/569
2021-06-11 15:24:39 +02:00
Jakob Unterwurzacher 9046d6d922 README: sync compile instructions with gocryptfs-website 2021-06-08 10:07:49 +02:00
Jakob Unterwurzacher 3dad9e0648 README: recommend build-without-openssl.bash
For build.bash, list missing dependencies: gcc, pkg-config

Fixes https://github.com/rfjakob/gocryptfs/issues/575
2021-06-08 09:55:04 +02:00
Jakob Unterwurzacher 14d422dc9c README: update for v2.0.1 release 2021-06-07 09:34:18 +02:00
Jakob Unterwurzacher a08ef90811 crossbuild.bash: skip Apple Silicon build on old Go versions 2021-06-06 21:10:24 +02:00
Jakob Unterwurzacher e44d4fcb96 crossbuild.bash: disable CGO
build-without-openssl.bash also disables CGO, so
this makes it more real-world-y.

But the real reason is that disabling CGO hopefully
fixes this travis ci build failure:

+GOOS=darwin
+GOARCH=arm64
+go build -tags without_openssl
/home/travis/.gimme/versions/go1.13.15.linux.amd64/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: unrecognized option '-pagezero_size'
/usr/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status
The command "./crossbuild.bash" exited with 2.
2021-06-06 19:28:02 +02:00
Jakob Unterwurzacher 17f859d3c4 fusefronted: report plaintext size on symlink creation
gocryptfs 2.0 introduced the regression that the size
reported at symlink creation was the ciphertext size,
which is wrong.

Report the plaintext size.

Fixes https://github.com/rfjakob/gocryptfs/issues/574
2021-06-06 19:22:16 +02:00
Jakob Unterwurzacher 6910f86705 crossbuild.bash: also build for Apple M1 2021-06-05 15:07:18 +02:00
Jakob Unterwurzacher e48f2377ec syscallcompat: drop obsolete wrappers
These are now available cross-platform in the unix
package.
2021-06-05 15:06:30 +02:00
Jakob Unterwurzacher c0e7530216 Update README for v2.0 2021-06-05 14:47:57 +02:00
Jakob Unterwurzacher 5ed1a90c7e doc: add benchmark for v2.0, reformat table 2021-06-05 14:46:49 +02:00
Jakob Unterwurzacher 3a1f009c1a Add contrib/atomicrename
$ ./contrib/atomicrename/atomicrename -h
atomicrename creates 100 "src" files in the current directory, renames
them in random order over a single "dst" file while reading the "dst"
file concurrently in a loop.

Progress and errors are reported as they occour in addition to a summary
printed at the end. cifs and fuse filesystems are known to fail, local
filesystems and nfs seem ok.

See https://github.com/hanwen/go-fuse/issues/398 for background info.
2021-06-04 22:17:13 +02:00
Jakob Unterwurzacher 015cd066e1 fido2: quote argument strings in debug output
Tested using

  gocryptfs -init -debug -fido2 "hello world" cipherdir

Output before:

  callFidoCommand: executing "/usr/bin/fido2-cred" with args [fido2-cred -M -h -v hello world]

After:

  callFidoCommand: executing "/usr/bin/fido2-cred" with args ["fido2-cred" "-M" "-h" "-v" "hello world"]

Related: https://github.com/rfjakob/gocryptfs/issues/571
2021-06-03 22:03:21 +02:00
Jakob Unterwurzacher 8f2be5d93c fsck: mark temporary mount read-only
We don't write during fsck, but somebody else might try
to.
2021-06-02 19:32:20 +02:00
Jakob Unterwurzacher 307dfd7f71 fsck: clean up temporary mountpoint
We used to leave directories like

  /tmp/gocryptfs.fsck.104431245

behind. Let's clean up after ourselves.
2021-06-02 19:31:55 +02:00
Jakob Unterwurzacher a38e5988ba fusefrontend: run acl Setxattr in user context
The result of setting an acl depends on who runs the
operation!

Fixes fuse-xfstests generic/375
(see https://github.com/rfjakob/fuse-xfstests/wiki/results_2021-05-19)
2021-06-02 19:10:36 +02:00
Jakob Unterwurzacher b23e21f61f fusefrontend: catch ReadAt integer overflow
Discovered by xfstests generic/564 .

Failure was:

generic/564	- output mismatch (see /opt/fuse-xfstests/results//generic/564.out.bad)
    --- tests/generic/564.out	2021-05-08 21:11:05.307395966 +0200
    +++ /opt/fuse-xfstests/results//generic/564.out.bad	2021-05-19 19:01:16.912888879 +0200
    @@ -31,7 +31,7 @@
     source range beyond 8TiB returns 0

     destination range beyond 8TiB returns EFBIG
    -copy_range: File too large
    +copy_range: Function not implemented
2021-06-02 18:20:05 +02:00
Jakob Unterwurzacher 04858ddd22 nametransform: check name validity on encryption
xfstests generic/523 discovered that we allowed to set
xattrs with "/" in the name, but did not allow to read
them later.

With this change we do not allow to set them in the first
place.
2021-06-02 14:29:48 +02:00
Jakob Unterwurzacher 242cdf966f go.mod: update go-fuse to get acl fixes
Done using:

go get github.com/hanwen/go-fuse/v2@master
go mod tidy
2021-05-30 09:34:03 +02:00
Jakob Unterwurzacher 198eb9797b Revert "go.mod: switch to go-fuse acl branch"
Change https://review.gerrithub.io/c/hanwen/go-fuse/+/516154
was merged upstream.

This reverts commit 3374afccc4.
2021-05-30 09:31:56 +02:00
Jakob Unterwurzacher 0e37fbe042 tests: TestFileHoleCopy: accept +-4kB
Failure looked like this:

--- FAIL: TestFileHoleCopy (3.73s)
    --- FAIL: TestFileHoleCopy/k81 (0.04s)
        file_holes_test.go:93: size changed: st0.Blocks=88 st2.Blocks=96
    file_holes_test.go:147: aborting further subtests

$ findholes TestFileHoleCopy.k81.1
         0 data
     36864 hole
     45056 data
     50434 hole
     50434 eof

$ findholes TestFileHoleCopy.k81.2
         0 data
     36864 hole
     45056 data
     50434 hole
     50434 eof

$ filefrag -v TestFileHoleCopy.k81.1
Filesystem type is: ef53
File size of TestFileHoleCopy.k81.1 is 50434 (13 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       2:   23702311..  23702313:      3:
   1:        3..       8:   20389855..  20389860:      6:   23702314:
   2:       11..      12:   23702314..  23702315:      2:   20389863: last,eof
TestFileHoleCopy.k81.1: 3 extents found

$ filefrag -v TestFileHoleCopy.k81.2
Filesystem type is: ef53
File size of TestFileHoleCopy.k81.2 is 50434 (13 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..       2:   20389861..  20389863:      3:
   1:        3..       4:   23702316..  23702317:      2:   20389864:
   2:        5..       6:   20389864..  20389865:      2:   23702318:
   3:        7..       8:   23702318..  23702319:      2:   20389866:
   4:       11..      12:   23702320..  23702321:      2:             last,eof
TestFileHoleCopy.k81.2: 4 extents found
2021-05-29 16:56:20 +02:00
Jakob Unterwurzacher 18befda0e6 fusefrontend: list "." and ".." in dir entries
Fixes xfstests generic/401
2021-05-29 16:44:38 +02:00
Jakob Unterwurzacher 738a9e006a fusefrontend: rewrite Lseek SEEK_DATA / SEEK_HOLE
In response to the discussion of the xfstests mailing list [1],
I looked at the Lseek implementation, which was naive and
did not handle all cases correctly.

The new implementation aligns the returned values to 4096 bytes
as most callers expect.

A lot of tests are added to verify that we handle all
cases correctly now.

[1]: https://www.spinics.net/lists/fstests/msg16554.html
2021-05-29 16:05:36 +02:00
Jakob Unterwurzacher c1d7e38761 findholes: add --create, --verify
Also, change the logic for the segment walk to not
rely on the total size. cp does not use the total
size either, and we miss bugs by cheating!
2021-05-29 16:00:40 +02:00
Jakob Unterwurzacher 07164cbb3a contentenc: add PlainOffToCipherOff helper
Will be used for improving Lseek()
2021-05-26 18:28:59 +02:00
Jakob Unterwurzacher b4794bedec contentenc: fix CipherSizeToPlainSize non-monoticity
For an illegal cipherSize, pretend we have an additional
1-byte block.

See code comment for details.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher bebd7ed81f contentenc: update comments
Also, replace one open-coded calculation with a
helper function.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 2a5ac3e9ba tests: contentenc: add TestSizeToSize
TestSizeToSize tests CipherSizeToPlainSize and PlainSizeToCipherSize.

Fails at the moment due to CipherSizeToPlainSize non-moniticity.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 9695f0e524 tests: add TestFileHoleCopy
Currently fails.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 629d0a5ad7 tests: re-enable TestInoReuseEvil
Problem in go-fuse has long been fixed.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 8bccd3b4bf Add contrib/findholes
Utility and libs to find hole/data segments using lseek.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 1b3c3b1347 syscallcompat: add GetdentsSpecial()
GetdentsSpecial calls then Getdents syscall,
with normal entries and "." / ".." split into two slices.
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 7d72baca69 tests: add TestHaveDotdot
As discovered by xfstests generic/401 [1], during the move to
the v2 api we seem to have lost the "." and ".." directory
entries.

[1]: 4ef5b032bc/screenlog.0 (L520)
2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher 1d2ac1e589 stupidgcm: prefer Go stdlib over OpenSSL on Apple M1
https://github.com/rfjakob/gocryptfs/issues/556
2021-05-26 09:20:22 +02:00