hardcoresushi/libgocryptfs
1
0
Fork 0
Code Issues Pull Requests Releases Activity
1,992 Commits 2 Branches 0 Tags 6 MiB
e9ecff7f07
Commit Graph

847 Commits

Author SHA1 Message Date
Jakob Unterwurzacher
84e702126a fusefrontend: implement recursive diriv caching 
The new contrib/maxlen.bash showed that we have exponential
runtime with respect to directory depth.

The new recursive diriv caching is a lot smarter as it caches
intermediate lookups. maxlen.bash now completes in a few seconds.

xfstests results same as
2d158e4c82/screenlog.0 :

  Failures: generic/035 generic/062 generic/080 generic/093 generic/099 generic/215 generic/285 generic/319 generic/426 generic/444 generic/467 generic/477 generic/523
  Failed 13 of 580 tests

benchmark.bash results are identical:

  $ ./benchmark.bash
  Testing gocryptfs at /tmp/benchmark.bash.BdQ: gocryptfs v2.0.1-17-g6b09bc0; go-fuse v2.1.1-0.20210611132105-24a1dfe6b4f8; 2021-06-25 go1.16.5 linux/amd64
  /tmp/benchmark.bash.BdQ.mnt is a mountpoint
  WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 0,4821 s, 544 MB/s
  READ:  262144000 bytes (262 MB, 250 MiB) copied, 0,266061 s, 985 MB/s
  UNTAR: 8,280
  MD5:   4,564
  LS:    1,745
  RM:    2,244
 2021-06-25 13:56:53 +02:00
Jakob Unterwurzacher
05b813f202 nametransform: rename BadNameFlag to BadnameSuffix 2021-06-21 12:12:44 +02:00
Jakob Unterwurzacher
689b74835b nametransform: gather badname functions in badname.go 2021-06-21 12:10:04 +02:00
Jakob Unterwurzacher
2efef1e270 nametransform: delete NameTransformer interface 
Useless layer of indirection.
 2021-06-21 11:53:33 +02:00
Jakob Unterwurzacher
6b0e63c1a8 Improve startup debug output 
The startup debug output was very verbose but still missing some
effective crypto settings.
 2021-06-21 11:32:04 +02:00
Jakob Unterwurzacher
c5d8fa83ae nametransform: pass badname patterns via New 
This means we can unexport the field.
 2021-06-20 19:09:46 +02:00
Jakob Unterwurzacher
50630e9f3d fido2: hide "FIDO2" in gocryptfs.conf if not used 
Result of:

$ gocryptfs -init foo
$ cat foo/gocryptfs.conf

Before:

{
	"Creator": "gocryptfs v2.0.1",
	"EncryptedKey": "FodEdNHD/cCwv1n5BuyAkbIOnJ/O5gfdCh3YssUCJ2DUr0A8DrQ5NH2SLhREeWRL3V8EMiPO2Ncr5IVwE4SSxQ==",
	"ScryptObject": {
		"Salt": "brGaw9Jg1kbPuSXFiwoxqK2oXFTgbniSgpiB+cu+67Y=",
		"N": 65536,
		"R": 8,
		"P": 1,
		"KeyLen": 32
	},
	"Version": 2,
	"FeatureFlags": [
		"GCMIV128",
		"HKDF",
		"DirIV",
		"EMENames",
		"LongNames",
		"Raw64"
	],
	"FIDO2": {
		"CredentialID": null,
		"HMACSalt": null
	}
}

After:

{
	"Creator": "gocryptfs v2.0.1-5-gf9718eb-dirty.DerDonut-badnamecontent",
	"EncryptedKey": "oFMj1lS1ZsM/vEfanNMeCTPw3PZr5VWeL7ap8Jd8YQm6evy2BAhtQ/pd6RzDx84wlCz9TpxqHRihuwSEMnOWWg==",
	"ScryptObject": {
		"Salt": "JZ/5mhy4a8EAQ/wDF1POIEe4/Ss38cfJgXgj26DuA4M=",
		"N": 65536,
		"R": 8,
		"P": 1,
		"KeyLen": 32
	},
	"Version": 2,
	"FeatureFlags": [
		"GCMIV128",
		"HKDF",
		"DirIV",
		"EMENames",
		"LongNames",
		"Raw64"
	]
}
 2021-06-20 18:09:21 +02:00
DerDonut
a611810ff4 Badname file content access 
This proposal is the counterpart of the modifications from the `-badname`
parameter. It modifies the plain -> cipher mapping for filenames when using
`-badname` parameter. The new function `EncryptAndHashBadName` tries to find a
cipher filename for the given plain name with the following steps:

1. If `badname` is disabled or direct mapping is successful: Map directly
(default and current behaviour)

2. If a file with badname flag has a valid cipher file, this is returned
(=File just ends with the badname flag)

3. If a file with a badname flag exists where only the badname flag was added,
this is returned (=File cipher name could not be decrypted by function
`DecryptName` and just the badname flag was added)

4. Search for all files which cipher file name extists when cropping more and
more characters from the end. If only 1 file is found, return this

5. Return an error otherwise

This allows file access in the file browsers but most important it allows that
you rename files with undecryptable cipher names in the plain directories.
Renaming those files will then generate a proper cipher filename One
backdraft: When mounting the cipher dir with -badname parameter, you can never
create (or rename to) files whose file name ends with the badname file flag
(at the moment this is " GOCRYPTFS_BAD_NAME"). This will cause an error.

I modified the CLI test function to cover additional test cases. Test [Case
7](https://github.com/DerDonut/gocryptfs/blob/badnamecontent/tests/cli/cli_test.go#L712)
cannot be performed since the cli tests are executed in panic mode. The
testing is stopped on error. Since the function`DecryptName` produces internal
errors when hitting non-decryptable file names, this test was omitted.

This implementation is a proposal where I tried to change the minimum amount
of existing code. Another possibility would be instead of creating the new
function `EncryptAndHashBadName` to modify the signature of the existing
function `EncryptAndHashName(name string, iv []byte)` to
`EncryptAndHashName(name string, iv []byte, dirfd int)` and integrate the
functionality into this function directly. You may allow calling with dirfd=-1
or other invalid values an then performing the current functionality.
 2021-06-20 18:09:21 +02:00
Jakob Unterwurzacher
17f859d3c4 fusefronted: report plaintext size on symlink creation 
gocryptfs 2.0 introduced the regression that the size
reported at symlink creation was the ciphertext size,
which is wrong.

Report the plaintext size.

Fixes https://github.com/rfjakob/gocryptfs/issues/574
 2021-06-06 19:22:16 +02:00
Jakob Unterwurzacher
e48f2377ec syscallcompat: drop obsolete wrappers 
These are now available cross-platform in the unix
package.
 2021-06-05 15:06:30 +02:00
Jakob Unterwurzacher
015cd066e1 fido2: quote argument strings in debug output 
Tested using

  gocryptfs -init -debug -fido2 "hello world" cipherdir

Output before:

  callFidoCommand: executing "/usr/bin/fido2-cred" with args [fido2-cred -M -h -v hello world]

After:

  callFidoCommand: executing "/usr/bin/fido2-cred" with args ["fido2-cred" "-M" "-h" "-v" "hello world"]

Related: https://github.com/rfjakob/gocryptfs/issues/571
 2021-06-03 22:03:21 +02:00
Jakob Unterwurzacher
a38e5988ba fusefrontend: run acl Setxattr in user context 
The result of setting an acl depends on who runs the
operation!

Fixes fuse-xfstests generic/375
(see https://github.com/rfjakob/fuse-xfstests/wiki/results_2021-05-19)
 2021-06-02 19:10:36 +02:00
Jakob Unterwurzacher
b23e21f61f fusefrontend: catch ReadAt integer overflow 
Discovered by xfstests generic/564 .

Failure was:

generic/564	- output mismatch (see /opt/fuse-xfstests/results//generic/564.out.bad)
    --- tests/generic/564.out	2021-05-08 21:11:05.307395966 +0200
    +++ /opt/fuse-xfstests/results//generic/564.out.bad	2021-05-19 19:01:16.912888879 +0200
    @@ -31,7 +31,7 @@
     source range beyond 8TiB returns 0

     destination range beyond 8TiB returns EFBIG
    -copy_range: File too large
    +copy_range: Function not implemented
 2021-06-02 18:20:05 +02:00
Jakob Unterwurzacher
04858ddd22 nametransform: check name validity on encryption 
xfstests generic/523 discovered that we allowed to set
xattrs with "/" in the name, but did not allow to read
them later.

With this change we do not allow to set them in the first
place.
 2021-06-02 14:29:48 +02:00
Jakob Unterwurzacher
18befda0e6 fusefrontend: list "." and ".." in dir entries 
Fixes xfstests generic/401
 2021-05-29 16:44:38 +02:00
Jakob Unterwurzacher
738a9e006a fusefrontend: rewrite Lseek SEEK_DATA / SEEK_HOLE 
In response to the discussion of the xfstests mailing list [1],
I looked at the Lseek implementation, which was naive and
did not handle all cases correctly.

The new implementation aligns the returned values to 4096 bytes
as most callers expect.

A lot of tests are added to verify that we handle all
cases correctly now.

[1]: https://www.spinics.net/lists/fstests/msg16554.html
 2021-05-29 16:05:36 +02:00
Jakob Unterwurzacher
07164cbb3a contentenc: add PlainOffToCipherOff helper 
Will be used for improving Lseek()
 2021-05-26 18:28:59 +02:00
Jakob Unterwurzacher
b4794bedec contentenc: fix CipherSizeToPlainSize non-monoticity 
For an illegal cipherSize, pretend we have an additional
1-byte block.

See code comment for details.
 2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher
bebd7ed81f contentenc: update comments 
Also, replace one open-coded calculation with a
helper function.
 2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher
2a5ac3e9ba tests: contentenc: add TestSizeToSize 
TestSizeToSize tests CipherSizeToPlainSize and PlainSizeToCipherSize.

Fails at the moment due to CipherSizeToPlainSize non-moniticity.
 2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher
1b3c3b1347 syscallcompat: add GetdentsSpecial() 
GetdentsSpecial calls then Getdents syscall,
with normal entries and "." / ".." split into two slices.
 2021-05-26 13:17:56 +02:00
Jakob Unterwurzacher
1d2ac1e589 stupidgcm: prefer Go stdlib over OpenSSL on Apple M1 
https://github.com/rfjakob/gocryptfs/issues/556
 2021-05-26 09:20:22 +02:00
Jakob Unterwurzacher
09870bfac5 syscallcompat: also refactor MkdiratUser on GOOS=darwin 
Breakage was:

+GOOS=darwin
+GOARCH=amd64
+go build -tags without_openssl
internal/fusefrontend/node_dir_ops.go:45:34: cannot use context (type *fuse.Context) as type *fuse.Caller in argument to syscallcompat.MkdiratUser
internal/fusefrontend/node_dir_ops.go:83:35: cannot use context (type *fuse.Context) as type *fuse.Caller in argument to syscallcompat.MkdiratUser
 2021-05-22 22:01:46 +02:00
Jakob Unterwurzacher
e1853e1011 syscallcompat: refactor MkdiratUser to take fuse.Context 
Let's have MkdiratUser take fuse.Context like everybody
else.
 2021-05-22 21:44:19 +02:00
Jakob Unterwurzacher
cb4f9f9e29 syscallcompat: deduplicate OpenatUser/MknodatUser/SymlinkatUser/MkdiratUser 
Turns out the whole euid switching logic can be shared when
wrapping the syscall in a closure.
 2021-05-22 21:39:29 +02:00
Jakob Unterwurzacher
0650a512bb fsck: redirect go-fuse noise to syslog 2021-05-18 18:38:23 +02:00
Jakob Unterwurzacher
5da20da977 -speed: note that XChaCha20 is not selectable 
This is unclear to users, as seen on
https://github.com/rfjakob/gocryptfs/issues/452#issuecomment-828836651
 2021-05-18 09:53:23 +02:00
Jakob Unterwurzacher
a91ad29d36 fusefrontend: fix RENAME_NOREPLACE darwin build failure 
Error was:

internal/fusefrontend/node.go:371:2: duplicate case syscallcompat.RENAME_NOREPLACE (value 0) in switch
	previous case at internal/fusefrontend/node.go:368:7

Rewrite using "if"s instead.
 2021-05-15 17:20:43 +02:00
Jakob Unterwurzacher
1ba2e42234 fusefrontend: avoid duplicate const definition 
RENAME_NOREPLACE is already defined in syscallcompat.
 2021-05-08 20:42:42 +02:00
Jakob Unterwurzacher
a267562d29 fusefrontend: reject broken RENAME_EXCHANGE and RENAME_WHITEOUT 
Discovered by xfstests generic/013: or implementation for
RENAME_EXCHANGE and RENAME_WHITEOUT is incomplete. Reject the
flags so that the caller retries with regular rename.
 2021-05-08 15:39:49 +02:00
Jakob Unterwurzacher
4a07d6598c fusefrontend: make dirCache work for "node itself" 
"node itself" can be converted to node + child by
ascending one level.

Performance gains are spectacular, as will be seen
in the next commit.
 2021-04-07 07:18:35 +02:00
Jakob Unterwurzacher
f73aee72f8 fusefrontend: print dirCache stats after unmount 2021-04-05 18:20:17 +02:00
Jakob Unterwurzacher
043f81dd01 fs: more idiomatic dirCache struct naming 2021-04-04 13:05:47 +02:00
Jakob Unterwurzacher
24d5d39300 fs: add initial dirfd caching 
dirfd caching was temporarily removed when moving
to the v2api. Add it back to gain back some lost speed.
 2021-04-03 13:08:28 +02:00
Jakob Unterwurzacher
6aae2aad97 tests: fusefronted: fix TestOpenBackingDir 
This test only worked accidentially, and would
break once dirfd caching is added.

fs.Mkdir(..., "dir1/dir2", ...) is illegal
(child name cannot contain slashes).
 2021-04-03 12:34:18 +02:00
Jakob Unterwurzacher
6b28c0be2c fusefronted: replace last rn.openBackingDir() calls 
Use the n.prepareAtSyscall() wrapper instead. Prepares
for adding caching into n.prepareAtSyscall().
 2021-04-03 11:19:04 +02:00
Jakob Unterwurzacher
3b9a1b628b fusefronted: move Create() and Open() to new file 
And deduplicate the code a little.
 2021-03-21 09:31:05 +01:00
Jakob Unterwurzacher
47a4d33f24 fusefrontend: -sharedstorage: fix TestRmwRace failure 
The Open() and Create() paths used different inode numbers,
which broke openfiletable locking against concurred readers.
 2021-03-21 09:02:50 +01:00
Jakob Unterwurzacher
f6036c429a syscallcompat: getdents: link to #483 
Give a user receiving the Getdents warning some background info.
 2021-03-14 14:43:11 +01:00
Jakob Unterwurzacher
e2dc52a965 v2api: -sharestorage: disable hard link tracking & add tests 
Hard link tracking was not correctly disabled
since the migration to the go-fuse v2 api.

Add a test to ensure it stays off.

Fixes https://github.com/rfjakob/gocryptfs/issues/525
 2021-03-07 17:22:29 +01:00
Jakob Unterwurzacher
eaca820e87 fusefrontend: do not encrypt ACLs 
Pass through system.posix_acl_access and system.posix_acl_default
unencrypted to fix "cp -a" problems.

"cp -a" uses "setxattr" even to set normal permissions, see
https://www.spinics.net/lists/linux-nfs/msg63986.html .

Fixes https://github.com/rfjakob/gocryptfs/issues/543
 2021-02-07 20:01:16 +01:00
Jakob Unterwurzacher
80a651a194 syscallcompat: MknodatUser: work around changed syscall.Setgroups semantics 
Since go1.16beta1 (commit d1b1145cace8b968307f9311ff611e4bb810710c ,
https://go-review.googlesource.com/c/go/+/210639 )
syscall.{Setgroups,Setregid,Setreuid} affects all threads, which
is exactly what we not want.

We now use unix.{Setgroups,Setregid,Setreuid} instead.

Workarounds https://github.com/golang/go/issues/1435 .
 2021-02-06 11:38:25 +01:00
Jakob Unterwurzacher
bed60101f4 nametransform: make gocryptfs.diriv and gocryptfs.xxx.name files world-readable 
Make `gocryptfs.diriv` and `gocryptfs.xxx.name` files world-readable to make encrypted backups easier
when mounting via fstab.

Having the files follow chmod/chown of their parent does not seem
to be worth the hassle. The content of the diriv files is not
secret, and both diriv and name files are protected by the
perms of the parent dir.

Fixes https://github.com/rfjakob/gocryptfs/issues/539
 2021-01-10 08:07:10 +01:00
Jakob Unterwurzacher
f3394ae286 nametransform: move permission constants to perms.go 
Prep for https://github.com/rfjakob/gocryptfs/issues/539
 2021-01-10 07:27:04 +01:00
gmd20
c20c7992a0 main: add "-kernel_cache" flag 
This option is similar to fuse(8) kernel_cache

Verified using vmtouch.

Without -kernel_cache:

$ dd if=/dev/zero of=foo bs=1M count=10 ; vmtouch -t foo ; vmtouch foo
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0,0242321 s, 433 MB/s
           Files: 1
     Directories: 0
   Touched Pages: 2560 (10M)
         Elapsed: 0.011159 seconds
           Files: 1
     Directories: 0
  Resident Pages: 0/2560  0/10M  0%
         Elapsed: 0.000993 seconds

With -kernel_cache:

$ dd if=/dev/zero of=foo bs=1M count=10 ; vmtouch -t foo ; vmtouch foo
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0,0244015 s, 430 MB/s
           Files: 1
     Directories: 0
   Touched Pages: 2560 (10M)
         Elapsed: 0.011564 seconds
           Files: 1
     Directories: 0
  Resident Pages: 2560/2560  10M/10M  100%
         Elapsed: 0.000369 seconds
 2020-12-20 09:55:04 +01:00
Jakob Unterwurzacher
1c1692c4d9 fusefrontend_reverse: fix GETATTR panic 
We don't implement Getattr(), so don't try to call it.

Reported at https://github.com/rfjakob/gocryptfs/issues/519#issuecomment-718790790 :

15:22:53.414101 rx 3344: READ n2565 {Fh 7 [42143744 +131072)  L 0 RDONLY,0x8000}
15:22:53.414274 rx 3342: READ n2565 {Fh 7 [42012672 +131072)  L 0 RDONLY,0x8000}
15:22:53.414787 rx 3346: READ n2565 {Fh 7 [42274816 +131072)  L 0 RDONLY,0x8000}
15:22:53.414806 tx 3336:     OK,  131072b data "\xcb\xd3<\"!-\xa7\xc4"...
15:22:53.414874 rx 3348: GETATTR n1446 {Fh 0}
panic: interface conversion: *fusefrontend_reverse.File is not fs.FileGetattrer: missing method Getattr

goroutine 451 [running]:
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*Node).Getattr(0xc00034c880, 0x5620579784c0, 0xc000593e60, 0x562057939800, 0xc000218050, 0xc0000fc108, 0x0)
        github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/node.go:69 +0x273
github.com/hanwen/go-fuse/v2/fs.(*rawBridge).getattr(0xc00011e000, 0x5620579784c0, 0xc000593e60, 0xc00034c880, 0x562057939800, 0xc000218050, 0xc0000fc108, 0xbfded1ef58ba7b13)
        github.com/hanwen/go-fuse/v2@v2.0.4-0.20200908172753-0b6cbc515082/fs/bridge.go:500 +0x2d4
github.com/hanwen/go-fuse/v2/fs.(*rawBridge).GetAttr(0xc00011e000, 0xc0000e0000, 0xc0000fc198, 0xc0000fc108, 0x0)
        github.com/hanwen/go-fuse/v2@v2.0.4-0.20200908172753-0b6cbc515082/fs/bridge.go:488 +0x11c
github.com/hanwen/go-fuse/v2/fuse.doGetAttr(0xc000120000, 0xc0000fc000)
        github.com/hanwen/go-fuse/v2@v2.0.4-0.20200908172753-0b6cbc515082/fuse/opcode.go:287 +0x6f
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc000120000, 0xc0000fc000, 0xc000000000)
        github.com/hanwen/go-fuse/v2@v2.0.4-0.20200908172753-0b6cbc515082/fuse/server.go:472 +0x2c1
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc000120000, 0xc000288001)
        github.com/hanwen/go-fuse/v2@v2.0.4-0.20200908172753-0b6cbc515082/fuse/server.go:445 +0x18c
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest
        github.com/hanwen/go-fuse/v2@v2.0.4-0.20200908172753-0b6cbc515082/fuse/server.go:312 +0x419
 2020-11-10 19:37:49 +01:00
Jakob Unterwurzacher
832e58cad4 Drop two more generated files 
These were committed by mistake.
 2020-10-19 19:27:47 +02:00
Jakob Unterwurzacher
165bf6c849 Drop generated files 
These were committed by mistake.
 2020-10-19 19:25:47 +02:00
Jakob Unterwurzacher
6697ffd6e2 fusefronted: reject GETXATTR "security.capability" 
Unless we are mounted with -suid, we can reject
these requests, and gain back some lost speed.

Closes https://github.com/rfjakob/gocryptfs/issues/515
 2020-10-18 21:07:30 +02:00
Jakob Unterwurzacher
c943ed32aa syscallcompat: add getxattr fastpaths 
The allocations from Lgetxattr were #1 in the tar extract
allocation profile (caused by security.capability lookups).
No more!

$ benchstat old.txt new.txt
name         old time/op  new time/op  delta
Lgetxattr-4  15.2µs ± 0%   1.8µs ± 0%   ~     (p=1.000 n=1+1)

$ ./benchmark.bash
Testing gocryptfs at /tmp/benchmark.bash.H8p: gocryptfs v2.0-beta1-4-g95ea738-dirty; go-fuse v2.0.4-0.20200908172753-0b6cbc515082 => github.com/rfjakob/go-fuse/v2 v2.0.4-0.20201015204057-88b12c99f8af; 2020-10-18 go1.15.3 linux/amd64
/tmp/benchmark.bash.H8p.mnt is a mountpoint
WRITE: 262144000 bytes (262 MB, 250 MiB) copied, 0,520109 s, 504 MB/s
READ:  262144000 bytes (262 MB, 250 MiB) copied, 0,255672 s, 1,0 GB/s
UNTAR: 30,238
MD5:   12,721
LS:    10,038
RM:    16,536
 2020-10-18 00:25:42 +02:00
Jakob Unterwurzacher
83a324a46b syscallcompat: add Lgetxattr benchmark 2020-10-16 20:04:22 +02:00
Jakob Unterwurzacher
fcb28e4ff3 v2pai: delete fusefrontend_reverse_v1api 
Served its mission a copy-paste source but can now be deleted.
 2020-10-15 23:18:21 +02:00
Jakob Unterwurzacher
ec3eaf0b87 syscallcompat: don't retry Close() 
After Close() returns, the fd is dead, even if we
received EINTR. Don't retry, we could shoot down
an unrelated fd that received the same fd number.
 2020-10-14 13:40:12 +02:00
Jakob Unterwurzacher
af4c1fb7a3 syscallcompat: retry ops on EINTR 
Retry operations that have been shown to throw EINTR
errors on CIFS.

Todo: Solution for this pain in the back:

	warning: unix.Getdents returned errno 2 in the middle of data
	rm: cannot remove 'linux-3.0.old3/Documentation/ABI/removed': Input/output error

Progress towards fixing https://github.com/rfjakob/gocryptfs/issues/483 .
 2020-10-14 00:35:16 +02:00
Jakob Unterwurzacher
803fdf410b syscallcompat: Openat: retry on EINTR 
Towards fixing https://github.com/rfjakob/gocryptfs/issues/507
 2020-10-11 01:31:09 +02:00
Pavol Rusnak
1e624a4cc3 Add support for FIDO2 tokens 2020-09-12 18:06:54 +02:00
Jakob Unterwurzacher
ac687d5359 v2api: add Darwin xattr support 2020-09-09 11:17:19 +02:00
Jakob Unterwurzacher
8b1df08b8a syscallcompat: add Renameat2 for Darwin 2020-09-09 11:16:29 +02:00
Jakob Unterwurzacher
598e5f385e v2api/reverse: update TODO comment on xattrs 2020-08-30 16:39:46 +02:00
Jakob Unterwurzacher
d212b246c5 v2api/reverse: implement Lseek 2020-08-16 19:58:47 +02:00
Jakob Unterwurzacher
58a2726977 v2api/reverse: implement Statfs 2020-08-16 19:41:00 +02:00
Jakob Unterwurzacher
6f3cca2cdd v2api: clean up api TODOs 2020-08-16 13:52:27 +02:00
Jakob Unterwurzacher
ee5ab1cc29 v2api: rename "File2" to just "File" 
Rename the symbols and the files.
 2020-08-16 12:50:33 +02:00
Jakob Unterwurzacher
94e8fc12ea v2api/reverse: finish -exclude 
Tests pass now.
 2020-08-15 17:31:45 +02:00
Jakob Unterwurzacher
15b0b4a5fd v2api/reverse: start wiring up -exclude functionality 
Exclude in readdir is missing.
 2020-08-15 16:08:16 +02:00
Jakob Unterwurzacher
1ea1b179c2 v2api/reverse: fix two fd leaks 2020-08-15 15:28:40 +02:00
Jakob Unterwurzacher
35055030a1 v2api/reverse: implement ctlsocksrv.Interface 2020-08-15 15:08:28 +02:00
Jakob Unterwurzacher
b6580a87e5 v2api/reverse: implement gocryptfs.conf mapping with -plaintextnames 2020-08-15 15:05:25 +02:00
Jakob Unterwurzacher
6d4f1a6888 v2api/reverse: implement Read 2020-08-09 22:11:46 +02:00
Jakob Unterwurzacher
5276092663 v2api/reverse: implement Readlink 2020-08-08 18:45:47 +02:00
Jakob Unterwurzacher
84ed139cd2 v2api/reverse: implement Lookup for longname 2020-08-02 19:33:12 +02:00
Jakob Unterwurzacher
4674bac838 v2api/reverse: implement Lookup for gocryptfs.conf & gocryptfs.diriv 2020-08-02 13:25:53 +02:00
Jakob Unterwurzacher
47d8f56b7f v2api/reverse: add missing decryptPath call openBackingDir 2020-08-01 23:06:35 +02:00
Jakob Unterwurzacher
6c26cda531 v2api/reverse: implement Readdir 2020-08-01 22:28:25 +02:00
Jakob Unterwurzacher
f54d21c384 v2api/reverse: implement Lookup & Getattr 2020-08-01 21:14:33 +02:00
Jakob Unterwurzacher
18b3bdb158 v2api/reverse: start fusefrontend_reverse v2 API implementation 2020-08-01 21:14:17 +02:00
Jakob Unterwurzacher
13dc7657ba v2api/reverse: move old fusefrontend_reverse out of the way 
fusefrontend_reverse -> fusefrontend_reverse_v1api
 2020-08-01 21:14:10 +02:00
Jakob Unterwurzacher
dd3d8c100b fusefrontend_reverse: collapse getFileType 2020-08-01 20:28:06 +02:00
Jakob Unterwurzacher
1867fdaef4 v2api: move helpers from node.go to node_helpers.go 2020-08-01 19:26:42 +02:00
Jakob Unterwurzacher
4572cd2103 v2api: fix missing size translation in Lookup 2020-07-26 19:49:26 +02:00
Jakob Unterwurzacher
777b95f82f v2api: delete (most) fusefrontend v1 files 
All the functionality in these files has been reimplemented
for the v2 api. Drop the old files.
 2020-07-26 18:35:12 +02:00
Jakob Unterwurzacher
9cd24d79a2 v2api: implement Lseek 
This also fixes the last remaining tests/fsck failure.
 2020-07-23 22:55:07 +02:00
Jakob Unterwurzacher
49fc3abcb4 v2api: Lookup: use newChild() helper 2020-07-19 13:15:18 +02:00
Jakob Unterwurzacher
6b7ff09373 v2api: Node: make Path() public 
Helpful for fsck.
 2020-07-18 23:40:54 +02:00
Jakob Unterwurzacher
7eae35e2d3 v2api: implement ctlsocksrv.Interface 2020-07-17 22:14:40 +02:00
Jakob Unterwurzacher
57d572dbc1 v2api: implement Getxattr, Setxattr, Removexattr, Listxattr 
gocryptfs/tests/xattr passes.
 2020-07-14 19:55:20 +02:00
Jakob Unterwurzacher
4a0966e79e v2api: Setattr: fill out structure 2020-07-12 21:17:52 +02:00
Jakob Unterwurzacher
ca5acae6f0 v2api: fix Mkfifo 2020-07-12 20:19:29 +02:00
Jakob Unterwurzacher
0fa824933c v2api: properly implement Node.Setattr 
We used to always open a *File2 and letting the *File2
handle Setattr. This does not work it we cannot open the file!

Before:

    $ go test
    2020/07/12 20:14:57 writer: Write/Writev failed, err: 2=no such file or directory. opcode: INTERRUPT
    2020/07/12 20:14:57 writer: Write/Writev failed, err: 2=no such file or directory. opcode: INTERRUPT
    --- FAIL: TestLchown (0.00s)
        matrix_test.go:634: lchown /tmp/gocryptfs-test-parent-1026/014500839/default-plain/symlink: too many levels of symbolic links
    touch: setting times of '/tmp/gocryptfs-test-parent-1026/014500839/default-plain/utimesnano_symlink': Too many levels of symbolic links
    --- FAIL: TestUtimesNanoSymlink (0.00s)
        matrix_test.go:655: exit status 1
    --- FAIL: TestMkfifo (0.00s)
        matrix_test.go:755: file exists
    --- FAIL: TestMagicNames (0.00s)
        matrix_test.go:773: Testing n="gocryptfs.longname.QhUr5d9FHerwEs--muUs6_80cy6JRp89c1otLwp92Cs"
        matrix_test.go:773: Testing n="gocryptfs.diriv"
        matrix_test.go:815: open /tmp/gocryptfs-test-parent-1026/014500839/default-plain/linktarget: permission denied
    --- FAIL: TestChmod (0.00s)
        matrix_test.go:840: chmod 444 -> 000 failed: permission denied
        matrix_test.go:840: chmod 444 -> 111 failed: permission denied
        matrix_test.go:840: chmod 444 -> 123 failed: permission denied
        matrix_test.go:840: chmod 444 -> 321 failed: permission denied
    FAIL
    exit status 1
    FAIL	github.com/rfjakob/gocryptfs/tests/matrix	0.790s

After:

    $ go test
    --- FAIL: TestMkfifo (0.00s)
        matrix_test.go:755: file exists
    --- FAIL: TestMagicNames (0.00s)
        matrix_test.go:773: Testing n="gocryptfs.longname.QhUr5d9FHerwEs--muUs6_80cy6JRp89c1otLwp92Cs"
        matrix_test.go:773: Testing n="gocryptfs.diriv"
        matrix_test.go:815: open /tmp/gocryptfs-test-parent-1026/501766059/default-plain/linktarget: permission denied
    --- FAIL: TestChmod (0.00s)
        matrix_test.go:849: modeHave 0644 != modeWant 0
    FAIL
    exit status 1
    FAIL    github.com/rfjakob/gocryptfs/tests/matrix   0.787s
 2020-07-12 20:17:15 +02:00
Jakob Unterwurzacher
f11432d02a v2api: Getattr: use file handle if passed 2020-07-12 15:08:17 +02:00
Jakob Unterwurzacher
4d99b02279 v2api: Getattr: fix file size 2020-07-12 15:03:42 +02:00
Jakob Unterwurzacher
0d385846da v2api: call InitSerializer 
If we don't, reads hang with -serialize_reads.
 2020-07-12 15:02:53 +02:00
Jakob Unterwurzacher
230d92c4f4 v2api: fix Rename trying to overwrite itself 
We used to do this

  [pid 99182] renameat2(14, "Y_4DAxKvj1QnXmJx2AkrKA", 15, ".", RENAME_NOREPLACE <unfinished ...>

which was not the intention.
 2020-07-12 13:55:24 +02:00
Jakob Unterwurzacher
735e2aa65b v2api: fix Mkdir crash when using plaintextnames 2020-07-12 13:35:37 +02:00
Jakob Unterwurzacher
b1d631d432 v2api: fix double-lock in truncate 2020-07-12 13:13:24 +02:00
Jakob Unterwurzacher
6196a5b5fe v2api: File2: implement Release, Read, Write, Fsync, Flush, Allocate 
Fortunately, this just means fixing up the function
signatures.
 2020-07-12 12:59:01 +02:00
Jakob Unterwurzacher
d539a4c214 v2api: fix RootNode cast 2020-07-11 20:27:47 +02:00
Jakob Unterwurzacher
b0342fae5d v2api: fix TestOpenBackingDir 2020-07-11 20:15:47 +02:00
Jakob Unterwurzacher
7de3330d70 v2api: implement Rename 2020-07-11 19:56:45 +02:00
Jakob Unterwurzacher
84344834c4 v2api: remove OpenatUserCtx, MknodatUserCtx helpers 
Instead, use the new toFuseCtx() function introduced
in an earlier commit.
 2020-07-11 19:44:45 +02:00
Jakob Unterwurzacher
250dbc6436 v2api: implement Symlink 2020-07-11 19:43:07 +02:00
Jakob Unterwurzacher
c35b575d5f v2api: implement Link 2020-07-11 19:32:38 +02:00
Jakob Unterwurzacher
b971c75e67 v2api: implement Mknod 2020-07-11 19:23:04 +02:00
Jakob Unterwurzacher
855b4a95b7 v2api: implement Statfs 2020-07-11 18:59:54 +02:00
Jakob Unterwurzacher
40a31e0c32 v2api: list interfaces to be implemented 2020-07-05 20:22:24 +02:00
Jakob Unterwurzacher
63f68a0fcd v2api: implement Setattr 2020-07-05 20:10:53 +02:00
Jakob Unterwurzacher
c22e78ee41 v2api: implement Opendir 2020-07-05 19:34:30 +02:00
Jakob Unterwurzacher
1f4e554168 v2api: merge openBackingDir into root_node.go 2020-07-04 21:42:04 +02:00
Jakob Unterwurzacher
d2139e18ef v2api: implement Open() 2020-07-04 21:37:44 +02:00
Jakob Unterwurzacher
d73e4b3f7c v2api: add prepareAtSyscall helper 2020-07-04 21:16:20 +02:00
Jakob Unterwurzacher
23180794fe v2api: implement Readlink 2020-07-04 20:32:02 +02:00
Jakob Unterwurzacher
1618fbbac5 v2api: implement Unlink 2020-06-21 14:08:53 +02:00
Jakob Unterwurzacher
cc0b94a3c5 v2api: implement Rmdir 2020-06-21 13:57:04 +02:00
Jakob Unterwurzacher
192a29075a v2api: implement Mkdir 2020-06-21 13:46:08 +02:00
Jakob Unterwurzacher
f6ded09e36 v2api: implement Create 2020-06-21 13:25:12 +02:00
Jakob Unterwurzacher
74a4accf0c v2api: collect RootNode code in root_node.go 2020-06-21 12:44:32 +02:00
Jakob Unterwurzacher
ebdf58b9eb v2api: implement GetAttr and Readdir 2020-06-21 12:42:18 +02:00
Jakob Unterwurzacher
6aa9f5636f v2api: implement Lookup() 
Compiles, but untested otherwise. No caching.
 2020-06-21 12:01:34 +02:00
Jakob Unterwurzacher
9b8ce55383 inomap: clarify TranslateStat function comment 
Also, fix the namespaceMap comment.
 2020-06-21 12:01:34 +02:00
Jakob Unterwurzacher
2aad58f9ec v2api (go-fuse v2 api): initial noop implementation 
Compiles and mounts but does nothing useful.
 2020-06-21 12:01:24 +02:00
Jakob Unterwurzacher
261cf12829 badname: stop trying to decrypt at length 22 
At least 16 bytes due to AES --> at least 22 characters in base64
 2020-06-06 12:49:45 +02:00
DerDonut
a8230d271f Added auto decryption of invalid file names 
Changed invalid file decoding and decryption. Function
DecryptName now shortens the filename until the filename is
decodable and decryptable. Will work with valid **and**
invalid Base64URL delimiter (valid delimiter [0-9a-zA-z_\\-].
If the filename is not decryptable at all, it returns the
original cipher name with flag suffix Changed cli tests to
generate decryptable and undecryptable file names with correct
encrypted content. Replacing #474, extends #393
 2020-06-06 12:20:31 +02:00
Jakob Unterwurzacher
71c0481f0e Revert "fusefrontend: don't always clear the dircache in Rename" 
As noticed by @slackner in
cb8872577d (commitcomment-39405233) ,
this is not safe.

This reverts commit cb8872577d.
 2020-05-24 23:36:11 +02:00
Jakob Unterwurzacher
15ff79bf14 syscallcompat: warn when Getdents truncates data 
On CIFS mounts, unix.Getdents can return sudden ENOENT
in the middle of data. This will not be reported as an error
by user space tools, so return EIO instead.

Also log it as a warning.

https://github.com/rfjakob/gocryptfs/issues/483
 2020-05-24 23:30:25 +02:00
Jakob Unterwurzacher
cb8872577d fusefrontend: don't always clear the dircache in Rename 
When filename encryption is on, we do know when we
overwrite a directory, and can clear only in this case.

sshfs-benchmark.bash:    sshfs  gocryptfs-on-sshfs
git init                  1.74                7.80
rsync                     6.19               11.63
 2020-05-24 20:19:27 +02:00
Jakob Unterwurzacher
2a9b99a0ef fusefrontend: don't clear dircache on Mkdir 
Mkdir can not cause existing entries in the cache to go
stale. So don't clear it. Benchmark results:

sshfs-benchmark.bash:    sshfs  gocryptfs-on-sshfs
git init                  1.65                8.74
rsync                     6.09               17.54
 2020-05-24 15:30:14 +02:00
Jakob Unterwurzacher
25f1727de9 syscallcompat: getdents: retry on EINTR 
Fixes: https://github.com/rfjakob/gocryptfs/issues/483
Related: https://github.com/golang/go/issues/38836
 2020-05-23 22:54:23 +02:00
Jakob Unterwurzacher
f8ad2ac3e2 dircache: increase cache size & lifetime 
Looking at the dircache debug output, we see
that a "git status" workload has a very bad
cache hit rate because the entries expire or
get evicted before they can be reused.

Increase both cache size and lifetime for
a 4x speedup:

Before: 75s
After:  17s

https://github.com/rfjakob/gocryptfs/issues/410
 2020-05-17 21:37:36 +02:00
Jakob Unterwurzacher
f6088e5008 dircache: improve debug messages 
Before:

Lookup "errno.html/1/2/3/4/5": miss
Store: "errno.html/1/2/3/4/5" fd=26 iv=21be6e083d60dcabfe7368264d5082b7
Lookup "errno.html": hit 25 6d68a16d217978915036a3bd55428ae7
Lookup "errno.html/1": hit 25 932a464c299b3430c5e55c924f98ac4d
Lookup "errno.html/1/2": hit 25 7d53348b1692d537f017bf86b3cf5feb
Lookup "errno.html/1/2/3": hit 25 2aef1c9d1ab2b55b163215053fefe703
Lookup "errno.html/1/2/3/4": hit 25 cb802be53721c46a46247c5e4e0f4ce6
Lookup "errno.html/1/2/3/4": hit 25 cb802be53721c46a46247c5e4e0f4ce6
Lookup "errno.html": hit 25 6d68a16d217978915036a3bd55428ae7

After:

Lookup "earlyoom/.git/refs"                     hit fd=10 dup=17 iv=6ae2cecd269a25e8d946aff6afe9b8b8
Lookup "earlyoom/.git/refs/remotes"             hit fd=19 dup=17 iv=f04c2d2a5bcc33ebdeaca664859c980d
Lookup "earlyoom/.git/refs/remotes/origin"      miss
Store  "earlyoom/.git/refs/remotes/origin"      fd=17 iv=834a64a1697c9f5705455ba6dbed22b5
Lookup "earlyoom"                               hit fd=7 dup=25 iv=2303a892d6e2357c483574a8070b7679
Lookup "earlyoom/.git"                          hit fd=11 dup=25 iv=d43ca4aff23720c57789c9f62f0aee00
Lookup "earlyoom/.git"                          hit fd=11 dup=25 iv=d43ca4aff23720c57789c9f62f0aee00
Lookup "earlyoom/.git/refs"                     hit fd=10 dup=25 iv=6ae2cecd269a25e8d946aff6afe9b8b8
Lookup "earlyoom/.git/refs/heads"               hit fd=13 dup=25 iv=f9245e7c066b9adc768a1a666da9fbc8
 2020-05-17 21:26:56 +02:00
Jakob Unterwurzacher
416080203b main: accept multiple -passfile options 
Each file will be read and then concatenated
for the effictive password. This can be used as a
kind of multi-factor authenticiton.

Fixes https://github.com/rfjakob/gocryptfs/issues/288
 2020-05-17 19:31:04 +02:00
Jakob Unterwurzacher
ec74d1d2f4 Update go-fuse import path to github.com/hanwen/go-fuse/v2 
We need
fd7328faf9
to fix a crash reported in https://github.com/rfjakob/gocryptfs/issues/430 :

  2019/10/30 17:14:16 Unknown opcode 2016
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x508d38]

This patch is only in the v2.x.x branch. Upgrade to v2, as the
old API is also supported there.

Running

  git grep hanwen/go-fuse | grep -v hanwen/go-fuse/v2

to check for forgotten references comes back clean.
 2020-05-17 14:23:47 +02:00
Jakob Unterwurzacher
ead7008a08 Fix spelling mistakes found by misspell 
https://github.com/client9/misspell
 2020-05-10 00:25:49 +02:00
Jakob Unterwurzacher
16221facb9 ctlsock: create exported ctlsock client library 
The former interal ctlsock server package is renamed
to ctlsocksrv.
 2020-05-09 17:36:41 +02:00
Jakob Unterwurzacher
feaeee90e2 inomap: fix TestSpill bit check 
Wrong bit operator was used.
 2020-05-03 20:28:26 +02:00
Jakob Unterwurzacher
8c9c68fb72 inomap: fix spillBit not set on 2nd hit 
Also add a test for this.

Thanks @slackner for the comment.
 2020-05-03 20:21:11 +02:00
Jakob Unterwurzacher
91f5c242a8 inomap: remove leftover debug output 
This was committed by accident.
 2020-05-03 20:01:12 +02:00
Jakob Unterwurzacher
518771e4e2 fusefrontend_reverse: use inomap for inode number translation 
Gets rid of static inode number value limitations.

Fixes https://github.com/rfjakob/gocryptfs/issues/457
 2020-05-03 15:22:10 +02:00
Jakob Unterwurzacher
483054efaa inomap: comment constants 2020-04-19 22:09:21 +02:00
Jakob Unterwurzacher
9f9d59ded9 inomap: rework logic to efficiently support flags 
Adding flags allows to use inomap in reverse mode,
replacing the clunky inoBaseDirIV/inoBaseNameFile
logic that causes problems with high underlying
inode numbers ( https://github.com/rfjakob/gocryptfs/issues/457 )

Microbenchmarks (values below) show that the "SingleDev"
case is now much slower due to an extra map lookup,
but this has no visible effects in ./test.bash results,
so there was no time spent optimizing the case further.

$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/rfjakob/gocryptfs/internal/inomap
BenchmarkTranslateSingleDev-4   	18757510	        61.5 ns/op
BenchmarkTranslateManyDevs-4    	18061515	        64.5 ns/op
PASS
ok  	github.com/rfjakob/gocryptfs/internal/inomap	2.467s
 2020-04-19 22:00:56 +02:00
Jakob Unterwurzacher
fcdeb52390 inomap: add benchmark 
$ go test -bench=.
goos: linux
goarch: amd64
pkg: github.com/rfjakob/gocryptfs/internal/inomap
BenchmarkTranslateSingleDev-4   	202479382	         5.88 ns/op
BenchmarkTranslateManyDevs-4    	16095795	        71.9 ns/op
PASS
ok  	github.com/rfjakob/gocryptfs/internal/inomap	3.039s
 2020-04-19 21:35:06 +02:00
Jakob Unterwurzacher
f6b1c680b3 nametransform: update comment & simplify tests 
The comment still mentioned CBC, which has been removed
a long time ago.

The test definition can be rewritten using slice literals,
saving sume stuttering.
 2020-04-18 16:14:48 +02:00
Jakob Unterwurzacher
8f5c2a613d Prefer Go stdlib aes-gcm on arm64 with aes instructions 
We used to prefer openssl in this situation, which
used to make sense, but now Go gained an optimized
assembly implementation for aes-gcm on arm64 with
aes instructions:

  root@q1:~/go/src/github.com/rfjakob/gocryptfs# ./gocryptfs -speed
  gocryptfs v1.7.1-46-g73436d9; go-fuse v1.0.1-0.20190319092520-161a16484456; 2020-04-13 go1.14.2 linux/arm64
  AES-GCM-256-OpenSSL      212.30 MB/s    (selected in auto mode)
  AES-GCM-256-Go           452.30 MB/s
  AES-SIV-512-Go           100.25 MB/s
  XChaCha20-Poly1305-Go    137.35 MB/s

https://github.com/rfjakob/gocryptfs/issues/452
 2020-04-13 22:34:07 +02:00
Jakob Unterwurzacher
488111ce39 inomap: split into separate package 
inomap will also be used by fusefrontend_reverse
in the future. Split if off openfiletable to make
it independent.
 2020-04-13 14:54:04 +02:00
Jakob Unterwurzacher
194030f18a speed: add XChaCha20-Poly1305-Go 
https://github.com/rfjakob/gocryptfs/issues/452
 2020-04-13 14:54:04 +02:00
Jakob Unterwurzacher
f82b9caa9c speed: add code comments 2020-02-29 21:26:28 +01:00
Jakob Unterwurzacher
ca9e912a28 fusefrontend: drop xattr user namespace restriction 
We used to restrict setting xattrs to the "user."
namespace. I don't see a real reason for this
anymore, and it causes trouble for users who are using
acls.

Tests will be added in the next commit.

https://github.com/rfjakob/gocryptfs/issues/453
 2020-02-29 20:12:43 +01:00
orcas
9ec042f2f6 Show undecryptable filenames if they match supplied glob 
Resolves https://github.com/rfjakob/gocryptfs/issues/393
 2020-02-28 22:17:59 +01:00
Jakob Unterwurzacher
abc59fa968 contentenc: encryptBlocksParallel: explain why last part runs in new goroutine 
The result is counter-intuitive, so explain it here.
 2020-02-15 21:28:12 +01:00
Jakob Unterwurzacher
d5ce340c02 merge prefer_openssl package into stupidgcm 
Now that I have discovered golang.org/x/sys/cpu and that Go
versions below 1.6 are uncommon, there was not much useful
code left in prefer_openssl.

Merge the remains into stupidgcm.
 2020-02-15 17:21:30 +01:00
Jakob Unterwurzacher
9aeb2a3df6 contentenc: move parallel encryption into encryptBlocksParallel 
Make the logic self-contained in the new helper function.
 2020-02-15 15:49:12 +01:00