Comment by jerf
2 years ago
A missing tidbit that may help contextualize this post: One of the things about Go that surprised me is that if you have a slice which does not represent the full capacity of the underlying array, you can go ahead and reslice it up to that full capacity even though it's a panic to access the things you're reslicing directly: https://go.dev/play/p/oThz2bNFwgr
Consequently, the GC has to assume that anything forward of any given slice into the underlying array may become accessible in the future as there is legal Go that can access it. It's still memory safe, but it surprised me.
I had some code that was using my incorrect belief that slices could not be resliced up in size to implement some light security boundaries. Fortunately it was still legal, because the code in question simply didn't slice things larger and it's not like I was allowing arbitrary user-supplied code to run, so it was still correct in what it was doing. But I was expecting the runtime to scream if I did somehow screw it up when in fact it may not, depending on the exact capacity and what happened when.
It's also asymmetric, as far as I know; you can slice forward into the array if there is capacity, but if you've got a slice that starts after index 0 in the backing array you can't use that slice to walk back into the underlying array. That is, with
s := []int{11, 12, 13, 14, 15}
s = s[2:]
as far as I know, 11 and 12 are no longer accessible to any legal Go code (not using "unsafe") after that second line executes.
Corrections (again not involving "unsafe", it's obvious those two values are still accessible through "unsafe") welcome. I was wrong once, it's easy to believe I could be wrong again.
One detail in the latest 1.22 release is a change in the "slices" library [1] (the library is based on generics, introduced in 1.21) that helps avoid such cases:
> Functions that shrink the size of a slice (Delete, DeleteFunc, Compact, CompactFunc, and Replace) now zero the elements between the new length and the old length.
[1]: https://tip.golang.org/doc/go1.22#slices
[2]: https://pkg.go.dev/slices
You can prevent later longer reslicing by add an additional cap() element to the slicing to shrink the capacity.
from your first example link.
In your playground example, if you print the capacity and the length before and after “re-extension”, it becomes clear what happened. In fact, accessing item 5 after reduction gives a size panic, where as accessing item 6 after re-extension gives you a capacity panic.
Understanding rsc’s “Go Slices” blog is very helpful here. Coming from Java or something, this exposure of underlying storage could be jarring, but coming from C, Go slices are basically built in fat arrays, and this behavior doesn’t surprise me. Maybe it was a design mistake to expose so much of the underlying machinery. Ymmv.
It is reinforcing my belief that language design is hard. Go is supposed to be a simple language, one may write it for long time, but there will be some trap you discover once in a while.
Language design is hard, but... Jesus Christ, Go really just threw away 50 years of CS language research/experience in the name of “simplicity”.
it's got all the latest features from the 1980s built right in
> ... the GC has to assume that anything forward of any given slice into the underlying array may become accessible in the future as there is legal Go that can access it.
This property complicates the semantics of slices.Delete() in an interesting way:
https://github.com/golang/go/issues/63393
interesting catch, thanks for sharing. my poor memory is already racing thinking about cases where i may have left traps with the same assumption