Involved Source Filesbimport.goexportdata.go Package gcimporter provides various functions for reading
gc-generated object files that can be used to implement the
Importer interface defined by the Go 1.5 standard library package.
The encoding is deterministic: if the encoder is applied twice to
the same types.Package data structure, both encodings are equal.
This property may be important to avoid spurious changes in
applications such as build systems.
However, the encoder is not necessarily idempotent. Importing an
exported package may yield a types.Package that, while it
represents the same set of Go types as the original, may differ in
the details of its internal representation. Because of these
differences, re-encoding the imported package may yield a
different, but equally valid, encoding of the package.iexport.goiimport.gopredeclared.gosupport.goureader_yes.go
Package-Level Type Names (total 24, in which 3 are exported)
/* sort exporteds by: | */
A GetPackagesFunc function obtains the non-nil symbols for a set of
packages, creating and recursively importing them as needed. An
implementation should store each package symbol is in the Pkg
field of the items array.
Any error causes importing to fail. This can be used to quickly read
the import manifest of an export data file without fully decoding it.
A GetPackagesItem is a request from the importer for the package
symbol of the specified name and path.NamestringPathstring // to be filled in by GetPackagesFunc callnameIndexmap[string]uint64 private importer state
ReportFunc is the type of a function used to report formatted bugs.
exportWriter handles writing out individual data section chunks.dataintWriterp*iexporterprevColumnint64prevFilestringprevLineint64(*exportWriter) bool(b bool) bool(*exportWriter) doTyp(t types.Type, pkg *types.Package)(*exportWriter) exportPath(pkg *types.Package) string(*exportWriter) flush() uint64(*exportWriter) int64(x int64)(*exportWriter) localIdent(obj types.Object) mpfloat exports a multi-precision floating point number.
The number's value is decomposed into mantissa × 2**exponent, where
mantissa is an integer. The value is written out as mantissa (as a
multi-precision integer) and then the exponent, except exponent is
omitted if mantissa is zero. mpint exports a multi-precision integer.
For unsigned types, small values are written out as a single
byte. Larger values are written out as a length-prefixed big-endian
byte string, where the length prefix is encoded as its complement.
For example, bytes 0, 1, and 2 directly represent the integer
values 0, 1, and 2; while bytes 255, 254, and 253 indicate a 1-,
2-, and 3-byte big-endian string follow.
Encoding for signed types use the same general approach as for
unsigned types, except small values use zig-zag encoding and the
bottom bit of length prefix byte for large values is reserved as a
sign bit.
The exact boundary between small and large encodings varies
according to the maximum number of bytes needed to encode a value
of type typ. As a special case, 8-bit types are always encoded as a
single byte.
TODO(mdempsky): Is this level of complexity really worthwhile? objectPath writes the package and objectPath to use to look up obj in a
different package, when encoding in "shallow" mode.
When doing a shallow import, the importer creates only the local package,
and requests package symbols for dependencies from the client.
However, certain types defined in the local package may hold objects defined
(perhaps deeply) within another package.
For example, consider the following:
package a
func F() chan * map[string] struct { X int }
package b
import "a"
var B = a.F()
In this example, the type of b.B holds fields defined in package a.
In order to have the correct canonical objects for the field defined in the
type of B, they are encoded as objectPaths and later looked up in the
importer. The same problem applies to interface methods.(*exportWriter) param(obj types.Object)(*exportWriter) paramList(tup *types.Tuple)(*exportWriter) pkg(pkg *types.Package)(*exportWriter) pos(pos token.Pos)(*exportWriter) posV0(pos token.Pos)(*exportWriter) posV1(pos token.Pos) posV2 encoding (used only in shallow mode) records positions as
(file, offset), where file is the index in the token.File table
(which records the file name and newline offsets) and offset is a
byte offset. It effectively ignores //line directives.(*exportWriter) qualifiedType(obj *types.TypeName)(*exportWriter) signature(sig *types.Signature)(*exportWriter) startType(k itag)(*exportWriter) string(s string)(*exportWriter) tag(tag byte)(*exportWriter) tparamList(prefix string, list *types.TypeParamList, pkg *types.Package) TODO(rfindley): what does 'pkg' even mean here? It would be better to pass
it in explicitly into signatures and structs that may use it for
constructing fields.(*exportWriter) typeList(ts *types.TypeList, pkg *types.Package)(*exportWriter) uint64(x uint64)(*exportWriter) value(typ types.Type, v constant.Value) writeIndex writes out an object index. mainIndex indicates whether
we're writing out the main index, which is also read by
non-compiler tools and includes a complete package description
(i.e., name and height).
allPkgs tracks all packages that have been referenced by
the export data, so we can ensure to include them in the
main index.data0intWriterdeclIndexmap[types.Object]uint64declTodoobjQueue In shallow mode, object positions are encoded as (file, offset).
Each file is recorded as a line-number table.
Only the lines of needed positions are saved faithfully. // value is index in fileInfosfileInfos[]*filePositionsfset*token.FileSet // for tracing support // (nil in bundle mode) // encodes objects from other packages in shallow mode; lazily allocatedout*bytes.Buffer // don't put types from other packages in the indexstringIndexmap[string]uint64stringsintWriter // typeparam->exported nametypIndexmap[types.Type]uint64versionint(*iexporter) doDecl(obj types.Object) encodeFile writes to w a representation of the file sufficient to
faithfully restore position information about all needed offsets.
Mutates the needed array. exportName returns the 'exported' name of an object. It differs from
obj.Name() only for type parameters (see tparamExportName for details). fileIndexAndOffset returns the index of the token.File and the byte offset of pos within it.(*iexporter) newWriter() *exportWriter objectpathEncoder returns the lazily allocated objectpath.Encoder to use
when encoding objects in other packages during shallow export.
Using a shared Encoder amortizes some of cost of objectpath search. pushDecl adds n to the declaration work queue, if not already present. stringOff returns the offset of s within the string section.
If not already present, it's added to the end.(*iexporter) trace(format string, args ...any)(*iexporter) typOff(t types.Type, pkg *types.Package) uint64
internalError represents an error generated inside this package.( internalError) Error() string
internalError : error
Reader*bytes.Readerpathstring // current reading index // index of previous rune; or < 0Reader.s[]byte Len returns the number of bytes of the unread portion of the
slice. Read implements the [io.Reader] interface. ReadAt implements the [io.ReaderAt] interface. ReadByte implements the [io.ByteReader] interface. ReadRune implements the [io.RuneReader] interface. Reset resets the [Reader] to be reading from b. Seek implements the [io.Seeker] interface. Size returns the original length of the underlying byte slice.
Size is the number of bytes available for reading via [Reader.ReadAt].
The result is unaffected by any method calls except [Reader.Reset]. UnreadByte complements [Reader.ReadByte] in implementing the [io.ByteScanner] interface. UnreadRune complements [Reader.ReadRune] in implementing the [io.RuneScanner] interface. WriteTo implements the [io.WriterTo] interface.(*intReader) int64() int64(*intReader) uint64() uint64
intReader : compress/flate.Reader
intReader : io.ByteReader
intReader : io.ByteScanner
intReader : io.Reader
intReader : io.ReaderAt
intReader : io.ReadSeeker
intReader : io.RuneReader
intReader : io.RuneScanner
intReader : io.Seeker
intReader : io.WriterTo
Bufferbytes.Buffer // contents are the bytes buf[off : len(buf)] // last read operation, so that Unread* can work correctly. // read at &buf[off], write at &buf[len(buf)] Available returns how many bytes are unused in the buffer. AvailableBuffer returns an empty buffer with b.Available() capacity.
This buffer is intended to be appended to and
passed to an immediately succeeding [Buffer.Write] call.
The buffer is only valid until the next write operation on b. Bytes returns a slice of length b.Len() holding the unread portion of the buffer.
The slice is valid for use only until the next buffer modification (that is,
only until the next call to a method like [Buffer.Read], [Buffer.Write], [Buffer.Reset], or [Buffer.Truncate]).
The slice aliases the buffer content at least until the next buffer modification,
so immediate changes to the slice will affect the result of future reads. Cap returns the capacity of the buffer's underlying byte slice, that is, the
total space allocated for the buffer's data. Grow grows the buffer's capacity, if necessary, to guarantee space for
another n bytes. After Grow(n), at least n bytes can be written to the
buffer without another allocation.
If n is negative, Grow will panic.
If the buffer can't grow it will panic with [ErrTooLarge]. Len returns the number of bytes of the unread portion of the buffer;
b.Len() == len(b.Bytes()). Next returns a slice containing the next n bytes from the buffer,
advancing the buffer as if the bytes had been returned by [Buffer.Read].
If there are fewer than n bytes in the buffer, Next returns the entire buffer.
The slice is only valid until the next call to a read or write method. Read reads the next len(p) bytes from the buffer or until the buffer
is drained. The return value n is the number of bytes read. If the
buffer has no data to return, err is [io.EOF] (unless len(p) is zero);
otherwise it is nil. ReadByte reads and returns the next byte from the buffer.
If no byte is available, it returns error [io.EOF]. ReadBytes reads until the first occurrence of delim in the input,
returning a slice containing the data up to and including the delimiter.
If ReadBytes encounters an error before finding a delimiter,
it returns the data read before the error and the error itself (often [io.EOF]).
ReadBytes returns err != nil if and only if the returned data does not end in
delim. ReadFrom reads data from r until EOF and appends it to the buffer, growing
the buffer as needed. The return value n is the number of bytes read. Any
error except io.EOF encountered during the read is also returned. If the
buffer becomes too large, ReadFrom will panic with [ErrTooLarge]. ReadRune reads and returns the next UTF-8-encoded
Unicode code point from the buffer.
If no bytes are available, the error returned is io.EOF.
If the bytes are an erroneous UTF-8 encoding, it
consumes one byte and returns U+FFFD, 1. ReadString reads until the first occurrence of delim in the input,
returning a string containing the data up to and including the delimiter.
If ReadString encounters an error before finding a delimiter,
it returns the data read before the error and the error itself (often [io.EOF]).
ReadString returns err != nil if and only if the returned data does not end
in delim. Reset resets the buffer to be empty,
but it retains the underlying storage for use by future writes.
Reset is the same as [Buffer.Truncate](0). String returns the contents of the unread portion of the buffer
as a string. If the [Buffer] is a nil pointer, it returns "<nil>".
To build strings more efficiently, see the [strings.Builder] type. Truncate discards all but the first n unread bytes from the buffer
but continues to use the same allocated storage.
It panics if n is negative or greater than the length of the buffer. UnreadByte unreads the last byte returned by the most recent successful
read operation that read at least one byte. If a write has happened since
the last read, if the last read returned an error, or if the read read zero
bytes, UnreadByte returns an error. UnreadRune unreads the last rune returned by [Buffer.ReadRune].
If the most recent read or write operation on the buffer was
not a successful [Buffer.ReadRune], UnreadRune returns an error. (In this regard
it is stricter than [Buffer.UnreadByte], which will unread the last byte
from any read operation.) Write appends the contents of p to the buffer, growing the buffer as
needed. The return value n is the length of p; err is always nil. If the
buffer becomes too large, Write will panic with [ErrTooLarge]. WriteByte appends the byte c to the buffer, growing the buffer as needed.
The returned error is always nil, but is included to match [bufio.Writer]'s
WriteByte. If the buffer becomes too large, WriteByte will panic with
[ErrTooLarge]. WriteRune appends the UTF-8 encoding of Unicode code point r to the
buffer, returning its length and an error, which is always nil but is
included to match [bufio.Writer]'s WriteRune. The buffer is grown as needed;
if it becomes too large, WriteRune will panic with [ErrTooLarge]. WriteString appends the contents of s to the buffer, growing the buffer as
needed. The return value n is the length of s; err is always nil. If the
buffer becomes too large, WriteString will panic with [ErrTooLarge]. WriteTo writes data to w until the buffer is drained or an error occurs.
The return value n is the number of bytes written; it always fits into an
int, but it is int64 to match the [io.WriterTo] interface. Any error
encountered during the write is also returned. empty reports whether the unread portion of the buffer is empty. grow grows the buffer to guarantee space for n more bytes.
It returns the index where bytes should be written.
If the buffer can't grow it will panic with ErrTooLarge.(*intWriter) int64(x int64) readSlice is like ReadBytes but returns a reference to internal buffer data. tryGrowByReslice is an inlineable version of grow for the fast-case where the
internal buffer only needs to be resliced.
It returns the index where bytes should be written and whether it succeeded.(*intWriter) uint64(x uint64)
*intWriter : compress/flate.Reader
*intWriter : expvar.Var
*intWriter : fmt.Stringer
*intWriter : internal/bisect.Writer
*intWriter : io.ByteReader
*intWriter : io.ByteScanner
*intWriter : io.ByteWriter
*intWriter : io.Reader
*intWriter : io.ReaderFrom
*intWriter : io.ReadWriter
*intWriter : io.RuneReader
*intWriter : io.RuneScanner
*intWriter : io.StringWriter
*intWriter : io.Writer
*intWriter : io.WriterTo
*intWriter : golang.org/x/net/http2.pipeBuffer
*intWriter : golang.org/x/net/http2.stringWriter
*intWriter : context.stringer
*intWriter : crypto/tls.transcriptHash
*intWriter : net/http.http2pipeBuffer
*intWriter : net/http.http2stringWriter
*intWriter : runtime.stringer
objQueue is a FIFO queue of types.Object. The zero value of objQueue is
a ready-to-use empty queue.headintring[]types.Objecttailint empty returns true if q contains no Nodes. popHead pops a node from the head of the queue. It panics if q is empty. pushTail appends n to the tail of the queue.
A pkgReader holds the shared state for reading a unified IR package
description.PkgDecoderpkgbits.PkgDecoder // create types.Alias nodesctxt*types.ContextfakefakeFileSet ifaces holds a list of constructed Interfaces, which need to have
Complete called after importing is done. // previously imported packages, indexed by path laterFns holds functions that need to be invoked at the end of
import reading. laterFors is used in case of 'type A B' to ensure that B is processed before A. elemData is the full data payload of the encoded package.
Elements are densely and contiguously packed together.
The last 8 bytes of elemData are the package fingerprint. elemEnds stores the byte-offset end positions of element
bitstreams within elemData.
For example, element I's bitstream data starts at elemEnds[I-1]
(or 0, if I==0) and ends at elemEnds[I].
Note: elemEnds is indexed by absolute indices, not
section-relative indices. elemEndsEnds stores the index-offset end positions of relocation
sections within elemEnds.
For example, section K's end positions start at elemEndsEnds[K-1]
(or 0, if K==0) and end at elemEndsEnds[K]. pkgPath is the package path for the package to be decoded.
TODO(mdempsky): Remove; unneeded since CL 391014.PkgDecoder.scratchRelocEnt[]pkgbits.RelocEnt sync indicates whether the file uses sync markers. version is the file format version.pkgs[]*types.Package lazily initialized arrays corresponding to the unified IR
PosBase, Pkg, and Type sections, respectively. // position bases (i.e., file names)typs[]types.Type AbsIdx returns the absolute index for the given (section, index)
pair. DataIdx returns the raw element bitstream for the given (section,
index) pair. Fingerprint returns the package fingerprint. NewDecoder returns a Decoder for the given (section, index) pair,
and decodes the given SyncMarker from the element bitstream. NewDecoderRaw returns a Decoder for the given (section, index) pair.
Most callers should use NewDecoder instead. NumElems returns the number of elements in section k. PeekObj returns the package path, object name, and CodeObj for the
specified object index. PeekPkgPath returns the package path for the specified package
index. PkgPath returns the package path for the package
TODO(mdempsky): Remove; unneeded since CL 391014.(*pkgReader) RetireDecoder(d *pkgbits.Decoder) StringIdx returns the string value for the given string index. SyncMarkers reports whether pr uses sync markers. TempDecoder returns a Decoder for the given (section, index) pair,
and decodes the given SyncMarker from the element bitstream.
If possible the Decoder should be RetireDecoder'd when it is no longer
needed, this will avoid heap allocations.(*pkgReader) TempDecoderRaw(k pkgbits.RelocKind, idx pkgbits.Index) pkgbits.Decoder TotalElems returns the total number of elements across all sections. later adds a function to be invoked at the end of import reading. laterFor adds a function to be invoked at the end of import reading, and records the type that function is finishing.(*pkgReader) newReader(k pkgbits.RelocKind, idx pkgbits.Index, marker pkgbits.SyncMarker) *reader(*pkgReader) objDictIdx(idx pkgbits.Index) *readerDict(*pkgReader) objIdx(idx pkgbits.Index) (*types.Package, string)(*pkgReader) pkgIdx(idx pkgbits.Index) *types.Package(*pkgReader) posBaseIdx(idx pkgbits.Index) string(*pkgReader) retireReader(r *reader)(*pkgReader) tempReader(k pkgbits.RelocKind, idx pkgbits.Index, marker pkgbits.SyncMarker) *reader(*pkgReader) typIdx(info typeInfo, dict *readerDict) types.Type
A readerDict holds the state for type parameters that parameterize
the current unified IR element. bounds is a slice of typeInfos corresponding to the underlying
bounds of the element's type parameters. derived is a slice of types derived from tparams, which may be
instantiated while reading the current element. // lazily instantiated from derived tparams is a slice of the constructed TypeParams for the element.
Package-Level Functions (total 34, in which 15 are exported)
FindExportData positions the reader r at the beginning of the
export data section of an underlying cmd/compile created archive
file by reading from it. The reader must be positioned at the
start of the file before calling this function.
This returns the length of the export data in bytes.
This function is needed by [gcexportdata.Read], which must
accept inputs produced by the last two releases of cmd/compile,
plus tip.
FindPackageDefinition positions the reader r at the beginning of a package
definition file ("__.PKGDEF") within a GC-created archive by reading
from it, and returns the size of the package definition file in the archive.
The reader must be positioned at the start of the archive file before calling
this function, and "__.PKGDEF" is assumed to be the first file in the archive.
See cmd/internal/archive for details on the archive format.
FindPkg returns the filename and unique package id for an import
path based on package information provided by build.Import (using
the build.Default build.Context). A relative srcDir is interpreted
relative to the current working directory.
FindPkg is only used in tests within x/tools.
GetPackagesFromMap returns a GetPackagesFunc that retrieves
packages from the given map of package path to package.
The returned function may mutate m: each requested package that is not
found is created with types.NewPackage and inserted into m.
IExportBundle writes an indexed export bundle for pkgs to out.
IExportData writes indexed export data for pkg to out.
If no file set is provided, position info will be missing.
The package path of the top-level package will not be recorded,
so that calls to IImportData can override with a provided package path.
IExportShallow encodes "shallow" export data for the specified package.
For types, we use "shallow" export data. Historically, the Go
compiler always produced a summary of the types for a given package
that included types from other packages that it indirectly
referenced: "deep" export data. This had the advantage that the
compiler (and analogous tools such as gopls) need only load one
file per direct import. However, it meant that the files tended to
get larger based on the level of the package in the import
graph. For example, higher-level packages in the kubernetes module
have over 1MB of "deep" export data, even when they have almost no
content of their own, merely because they mention a major type that
references many others. In pathological cases the export data was
300x larger than the source for a package due to this quadratic
growth.
"Shallow" export data means that the serialized types describe only
a single package. If those types mention types from other packages,
the type checker may need to request additional packages beyond
just the direct imports. Type information for the entire transitive
closure of imports is provided (lazily) by the DAG.
No promises are made about the encoding other than that it can be decoded by
the same version of IIExportShallow. If you plan to save export data in the
file system, be sure to include a cryptographic digest of the executable in
the key to avoid version skew.
If the provided reportf func is non-nil, it is used for reporting
bugs (e.g. recovered panics) encountered during export, enabling us
to obtain via telemetry the stack that would otherwise be lost by
merely returning an error.
IImportBundle imports a set of packages from the serialized package bundle.
IImportData imports a package from the serialized package data
and returns 0 and a reference to the package.
If the export data version is not recognized or the format is otherwise
compromised, an error is returned.
IImportShallow decodes "shallow" types.Package data encoded by
[IExportShallow] in the same executable. This function cannot import data
from cmd/compile or gcexportdata.Write.
The importer calls getPackages to obtain package symbols for all
packages mentioned in the export data, including the one being
decoded.
If the provided reportf func is non-nil, it will be used for reporting bugs
encountered during import.
TODO(rfindley): remove reportf when we are confident enough in the new
objectpath encoding.
Import imports a gc-generated package given its import path and srcDir, adds
the corresponding package object to the packages map, and returns the object.
The packages map must contain all packages already imported.
Import is only used in tests.
ReadExportDataHeader reads the export data header and format from r.
It returns the number of bytes read, or an error if the format is no longer
supported or it failed to read.
The only currently supported format is binary export data in the
unified export format.
ReadObjectHeaders reads object headers from the reader. Object headers are
lines that do not start with an end-of-section marker "$$". The first header
is the objabi header. On success, the reader will be positioned at the beginning
of the end-of-section marker.
It returns an error if any header does not fit in r.Size() bytes.
ReadUnified reads the contents of the unified export data from a reader r
that contains the contents of a GC-created archive file.
On success, the reader will be positioned after the end-of-section marker "\n$$\n".
Supported GC-created archive files have 4 layers of nesting:
- An archive file containing a package definition file.
- The package definition file contains headers followed by a data section.
Headers are lines (≤ 4kb) that do not start with "$$".
- The data section starts with "$$B\n" followed by export data followed
by an end of section marker "\n$$\n". (The section start "$$\n" is no
longer supported.)
- The export data starts with a format byte ('u') followed by the <data> in
the given format. (See ReadExportDataHeader for older formats.)
Putting this together, the bytes in a GC-created archive files are expected
to look like the following.
See cmd/internal/archive for more details on ar file headers.
| <!arch>\n | ar file signature
| __.PKGDEF...size...\n | ar header for __.PKGDEF including size.
| go object <...>\n | objabi header
| <optional headers>\n | other headers such as build id
| $$B\n | binary format marker
| u<data>\n | unified export <data>
| $$\n | end-of-section marker
| [optional padding] | padding byte (0x0A) if size is odd
| [ar file header] | other ar files
| [ar file data] |
canReuse reports whether the type rhs on the RHS of the declaration for def
may be re-used.
Specifically, if def is non-nil and rhs is an interface type with methods, it
may not be re-used because we have a convention of setting the receiver type
for interface methods to def.
TODO(adonovan): make this call panic, so that it's symmetric with errorf.
Otherwise it's easy to forget to do anything with the error.
TODO(adonovan): also, consider switching the names "errorf" and
"internalErrorf" as the former is used for bugs, whose cause is
internal inconsistency, whereas the latter is used for ordinary
situations like bad input, whose cause is external.
lookupGorootExport returns the location of the export data
(normally found in the build cache, but located in GOROOT/pkg
in prior Go releases) for the package located in pkgDir.
(We use the package's directory instead of its import path
mainly to simplify handling of the packages in src/vendor
and cmd/vendor.)
lookupGorootExport is only used in tests within x/tools.
pkgScope returns pkg.Scope().
If pkg is nil, it returns types.Universe instead.
TODO(mdempsky): Remove after x/tools can depend on Go 1.19.
Copy of $GOROOT/src/cmd/internal/archive.ReadHeader.
readUnifiedPackage reads a package description from the given
unified IR export data decoder.
See cmd/compile/internal/types.SplitVargenSuffix.
tparamExportName returns the 'exported' name of a type parameter, which
differs from its actual object name: it is prefixed with a qualifier, and
blank type parameter names are disambiguated by their index in the type
parameter list.
tparamName returns the real name of a type parameter, after stripping its
qualifying prefix and reverting blank-name encoding. See tparamExportName
for details.
markBlack is redefined in iimport_go123.go, to work around golang/go#69912.
If TypeNames are not marked black (in the sense of go/types cycle
detection), they may be mutated when dot-imported. Fix this by punching a
hole through the type, when compiling with Go 1.23. (The bug has been fixed
for 1.24, but the fix was not worth back-porting).
varpkgExts [2]string // a file from the build cache will have no extension
predecl is a cache for the predeclared types in types.Universe.
Cache a distinct result based on the runtime value of any.
The pointer value of the any type varies based on GODEBUG settings.
Package-Level Constants (total 35, none are exported)