public class ConcurrentProjectBuildFileParser extends Object implements ProjectBuildFileParser
ProjectBuildFileParser by creating and
maintaining a pool of ProjectBuildFileParser's instances.
This PythonDslProjectBuildFileParser wrapper creates new instances of delegated parser
using a provided factory and adds them to worker pool. If free parser is available at the time a
request is made, it is reused, if not then it is recreated. Once parsing request (aka FileParser.getManifest(Path) is complete, parser is returned to the worker pool.
Worker pool of parsers can grow unconditionally so it is really up to the user of this class to
manage concurrency level by calling this class' methods appropriate number of times in parallel.
Note that reportProfile() and close() are not synchronized with the worker pool and just call
appropriate methods from all the parsers created to the moment. They should only be called when
no parsing is in effect.
The main reason for this class is to overcome limitations of current implementation of PythonDslProjectBuildFileParser which is stateful and not multithreaded, but Graph Engine
transformations require parsers to be concurrent.
Another option would be to modify existing PythonDslProjectBuildFileParser to be
concurrent. However the legacy Parser pipelines use ResourcePools to shard parser instances, though in fact it is
not needed for SkylarkProjectBuildFileParser which is
concurrent. So, it will require to refactor existing parser pipelines to get rid of ResourcePool which is a bigger change, so it probably should
be done once parsing is fully switched to Graph Engine.
| Constructor and Description |
|---|
ConcurrentProjectBuildFileParser(java.util.function.Supplier<ProjectBuildFileParser> projectBuildFileParserFactory)
Create new instance of
ConcurrentProjectBuildFileParser |
| Modifier and Type | Method and Description |
|---|---|
void |
close() |
com.google.common.collect.ImmutableSortedSet<String> |
getIncludedFiles(Path buildFile)
Collects the loaded files and extensions when parsing the
parseFile build spec. |
BuildFileManifest |
getManifest(Path buildFile)
Collect all information from a particular, along with metadata about the information, for
example which other files were also parsed.
|
boolean |
globResultsMatchCurrentState(Path buildFile,
com.google.common.collect.ImmutableList<GlobSpecWithResult> existingGlobsWithResults)
Checks if existing
GlobSpecs with results are the same as current state in the file
system. |
void |
reportProfile()
Reports profile information captured while parsing build files.
|
public ConcurrentProjectBuildFileParser(java.util.function.Supplier<ProjectBuildFileParser> projectBuildFileParserFactory)
ConcurrentProjectBuildFileParserprojectBuildFileParserFactory - Factory that will be used for creating new instances of
ProjectBuildFileParser on demand. In order to make wrapped ProjectBuildFileParser thread-safe this factory should create instances for each
invocation, not memoize them, otherwise the same object will be used for all threads.public BuildFileManifest getManifest(Path buildFile) throws BuildFileParseException, InterruptedException, IOException
FileParsergetManifest in interface FileParser<BuildFileManifest>buildFile - should be an absolute path to a file. Must have rootPath as its prefix.BuildFileParseExceptionInterruptedExceptionIOExceptionpublic void reportProfile()
throws IOException
ProjectBuildFileParserreportProfile in interface ProjectBuildFileParserIOExceptionpublic com.google.common.collect.ImmutableSortedSet<String> getIncludedFiles(Path buildFile) throws BuildFileParseException, InterruptedException, IOException
FileParserparseFile build spec.getIncludedFiles in interface FileParser<BuildFileManifest>buildFile - should be an absolute path to a file. Must have rootPath as its prefix.BuildFileParseExceptionInterruptedExceptionIOExceptionpublic boolean globResultsMatchCurrentState(Path buildFile, com.google.common.collect.ImmutableList<GlobSpecWithResult> existingGlobsWithResults) throws IOException, InterruptedException
ProjectBuildFileParserGlobSpecs with results are the same as current state in the file
system.globResultsMatchCurrentState in interface ProjectBuildFileParserbuildFile - the buildFile location to be used by the Globber.existingGlobsWithResults - the existing (deserialized) GlobSpecWithResult to check
the file system state against.true if glob expansion produces results matching previously recorded ones,
false otherwise.IOExceptionInterruptedExceptionpublic void close()
throws BuildFileParseException,
InterruptedException,
IOException
close in interface FileParser<BuildFileManifest>close in interface AutoCloseableBuildFileParseExceptionInterruptedExceptionIOException