macaron.build_spec_generator package

Subpackages

Submodules

macaron.build_spec_generator.build_command_patcher module

This module contains the implementation of the build command patching.

macaron.build_spec_generator.build_command_patcher.patch_commands(cmds_sequence, patches)

Patch a sequence of CLI commands.

For each command in this command sequence:

If the command is not a build command, or it’s a tool we do not support, it will be left intact.
If the command is a build command we support, it will be patched, if a patch value is provided in patches. If no patch value is provided for a build command, it will be left intact.

patches is a mapping with:

Key: an instance of the BuildTool enum
Value: the patch value provided to CLICommandParser.apply_patch. For more information on the patch value see the concrete implementations of the CLICommandParser.apply_patch method. For example: macaron.cli_command_parser.maven_cli_parser.MavenCLICommandParser.apply_patch, macaron.cli_command_parser.gradle_cli_parser.GradleCLICommandParser.apply_patch.

This means that all commands that match a BuildTool will be applied by the same patch value.

Returns:: The patched command sequence or None if there is an error. The errors that can happen if any command which we support is invalid in cmds_sequence, or the patch value is valid.
Return type:: list[list[str]] | None

macaron.build_spec_generator.build_spec_generator module

This module contains the functions used for generating build specs from the Macaron database.

class macaron.build_spec_generator.build_spec_generator.BuildSpecFormat(value)

Bases: str, Enum

The build spec formats that we support.

REPRODUCIBLE_CENTRAL = 'rc-buildspec'

macaron.build_spec_generator.build_spec_generator.gen_build_spec_for_purl(purl, database_path, build_spec_format, output_path)

Generate the build spec file for the given PURL in the specified output directory.

Parameters:

purl (PackageURL) – The package URL to generate build spec for.
database_path (str) – The path to the Macaron SQLite database file. This database will be accessed in read-only mode, ensuring that no modifications can be made during operations.
build_spec_format (BuildSpecFormat) – The format of the final build spec content.
output_path (str) – The path to the output directory.

Returns:

The exit code for this function. os.EX_OK if everything is fine, os.EX_OSERR if the buildspec file cannot be created in the local filesystem, os.EX_DATAERR if there was an error generating the content for the buildspec file.

Return type:

int

macaron.build_spec_generator.jdk_finder module

This module includes the functions for obtaining the JDK version from a Java artifact.

class macaron.build_spec_generator.jdk_finder.JavaArtifactExt(value)

Bases: str, Enum

The extensions for Java artifacts.

JAR = '.jar'

class macaron.build_spec_generator.jdk_finder.CacheStrategy(value)

Bases: Enum

The strategy for caching the downloaded artifacts for JDK version finding.

DISABLE = 0

MAVEN_LAYOUT = 1

macaron.build_spec_generator.jdk_finder.download_file(url, dest)

Stream a file into a local destination.

Parameters:

url (str) – The URL of the file to stream from.
dest (str) – The path to the destination file in the local file system. This path includes the file name.

Raises:

InvalidHTTPResponseError – If an error happens while streaming the file.
OSError – If the parent directory of dest doesn’t exist.

Return type:

None

macaron.build_spec_generator.jdk_finder.join_remote_maven_repo_url(remote_maven_url, maven_repo_path)

Join the base remote maven URL with a maven repository path.

Parameters:

remote_maven_url (str) – The url to a remove maven layout repository. For example: https://repo1.maven.org/maven2
maven_repo_path (str) – The maven repository path for a GAV coordinate or an artifact from the root of the remote maven layout repository.

Returns:

The joined path.

Return type:

str

Examples

>>> remote_maven_repo = "https://repo1.maven.org/maven2"
>>> artifact_path = "io/liftwizard/liftwizard-checkstyle/2.1.22/liftwizard-checkstyle-2.1.22.jar"
>>> join_remote_maven_repo_url(remote_maven_repo, artifact_path)
'https://repo1.maven.org/maven2/io/liftwizard/liftwizard-checkstyle/2.1.22/liftwizard-checkstyle-2.1.22.jar'
>>> join_remote_maven_repo_url(remote_maven_repo, "io/liftwizard/liftwizard-checkstyle/2.1.22/")
'https://repo1.maven.org/maven2/io/liftwizard/liftwizard-checkstyle/2.1.22/'
>>> join_remote_maven_repo_url(f"{remote_maven_repo}/", artifact_path)
'https://repo1.maven.org/maven2/io/liftwizard/liftwizard-checkstyle/2.1.22/liftwizard-checkstyle-2.1.22.jar'

macaron.build_spec_generator.jdk_finder.get_jdk_version_from_jar(artifact_path)

Return the JDK version obtained from a Java artifact.

Parameters:: artifact_path (str) – The path to the artifact to extract the jdk version.
Returns:: The version string extract from the artifact (as is) or None if there is an error, or if we couldn’t find any jdk version.
Return type:: str | None

macaron.build_spec_generator.jdk_finder.find_jdk_version_from_remote_maven_repo_standalone(group_id, artifact_id, version, asset_name, remote_maven_repo_url)

Return the JDK version string from an artifact matching a given GAV from a remote maven layout repository.

This function doesn’t cache the downloaded artifact, and removes it after the function exits. We assume that the remote maven layout repository supports downloading a file through a HTTPS URL.

Parameters:

group_id (str) – The group ID part of the GAV coordinate.
artifact_id (str) – The artifact ID part of the GAV coordinate.
version (str) – The version part of the GAV coordinate.
asset_name (str) – The name of artifact to download and extract the jdk version.
remote_maven_repo_url (str) – The URL to the remote maven layout repository.

Returns:

The version string extracted from the artifact (as is); or None if there is an error, or if we couldn’t find any jdk version.

Return type:

str | None

macaron.build_spec_generator.jdk_finder.find_jdk_version_from_remote_maven_repo_cache(group_id, artifact_id, version, asset_name, remote_maven_repo_url, local_cache_repo)

Return the JDK version string from an artifact matching a given GAV from a remote maven layout repository.

This function caches the downloaded artifact in a maven layout https://maven.apache.org/repository/layout.html under local_cache_repo. We assume that the remote maven layout repository supports downloading a file through a HTTPS URL.

Parameters:

group_id (str) – The group ID part of the GAV coordinate.
artifact_id (str) – The artifact ID part of the GAV coordinate.
version (str) – The version part of the GAV coordinate.
asset_name (str) – The name of artifact to download and extract the jdk version.
remote_maven_repo_url (str) – The URL to the remote maven layout repository.
local_cache_repo (str) – The path to a local directory for caching the downloaded artifact used in JDK version extraction.

Returns:

The version string extracted from the artifact (as is); or None if there is an error, or if we couldn’t find any jdk version.

Return type:

str | None

macaron.build_spec_generator.jdk_finder.find_jdk_version_from_central_maven_repo(group_id, artifact_id, version, cache_strat=CacheStrategy.MAVEN_LAYOUT)

Return the JDK version string from an artifact matching a given GAV from Maven Central repository.

The artifacts will be downloaded from https://repo1.maven.org/maven2/ for JDK version extraction.

We now only support JAR files.

Parameters:

group_id (str) – The group ID part of the GAV coordinate.
artifact_id (str) – The artifact ID part of the GAV coordinate.
version (str) – The version part of the GAV coordinate.
cache_strat (CacheStrategy) – Specify how artifacts from maven central are persisted.

Returns:

The version string extract from the artifact (as is); or None if there is an error, or if we couldn’t find any jdk version.

Return type:

str | None

macaron.build_spec_generator.jdk_version_normalizer module

This module contains the logic to normalize a JDK version string as a major version number.

macaron.build_spec_generator.jdk_version_normalizer.normalize_jdk_version(jdk_version_str)

Return the major JDK version number.

We assume that the JDK version string is already valid (e.g Using a JDK version that is available in the real world).

For 1.x versions, we return the major version as x.

Parameters:: jdk_version_str (str) – The JDK version string.
Returns:: The major JDK version number as a string, or None if there is an error.
Return type:: str | None

Examples

>>> normalize_jdk_version("19")
'19'
>>> normalize_jdk_version("19-ea")
'19'
>>> normalize_jdk_version("11.0.1")
'11'
>>> normalize_jdk_version("1.8")
'8'
>>> normalize_jdk_version("25.0.1")

macaron.build_spec_generator.macaron_db_extractor module

This module contains the logic to extract build relation information for a PURL from the Macaron database.

class macaron.build_spec_generator.macaron_db_extractor.GenericBuildCommandInfo(command, language, language_versions, build_tool_name)

Bases: object

Contains the build command information extracted from build related check facts.

command: list[str]

language: str

language_versions: list[str]

build_tool_name: str

__init__(command, language, language_versions, build_tool_name)

macaron.build_spec_generator.macaron_db_extractor.lookup_multiple(select_statement, session)

Perform a SELECT statement and returns all scalar results.

Parameters:

select_statement (Select[tuple[T]]) – The SQLAlchemy SELECT statement to execute.
session (Session) – The SQLAlchemy session of the database we are querying.

Returns:

The result of executing the SELECT statement as scalar values.

Return type:

Sequence[T]

Raises:

QueryMacaronDatabaseError – If the SELECT statement isn’t executed successfully. For example, if the schema of the target database doesn’t match the statement.

macaron.build_spec_generator.macaron_db_extractor.lookup_one_or_none(select_statement, session)

Perform a SELECT statement and returns at most one scalar result.

Parameters:

select_statement (Select[tuple[T]]) – The SQLAlchemy SELECT statement to execute
session (Session) – The SQLAlchemy session of the database we are querying.

Returns:

The result of executing the SELECT statement as one scalar value or None if there isn’t any available.

Return type:

T | None

Raises:

QueryMacaronDatabaseError – If the SELECT statement isn’t executed successfully. For example, if the schema of the target database doesn’t match the statement. Of if there are more than one result obtained from the SELECT statement.

macaron.build_spec_generator.macaron_db_extractor.compile_sqlite_select_statement(select_statement)

Return the equivalent SQLite SELECT statement from an SQLAlchemy SELECT statement.

This function also introduces additional cosmetic details so that it can be easily read from the log.

Parameters:: select_statement (Select) – The SQLAlchemy Select statement.
Returns:: The equivalent SQLite SELECT statement as a string.
Return type:: str

macaron.build_spec_generator.macaron_db_extractor.get_sql_stmt_latest_component_for_purl(purl)

Return an SQLAlchemy SELECT statement to query the latest Component.

Parameters:: purl (PackageURL) – The PackageURL object to find the Component instance.
Returns:: The SQLAlchemy SELECT statement to query the latest analyzed Component instance corresponding to the PackageURL.
Return type:: Select[tuple[Component]]

macaron.build_spec_generator.macaron_db_extractor.get_sql_stmt_build_tools(component_id)

Return an SQLAlchemy SELECT statement to query the BuildToolFacts for a given component.

Parameters:: component_id (int) – The unique identifier of the component for which BuildToolFacts are to be queried.
Returns:: The SQLAlchemy SELECT statement.
Return type:: Select[tuple[BuildToolFacts]]

macaron.build_spec_generator.macaron_db_extractor.get_sql_stmt_build_as_code_check(component_id)

Return an SQLAlchemy SELECT statement to query the BuildAsCodeFacts for a given component.

Parameters:: component_id (int) – The unique identifier of the component for which BuildAsCodeFacts are to be queried.
Returns:: The SQLAlchemy SELECT statement.
Return type:: Select[tuple[BuildAsCodeFacts]]

macaron.build_spec_generator.macaron_db_extractor.get_sql_stmt_build_service_check(component_id)

Return an SQLAlchemy SELECT statement to query the BuildServiceFacts for a given component.

Parameters:: component_id (int) – The unique identifier of the component for which BuildServiceFacts are to be queried.
Returns:: The SQLAlchemy SELECT statement.
Return type:: Select[tuple[BuildServiceFacts]]

macaron.build_spec_generator.macaron_db_extractor.get_sql_stmt_build_script_check(component_id)

Return an SQLAlchemy SELECT statement to query the BuildScriptFacts for a given component.

Parameters:: component_id (int) – The unique identifier of the component for which BuildServiceFacts are to be queried.
Returns:: The SQLAlchemy SELECT statement.
Return type:: Select[tuple[BuildScriptFacts]]

macaron.build_spec_generator.macaron_db_extractor.lookup_latest_component(purl, session)

Return the component of the latest analysis that matches a given PackageURL string.

Parameters:

purl (PackageURL) – The PackageURL object to look for the latest component id.
session (Session) – The SQLAlcemy Session that connects to the Macaron database.

Returns:

The latest component or None if there isn’t one available in the database.

Return type:

Component | None

Raises:

QueryMacaronDatabaseError – If there is an unexpected error when executing the SQLAlchemy query.

macaron.build_spec_generator.macaron_db_extractor.lookup_build_tools_check(component_id, session)

Return the sequence of BuildToolFacts instances for given PackageURL string.

Parameters:

purl_string (str) – The PackageURL string to look for the BuildToolFacts.
session (Session) – The SQLAlcemy Session that connects to the Macaron database.

Returns:

The sequence of BuildToolFacts instances obtained from querying the database.

Return type:

Sequence[BuildToolFacts]

Raises:

QueryMacaronDatabaseError – If there is an unexpected error when executing the SQLAlchemy query.

macaron.build_spec_generator.macaron_db_extractor.lookup_build_as_code_check(component_id, session)

Return the sequence of BuildAsCodeFacts instances for given PackageURL string.

Parameters:

purl_string (str) – The PackageURL string to look for the BuildAsCodeFacts.
session (Session) – The SQLAlcemy Session that connects to the Macaron database.

Returns:

The sequence of BuildAsCodeFacts instances obtained from querying the database.

Return type:

Sequence[BuildAsCodeFacts]

Raises:

QueryMacaronDatabaseError – If there is an unexpected error when executing the SQLAlchemy query.

macaron.build_spec_generator.macaron_db_extractor.lookup_build_service_check(component_id, session)

Return the sequence of BuildServiceFacts instances for given PackageURL string.

Parameters:

purl_string (str) – The PackageURL string to look for the BuildServiceFacts.
session (Session) – The SQLAlcemy Session that connects to the Macaron database.

Returns:

The sequence of BuildServiceFacts instances obtained from querying the database.

Return type:

Sequence[BuildServiceFacts]

Raises:

QueryMacaronDatabaseError – If there is an unexpected error when executing the SQLAlchemy query.

macaron.build_spec_generator.macaron_db_extractor.lookup_build_script_check(component_id, session)

Return the sequence of BuildScriptFacts instances for given PackageURL string.

Parameters:

purl_string (str) – The PackageURL string to look for the BuildScriptFacts.
session (Session) – The SQLAlcemy Session that connects to the Macaron database.

Returns:

The sequence of BuildScriptFacts instances obtained from querying the database.

Return type:

Sequence[BuildScriptFacts]

Raises:

QueryMacaronDatabaseError – If there is an unexpected error when executing the SQLAlchemy query.

macaron.build_spec_generator.macaron_db_extractor.extract_generic_build_command_info(check_facts)

Return the list of GenericBuildCommandInfo instances from a list of Build related Check Facts.

The following information are captured for each Check Facts

command: the build command, but this information is located in different attribute depending on the type of Build Check Fact (e.g. in BuildAsCodeFacts it is stored in deploy_command). It’s stored in the database as a serialized JSON object so we need to use json.loads to turn it into a list of strings.
language and build_tool_name are attributes of all Build Check Fact instances
language_versions is an attribute of all Build Check Fact instances. It’s stored in the database as a serialized JSON object so we need to use json.loads to turn it into a list of strings.

Parameters:: check_facts (Sequence[BuildAsCodeFacts] | Sequence[BuildServiceFacts] | Sequence[BuildScriptFacts]) – The sequence of check facts obtained from the database.
Returns:: The list of GenericBuildCommandInfo instances that store build command information representing by the Build Check Facts.
Return type:: list[GenericBuildCommandInfo]
Raises:: json.decoder.JSONDecodeError – If we failed to decode the JSON-serialized values stored in the Build*Facts instances.

macaron.build_spec_generator.macaron_db_extractor.lookup_any_build_command(component_id, session)

Return a list of GenericBuildCommandInfo instances from looking up any available build command.

We will look for available build command from build-related check facts.

Parameters:

component_id (int) – The component id to lookup the build command.
session (Session) – The SQLAlchemy session to the database for the lookup.

Returns:

This list will be empty if there is no available build command for this component.

Return type:

list[GenericBuildCommandInfo]

Raises:

QueryMacaronDatabaseError – If there is an unexpected error when executing the SQLAlchemy query for looking up the build commands. Raised by “lookup_*_check” functions