pre-commit-hooks/tests/detect_aws_credentials_test.py

171 lines
5.8 KiB
Python
Raw Normal View History

from __future__ import annotations
2020-03-01 12:34:45 +08:00
from unittest.mock import patch
import pytest
2019-02-12 11:56:15 +08:00
from pre_commit_hooks.detect_aws_credentials import get_aws_cred_files_from_env
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
from pre_commit_hooks.detect_aws_credentials import get_aws_secrets_from_env
from pre_commit_hooks.detect_aws_credentials import get_aws_secrets_from_file
from pre_commit_hooks.detect_aws_credentials import main
from testing.util import get_resource_path
2017-01-04 02:05:49 +08:00
@pytest.mark.parametrize(
('env_vars', 'values'),
(
({}, set()),
({'AWS_PLACEHOLDER_KEY': '/foo'}, set()),
2017-01-04 02:05:49 +08:00
({'AWS_CONFIG_FILE': '/foo'}, {'/foo'}),
({'AWS_CREDENTIAL_FILE': '/foo'}, {'/foo'}),
({'AWS_SHARED_CREDENTIALS_FILE': '/foo'}, {'/foo'}),
({'BOTO_CONFIG': '/foo'}, {'/foo'}),
({'AWS_PLACEHOLDER_KEY': '/foo', 'AWS_CONFIG_FILE': '/bar'}, {'/bar'}),
2017-01-04 02:05:49 +08:00
(
{
'AWS_PLACEHOLDER_KEY': '/foo', 'AWS_CONFIG_FILE': '/bar',
2017-07-13 09:35:24 +08:00
'AWS_CREDENTIAL_FILE': '/baz',
2017-01-04 02:05:49 +08:00
},
2017-07-18 08:41:44 +08:00
{'/bar', '/baz'},
2017-01-04 02:05:49 +08:00
),
(
{
'AWS_CONFIG_FILE': '/foo', 'AWS_CREDENTIAL_FILE': '/bar',
2017-07-13 09:35:24 +08:00
'AWS_SHARED_CREDENTIALS_FILE': '/baz',
2017-01-04 02:05:49 +08:00
},
2017-07-18 08:41:44 +08:00
{'/foo', '/bar', '/baz'},
2017-01-04 02:05:49 +08:00
),
),
)
def test_get_aws_credentials_file_from_env(env_vars, values):
with patch.dict('os.environ', env_vars, clear=True):
2019-02-12 11:56:15 +08:00
assert get_aws_cred_files_from_env() == values
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
2017-01-04 02:05:49 +08:00
@pytest.mark.parametrize(
('env_vars', 'values'),
(
({}, set()),
({'AWS_PLACEHOLDER_KEY': 'foo'}, set()),
2017-01-04 02:05:49 +08:00
({'AWS_SECRET_ACCESS_KEY': 'foo'}, {'foo'}),
({'AWS_SECURITY_TOKEN': 'foo'}, {'foo'}),
({'AWS_SESSION_TOKEN': 'foo'}, {'foo'}),
2020-02-13 20:01:38 +08:00
({'AWS_SESSION_TOKEN': ''}, set()),
({'AWS_SESSION_TOKEN': 'foo', 'AWS_SECURITY_TOKEN': ''}, {'foo'}),
(
{'AWS_PLACEHOLDER_KEY': 'foo', 'AWS_SECRET_ACCESS_KEY': 'bar'},
{'bar'},
),
2017-01-04 02:05:49 +08:00
(
{'AWS_SECRET_ACCESS_KEY': 'foo', 'AWS_SECURITY_TOKEN': 'bar'},
2017-07-13 09:35:24 +08:00
{'foo', 'bar'},
2017-01-04 02:05:49 +08:00
),
),
)
def test_get_aws_secrets_from_env(env_vars, values):
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
"""Test that reading secrets from environment variables works."""
2017-01-04 02:05:49 +08:00
with patch.dict('os.environ', env_vars, clear=True):
assert get_aws_secrets_from_env() == values
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
2017-01-04 02:05:49 +08:00
@pytest.mark.parametrize(
('filename', 'expected_keys'),
(
(
'aws_config_with_secret.ini',
2017-07-13 09:35:24 +08:00
{'z2rpgs5uit782eapz5l1z0y2lurtsyyk6hcfozlb'},
2017-01-04 02:05:49 +08:00
),
('aws_config_with_session_token.ini', {'foo'}),
2017-07-16 03:56:51 +08:00
(
'aws_config_with_secret_and_session_token.ini',
{'z2rpgs5uit782eapz5l1z0y2lurtsyyk6hcfozlb', 'foo'},
),
2017-01-04 02:05:49 +08:00
(
'aws_config_with_multiple_sections.ini',
{
'7xebzorgm5143ouge9gvepxb2z70bsb2rtrh099e',
'z2rpgs5uit782eapz5l1z0y2lurtsyyk6hcfozlb',
'ixswosj8gz3wuik405jl9k3vdajsnxfhnpui38ez',
2017-07-13 09:35:24 +08:00
'foo',
},
2017-01-04 02:05:49 +08:00
),
('aws_config_without_secrets.ini', set()),
('aws_config_without_secrets_with_spaces.ini', set()),
2017-01-04 02:05:49 +08:00
('nonsense.txt', set()),
('ok_json.json', set()),
),
)
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
def test_get_aws_secrets_from_file(filename, expected_keys):
"""Test that reading secrets from files works."""
keys = get_aws_secrets_from_file(get_resource_path(filename))
assert keys == expected_keys
2017-01-04 02:05:49 +08:00
@pytest.mark.parametrize(
('filename', 'expected_retval'),
(
('aws_config_with_secret.ini', 1),
('aws_config_with_session_token.ini', 1),
('aws_config_with_multiple_sections.ini', 1),
('aws_config_without_secrets.ini', 0),
('aws_config_without_secrets_with_spaces.ini', 0),
2017-01-04 02:05:49 +08:00
('nonsense.txt', 0),
('ok_json.json', 0),
),
)
def test_detect_aws_credentials(filename, expected_retval):
# with a valid credentials file
2016-05-28 05:09:50 +08:00
ret = main((
get_resource_path(filename),
2019-02-12 11:56:15 +08:00
'--credentials-file',
'testing/resources/aws_config_with_multiple_sections.ini',
2016-05-28 05:09:50 +08:00
))
assert ret == expected_retval
def test_allows_arbitrarily_encoded_files(tmpdir):
src_ini = tmpdir.join('src.ini')
src_ini.write(
'[default]\n'
'aws_access_key_id=AKIASDFASDF\n'
'aws_secret_Access_key=9018asdf23908190238123\n',
)
arbitrary_encoding = tmpdir.join('f')
arbitrary_encoding.write_binary(b'\x12\x9a\xe2\xf2')
ret = main((str(arbitrary_encoding), '--credentials-file', str(src_ini)))
assert ret == 0
2017-01-04 02:05:49 +08:00
@patch('pre_commit_hooks.detect_aws_credentials.get_aws_secrets_from_file')
@patch('pre_commit_hooks.detect_aws_credentials.get_aws_secrets_from_env')
def test_non_existent_credentials(mock_secrets_env, mock_secrets_file, capsys):
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
"""Test behavior with no configured AWS secrets."""
2017-01-04 02:05:49 +08:00
mock_secrets_env.return_value = set()
mock_secrets_file.return_value = set()
ret = main((
Improve searching for configured AWS credentials The previous approach for finding AWS credentials was pretty naive and only covered contents of a single file (~/.aws/credentials by default). The AWS CLI documentation states various other ways to configure credentials which weren't covered: https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials Even that aren't all ways, a look into the code shows: https://github.com/boto/botocore/blob/develop/botocore/credentials.py This commit changes the behavior so the hook will behave in a way that if the AWS CLI is able to obtain credentials from local files, the hook will find them as well. The changes in detail are: - detect AWS session tokens and handle them like secret keys. - always search credentials in the default AWS CLI file locations ( ~/.aws/config, ~/.aws/credentials, /etc/boto.cfg and ~/.boto) - detect AWS credentials configured via environment variables in AWS_SECRET_ACCESS_KEY, AWS_SECURITY_TOKEN and AWS_SESSION_TOKEN - check additional configuration files configured via environment variables (AWS_CREDENTIAL_FILE, AWS_SHARED_CREDENTIALS_FILE and BOTO_CONFIG) - print out the first four characters of each secret found in files to be checked in, to make it easier to figure out, what the secrets were, which were going to be checked in - improve error handling for parsing ini-files - improve tests There is a major functional change introduced by this commit: Locations the AWS CLI gets credentials from are always searched and there is no way to disable them. --credentials-file is still there to specify one or more additional files to search credentials in. It's the purpose of this hook to find and check files for found credentials, so it should work in any case. As this commit also improves error handling for not-existing or malformed configuration files, it should be no big deal. Receiving credentials via the EC2 and ECS meta data services is not covered intentionally, to not further increase the amount of changes in this commit and as it's probably an edge case anyway to have this hook running in such an environment.
2016-12-30 15:41:24 +08:00
get_resource_path('aws_config_without_secrets.ini'),
2019-02-12 11:57:37 +08:00
'--credentials-file=testing/resources/credentailsfilethatdoesntexist',
))
assert ret == 2
out, _ = capsys.readouterr()
2017-01-04 02:05:49 +08:00
assert out == (
'No AWS keys were found in the configured credential files '
'and environment variables.\nPlease ensure you have the '
'correct setting for --credentials-file\n'
)
@patch('pre_commit_hooks.detect_aws_credentials.get_aws_secrets_from_file')
@patch('pre_commit_hooks.detect_aws_credentials.get_aws_secrets_from_env')
2019-02-12 11:56:15 +08:00
def test_non_existent_credentials_with_allow_flag(
mock_secrets_env, mock_secrets_file,
):
mock_secrets_env.return_value = set()
mock_secrets_file.return_value = set()
ret = main((
get_resource_path('aws_config_without_secrets.ini'),
2019-02-12 11:57:37 +08:00
'--credentials-file=testing/resources/credentailsfilethatdoesntexist',
'--allow-missing-credentials',
))
assert ret == 0