Browse Source

use clouds.yml rather than openrc.sh

keep-around/285ad701179c1afaef0a8358c83644c20e8feb01
Loïc Dachary 10 months ago
parent
commit
7a51fec5b0
Signed by: dachary GPG Key ID: 992D23B392F9E4F2
  1. 1
      .gitignore
  2. 11
      conftest.py
  3. 17
      docs/community/contribute.rst
  4. 38
      docs/introduction.rst
  5. 11
      docs/user-guide.rst
  6. 1
      enough/common/__init__.py
  7. 58
      enough/common/dotenough.py
  8. 3
      enough/common/host.py
  9. 6
      enough/common/options.py
  10. 3
      enough/common/service.py
  11. 3
      enough/common/ssh.py
  12. 6
      enough/internal/cli/backup.py
  13. 4
      playbooks/backup/roles/backup/tasks/main.yml
  14. 1
      playbooks/backup/roles/backup/templates/backup.sh
  15. 1
      playbooks/backup/roles/backup/templates/prune-backup.sh
  16. 14
      playbooks/backup/tests/test_backup.py
  17. 3
      playbooks/backup/tests/test_snapshot.py
  18. 6
      playbooks/conftest.py
  19. 1
      tests/conftest.py
  20. 16
      tests/enough/common/conftest.py
  21. 3
      tests/enough/common/test_common_host.py
  22. 1
      tests/enough/common/test_common_service.py
  23. 3
      tests/enough/common/test_common_ssh.py
  24. 149
      tests/enough/common/test_dotenough.py
  25. 48
      tests/enough/common/test_init.py
  26. 20
      tests/enough/common/test_init/clone-clobber/clouds.yml
  27. 16
      tests/enough/common/test_openstack.py
  28. 33
      tests/run-tests.sh
  29. 2
      tox.ini

1
.gitignore

@ -12,7 +12,6 @@ tests/domain.yml
tests/clouds.yml
inventory/group_vars/all/domain.yml
inventory/group_vars/all/clouds.yml
openrc.sh
inventories/01-hosts.yml
.tox
.coverage

11
conftest.py

@ -1,16 +1,5 @@
import os
from tests.icinga_helper import IcingaHelper
def pytest_configure(config):
IcingaHelper.set_ansible_inventory(config.getoption("--ansible-inventory"))
def pytest_addoption(parser):
parser.addoption(
'--provider',
choices=('fuga', 'ovh'),
default=os.environ.get('ENOUGH_PROVIDER', 'ovh'),
help='Name of the OpenStack provider used for the tests'
)

17
docs/community/contribute.rst

@ -120,23 +120,11 @@ must be set to the ``ENOUGH_API_TOKEN`` environment variable.
* ``ENOUGH_API_TOKEN=XXXXXXX tests/run-tests.sh tox -e bind``
Set the OpenStack credentials using openrc.sh
+++++++++++++++++++++++++++++++++++++++++++++
Assuming you have your own OVH tenant or one was :ref:`provided to you
<infrastructure>`, the credentials must be set in environment
variables as follows:
* ``source openrc.sh``
Set the OpenStack credentials using clouds.yml
++++++++++++++++++++++++++++++++++++++++++++++
An alternative to ``openrc.sh`` is to use ``clouds.yml`` file:
* ``cp clouds.provider.yml inventory/group_vars/all/clouds.yml``
Note that ``openrc.sh`` file is ignored if ``inventory/group_vars/all/clouds.yml`` exists.
Assuming you have your own OpenStack tenant or one was :ref:`provided to you
<infrastructure>`, the ``clouds.yml`` file must be copied to `tests/clouds.yml`.
Running
+++++++
@ -159,7 +147,6 @@ to skip some steps to speed up test debugging:
--enough-no-create Do not run the create step
--enough-no-tests Do not run the tests step
--enough-no-destroy Do not run the destroy step
--provider={fuga,ovh} Name of the OpenStack provider used for the tests
...
$ tests/run-tests.sh tox -e openvpn -- --enough-no-destroy playbooks/openvpn/tests

38
docs/introduction.rst

@ -36,22 +36,46 @@ services that run on top of it, with Ansible.
Requirements
------------
* The ``openrc.sh`` credentials for a ``Public cloud`` project at `OVH
<https://www.ovh.com/manager/public-cloud/>`__. No other OpenStack
provider is supported.
* An account on a supported OpenStack provider:
* The ``Public cloud`` project has a `Private network
<https://www.ovh.com/world/solutions/vrack/>`__.
* A ``Public cloud`` project at `OVH <https://www.ovh.com/manager/public-cloud/>`__.
A `Private network <https://www.ovh.com/world/solutions/vrack/>`__ must be created for
the project.
* A `Fuga <https://fuga.cloud>`__ account.
* The `clouds.yml` credentials for:
* OVH, found `in horizon <https://horizon.cloud.ovh.net/project/api_access/clouds.yaml>`__
* Fuga, found `in the Team Credentials tab <https://my.fuga.cloud/account/team-credentials>`__
Quick start
-----------
* `Install Docker <http://docs.docker.com/engine/installation/>`__.
* Copy ``openrc.sh`` in ``~/.enough/myname.d.enough.community/openrc.sh`` and edit
to replace ``$OS_PASSWORD_INPUT`` with the actual password.
* Copy `clouds.yml` in `~/.enough/myname.d.enough.community/inventory/group_vars/all/clouds.yml` and edit
to add `password:` next to `auth_url:`, under the `production:` name. For instance:
::
---
openstack_provider: fuga
clouds:
production:
auth:
auth_url: "https://identity.api.ams.fuga.cloud:443/v3"
user_id: "ef5cae1b61b8424594a6ddf94a28381c"
password: "lDk9vOLIXFW09oWcuQEiq0sjB4cV"
user_domain_id: "b919e18e477a889bf89f89e9d9"
project_domain_id: "b919e186cb07a889bf89f89e9d9"
project_id: "25481e67871b4de39ae63fa2008029"
region_name: "ams"
interface: "public"
identity_api_version: 3
* Add the ``enough`` CLI to ``~/.bashrc``:
::
eval "$(docker run --rm enoughcommunity/enough:latest install)"

11
docs/user-guide.rst

@ -7,12 +7,7 @@ Requirements
Technical
~~~~~~~~~
* The ``openrc.sh`` credentials for a ``Public cloud`` project at `OVH
<https://www.ovh.com/manager/public-cloud/>`__. No other OpenStack
provider is supported.
* The ``Public cloud`` project has a `Private network
<https://www.ovh.com/world/solutions/vrack/>`__.
* The ``clouds.yml`` credentials for either OVH or Fuga.
* The current code base requires:
@ -54,8 +49,8 @@ Installation
* `Install Docker <http://docs.docker.com/engine/installation/>`__.
* Copy ``openrc.sh`` in ``~/.enough/example.com/openrc.sh`` and edit
to replace ``$OS_PASSWORD_INPUT`` with the actual password.
* Copy `clouds.yml` in `~/.enough/example.com/inventory/group_vars/all/clouds.yml` and edit
to add `password:` next to `auth_url:` under the `production:` name.
* Add the ``enough`` CLI to ``~/.bashrc``:

1
enough/common/__init__.py

@ -126,7 +126,6 @@ class Enough(object):
kwargs = {
'driver': self.args['driver'],
'domain': target_domain,
'provider': self.args['provider'],
}
clone = Enough(config_dir, self.share_dir, **kwargs)
return clone

58
enough/common/dotenough.py

@ -1,6 +1,4 @@
import copy
import os
import re
import sh
import shutil
import textwrap
@ -137,10 +135,9 @@ class DotEnough(object):
class DotEnoughOpenStack(DotEnough):
def __init__(self, config_dir, domain, provider):
def __init__(self, config_dir, domain):
super().__init__(config_dir, domain)
self.clouds_file = f'{self.config_dir}/inventory/group_vars/all/clouds.yml'
self.provider = provider
def set_clouds_file(self, clouds_file):
shutil.copy(clouds_file, self.clouds_file)
@ -148,59 +145,6 @@ class DotEnoughOpenStack(DotEnough):
def ensure(self):
super().ensure()
self.populate_config('letsencrypt')
self.set_missing_config_file_from_openrc(f'{self.config_dir}/openrc.sh', self.clouds_file)
d = f'{self.config_dir}/inventory/group_vars/all'
if not os.path.exists(f'{d}/openstack-provider.yml'):
open(f'{d}/openstack-provider.yml', 'w').write(textwrap.dedent(f"""\
---
openstack_provider: {self.provider}
"""))
@staticmethod
def set_missing_config_file_from_openrc(openrc, config_file):
if os.path.exists(openrc) and not os.path.exists(config_file):
DotEnoughOpenStack.openrc2clouds(openrc, config_file)
return True
else:
return False
@staticmethod
def openrc2clouds(openrc, clouds):
c = {
'auth': {
'user_domain_name': 'Default',
'password': 'PLACEHOLDER',
}
}
for line in open(openrc).readlines():
r = re.search(r'export\s+(OS_\w+)="(.*)"', line)
if not r:
r = re.search(r'export\s+(OS_\w+)=(.*)\s*', line)
if not r:
continue
(k, v) = r.group(1, 2)
if k == 'OS_REGION_NAME':
c['region_name'] = v
elif k == 'OS_AUTH_URL':
c['auth']['auth_url'] = v
elif k == 'OS_PROJECT_NAME' or k == 'OS_TENANT_NAME':
c['auth']['project_name'] = v
elif k == 'OS_PROJECT_ID' or k == 'OS_TENANT_ID':
c['auth']['project_id'] = v
elif k == 'OS_USERNAME':
c['auth']['username'] = v
elif k == 'OS_PASSWORD' and v != '$OS_PASSWORD_INPUT':
c['auth']['password'] = v
clone = copy.deepcopy(c)
clone['region_name'] = 'DE1'
open(clouds, 'w').write(yaml.dump({
'clouds': {
'production': c,
'clone': clone,
}
}))
return c
class DotEnoughDocker(DotEnough):

3
enough/common/host.py

@ -79,8 +79,7 @@ class HostOpenStack(Host):
def __init__(self, config_dir, share_dir, **kwargs):
super().__init__(config_dir, share_dir, **kwargs)
self.args = kwargs
self.dotenough = dotenough.DotEnoughOpenStack(config_dir, self.args['domain'],
self.args['provider'])
self.dotenough = dotenough.DotEnoughOpenStack(config_dir, self.args['domain'])
self.dotenough.ensure()
def create_or_update(self):

6
enough/common/options.py

@ -7,10 +7,4 @@ def set_options(parser):
'--cloud',
default='production',
help='Name of the cloud in which resources are provisionned')
o.add_argument(
'--provider',
default='ovh',
choices=('fuga', 'ovh'),
help='Name of the OpenStack provider')
return parser

3
enough/common/service.py

@ -112,8 +112,7 @@ class ServiceOpenStack(Service):
def __init__(self, config_dir, share_dir, **kwargs):
super().__init__(config_dir, share_dir, **kwargs)
self.args = kwargs
self.dotenough = dotenough.DotEnoughOpenStack(config_dir, self.args['domain'],
self.args['provider'])
self.dotenough = dotenough.DotEnoughOpenStack(config_dir, self.args['domain'])
self.dotenough.ensure()
def maybe_delegate_dns(self):

3
enough/common/ssh.py

@ -10,8 +10,7 @@ class SSH(object):
self.config_dir = config_dir
self.args = kwargs
self.dot = dotenough.DotEnoughOpenStack(self.config_dir,
self.args['domain'],
self.args['provider'])
self.args['domain'])
self.dot.ensure()
def ssh(self, host, args, interactive=True):

6
enough/internal/cli/backup.py

@ -21,8 +21,7 @@ class Create(Command):
def take_action(self, parsed_args):
args = vars(self.app.options)
args.update(vars(parsed_args))
d = dotenough.DotEnoughOpenStack(settings.CONFIG_DIR, args['domain'],
args['provider'])
d = dotenough.DotEnoughOpenStack(settings.CONFIG_DIR, args['domain'])
d.ensure()
o = OpenStack(settings.CONFIG_DIR, **args)
o.backup_create(args['volumes'])
@ -42,8 +41,7 @@ class Prune(Command):
def take_action(self, parsed_args):
args = vars(self.app.options)
args.update(vars(parsed_args))
d = dotenough.DotEnoughOpenStack(settings.CONFIG_DIR, args['domain'],
args['provider'])
d = dotenough.DotEnoughOpenStack(settings.CONFIG_DIR, args['domain'])
d.ensure()
o = OpenStack(settings.CONFIG_DIR, **args)
o.backup_prune(args['days'])

4
playbooks/backup/roles/backup/tasks/main.yml

@ -43,8 +43,8 @@
mode: 0750
- name: cp clouds.yml ~/.enough/{{ domain }}/inventory/group_vars/all/clouds.yml
copy:
src: inventory/group_vars/all/clouds.yml
template:
src: clouds.yml.j2
dest: "/root/.enough/{{ domain }}/inventory/group_vars/all/clouds.yml"
mode: 0640

1
playbooks/backup/roles/backup/templates/backup.sh

@ -1,6 +1,7 @@
#!/bin/bash
export OS_CLIENT_CONFIG_FILE=/usr/lib/backup/clouds.yml
export OS_CLOUD=production
function backup() {
local host=$1

1
playbooks/backup/roles/backup/templates/prune-backup.sh

@ -1,6 +1,7 @@
#!/bin/bash
export OS_CLIENT_CONFIG_FILE=/usr/lib/backup/clouds.yml
export OS_CLOUD=production
numdays=${1:-30}
stamp1=`date --date "$numdays days ago" '+%s'`

14
playbooks/backup/tests/test_backup.py

@ -3,7 +3,8 @@ testinfra_hosts = ['ansible://bind-host']
def expected_backups(host, count):
cmd = host.run("""
. /usr/lib/backup/openrc.sh
export OS_CLIENT_CONFIG_FILE=/usr/lib/backup/clouds.yml
export OS_CLOUD=production
openstack ${OS_INSECURE} image list | grep -c pet-host
""")
print(cmd.stderr)
@ -17,11 +18,7 @@ def test_backup(host):
# SSL exception connecting to
# https://auth.cloud.ovh.net/v2.0/tokens: [SSL: CERTIFICATE_VERIFY_FAILED]
with host.sudo():
cmd = host.run("echo export OS_INSECURE=--insecure >> /usr/lib/backup/openrc.sh")
print(cmd.stdout)
print(cmd.stderr)
assert 0 == cmd.rc
cmd = host.run("/etc/cron.daily/prune-backup 0")
cmd = host.run("OS_INSECURE=--insecure /etc/cron.daily/prune-backup 0")
print(cmd.stdout)
print(cmd.stderr)
assert 0 == cmd.rc
@ -29,6 +26,7 @@ def test_backup(host):
host.run("timedatectl set-ntp 0")
cmd = host.run("""
set -x
export OS_INSECURE=--insecure
date -s '-15 days'
bash -x /etc/cron.daily/backup
date -s '-30 days'
@ -39,8 +37,8 @@ def test_backup(host):
print(cmd.stderr)
assert 0 == cmd.rc
expected_backups(host, '2')
host.run("bash -x /etc/cron.daily/prune-backup 30")
host.run("OS_INSECURE=--insecure bash -x /etc/cron.daily/prune-backup 30")
expected_backups(host, '1')
finally:
host.run("timedatectl set-ntp 1")
host.run("bash -x /etc/cron.daily/prune-backup 0")
host.run("OS_INSECURE=--insecure bash -x /etc/cron.daily/prune-backup 0")

3
playbooks/backup/tests/test_snapshot.py

@ -3,7 +3,8 @@ testinfra_hosts = ['ansible://bind-host']
def openstack(host, cmd):
cmd = host.run(f"""
. /usr/lib/backup/openrc.sh
export OS_CLIENT_CONFIG_FILE=/usr/lib/backup/clouds.yml
export OS_CLOUD=production
openstack {cmd}
""")
print(cmd.stderr)

6
playbooks/conftest.py

@ -43,8 +43,6 @@ def pytest_sessionstart(session):
enough_destroy(session)
service_directory = session.config.getoption("--enough-service")
provider = session.config.getoption("--provider")
os.environ['ENOUGH_PROVIDER'] = provider
domain = f'{service_directory}.test'
enough_dot_dir = session.config.cache.makedir('dotenough')
config_dir = prepare_config_dir(domain, enough_dot_dir)
@ -52,7 +50,6 @@ def pytest_sessionstart(session):
e = Enough(config_dir, '.',
domain=domain,
driver='openstack',
provider=provider,
inventory=[f'playbooks/{service_directory}/inventory'],
route_to_internal=False)
(vpn_host, hosts) = get_hosts(session, e)
@ -95,8 +92,6 @@ def pytest_sessionfinish(session, exitstatus):
def enough_destroy(session):
service_directory = session.config.getoption("--enough-service")
provider = session.config.getoption("--provider")
os.environ['ENOUGH_PROVIDER'] = provider
domain = f'{service_directory}.test'
enough_dot_dir = session.config.cache.makedir('dotenough')
config_dir = make_config_dir(domain, enough_dot_dir)
@ -107,7 +102,6 @@ def enough_destroy(session):
e = Enough(config_dir, '.',
domain=domain,
driver='openstack',
provider=provider,
inventory=[f'playbooks/{service_directory}/inventory'])
e.destroy()

1
tests/conftest.py

@ -107,7 +107,6 @@ def pytest_runtest_teardown(item, nextitem):
assert os.path.isdir('inventory/group_vars/all') # with the expected structure
check_generated_files = [
'inventory/group_vars/all/openstack-provider.yml',
'inventory/group_vars/all/private-key.yml',
]
if os.environ.get('GITLAB_CI'):

16
tests/enough/common/conftest.py

@ -1,5 +1,3 @@
import os
import pytest
from enough import settings
@ -12,19 +10,14 @@ def pytest_configure(config):
@pytest.fixture(scope='session')
def openstack_provider(request):
def openstack_variables(request):
ansible = ansible_utils.Ansible(settings.CONFIG_DIR, settings.SHARE_DIR)
# enforce 'small' flavor
os_variables = ('openstack_flavor', 'openstack_image', 'openstack_network',
'network_primary_interface', 'network_secondary_interface',
'_provider')
os_variables = "{%s}" % ', '.join(f'"{x}": {x}' for x in os_variables)
try:
os.environ['ENOUGH_PROVIDER'] = request.config.option.provider
# Any host could be used
hostvars = ansible.get_variable(os_variables, 'bind-host')
finally:
del os.environ['ENOUGH_PROVIDER']
# Any host could be used
hostvars = ansible.get_variable(os_variables, 'bind-host')
hostvars['openstack_provider'] = hostvars.pop('_provider')
return hostvars
@ -32,7 +25,6 @@ def openstack_provider(request):
@pytest.fixture
def dot_openstack(tmp_config_dir, request):
domain = settings.CONFIG_DIR.split('/')[-1]
provider = request.config.option.provider
dot_openstack = dotenough.DotEnoughOpenStack(settings.CONFIG_DIR, domain, provider)
dot_openstack = dotenough.DotEnoughOpenStack(settings.CONFIG_DIR, domain)
dot_openstack.ensure()
yield dot_openstack

3
tests/enough/common/test_common_host.py

@ -72,6 +72,5 @@ def test_docker_delete(tmpdir, docker_name, tcp_port):
def test_host_factory(request, tmpdir):
h = host.host_factory(config_dir=tmpdir, driver='docker', domain='a.b')
assert type(h).__name__ == 'HostDocker'
h = host.host_factory(config_dir=tmpdir, driver='openstack', domain='a.b',
provider=request.config.option.provider)
h = host.host_factory(config_dir=tmpdir, driver='openstack', domain='a.b')
assert type(h).__name__ == 'HostOpenStack'

1
tests/enough/common/test_common_service.py

@ -23,7 +23,6 @@ def test_openstack_create_or_update(request, tmpdir, requests_mock):
'domain': domain,
'name': 'essential',
'cloud': 'production',
'provider': request.config.option.provider,
})
s.dotenough.set_certificate('ownca')
s.dotenough.set_clouds_file('inventory/group_vars/all/clouds.yml')

3
tests/enough/common/test_common_ssh.py

@ -11,8 +11,7 @@ def test_ssh(openstack_name, openstack_provider, tmpdir):
domain = 'enough.community'
d = Heat.hostvars_to_stack(openstack_name, openstack_provider)
config_dir = prepare_config_dir(domain, tmpdir)
provider = openstack_provider['openstack_provider']
ssh = SSH(config_dir, domain=domain, provider=provider)
ssh = SSH(config_dir, domain=domain)
s = Stack(config_dir, d)
s.set_public_key(f'{config_dir}/infrastructure_key.pub')
s.create_or_update()

149
tests/enough/common/test_dotenough.py

@ -1,7 +1,6 @@
import os
import yaml
from enough.common.dotenough import Hosts, DotEnough, DotEnoughOpenStack
from enough.common.dotenough import Hosts, DotEnough
#
@ -61,149 +60,3 @@ def test_service_add_to_group(tmpdir):
expected[group]['hosts'][other_host] = None
assert d.service_add_to_group(service, other_host) == expected
os.system(f'cat {tmpdir}/services.yml')
def test_openstack_openrc2clouds(tmpdir):
openrc_file = f'{tmpdir}/openrc.sh'
clouds_file = f'{tmpdir}/clouds.yml'
auth_url = 'https://auth.cloud.ovh.net/v3/'
project_name = 'project_name'
project_id = 'project_id'
username = 'username'
password = 'password'
region_name = 'region_name'
clone_region_name = 'DE1'
auth = {
'auth_url': auth_url,
'project_name': project_name,
'project_id': project_id,
'user_domain_name': "Default",
'username': username,
'password': password,
}
expected = {
'clouds': {
'production': {
'region_name': region_name,
'auth': auth,
},
'clone': {
'region_name': clone_region_name,
'auth': auth,
}
}
}
openrc_1 = f"""
#!/bin/bash
# To use an Openstack cloud you need to authenticate against keystone, which
# returns a **Token** and **Service Catalog**. The catalog contains the
# endpoint for all services the user/tenant has access to - including nova,
# glance, keystone, swift.
#
export OS_AUTH_URL={auth_url}
export OS_IDENTITY_API_VERSION=3
export OS_USER_DOMAIN_NAME=$OS_USER_DOMAIN_NAME:-"Default"
export OS_PROJECT_DOMAIN_NAME=$OS_PROJECT_DOMAIN_NAME:-"Default"s
# With the addition of Keystone we have standardized on the term **tenant**
# as the entity that owns the resources.
export OS_TENANT_ID={project_id}
export OS_TENANT_NAME="{project_name}"
# In addition to the owning entity (tenant), openstack stores the entity
# performing the action as the **user**.
export OS_USERNAME="{username}"
# With Keystone you pass the keystone password.
#echo "Please enter your OpenStack Password: "
#read -sr OS_PASSWORD_INPUT
#export OS_PASSWORD=$OS_PASSWORD_INPUT
export OS_PASSWORD={password}
# If your configuration has multiple regions, we set that information here.
# OS_REGION_NAME is optional and only valid in certain environments.
export OS_REGION_NAME="{region_name}"
# Don't leave a blank variable, unset it if it was empty
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
"""
open(openrc_file, 'w').write(openrc_1)
assert DotEnoughOpenStack.openrc2clouds(
openrc_file, clouds_file) == expected['clouds']['production']
assert yaml.safe_load(open(clouds_file)) == expected
os.unlink(openrc_file)
os.unlink(clouds_file)
openrc_2 = f"""
#!/usr/bin/env bash
# To use an OpenStack cloud you need to authenticate against the Identity
# service named keystone, which returns a **Token** and **Service Catalog**.
# The catalog contains the endpoints for all services the user/tenant has
# access to - such as Compute, Image Service, Identity, Object Storage, Block
# Storage, and Networking (code-named nova, glance, keystone, swift,
# cinder, and neutron).
#
# *NOTE*: Using the 3 *Identity API* does not necessarily mean any other
# OpenStack API is version 3. For example, your cloud provider may implement
# Image API v1.1, Block Storage API v2, and Compute API v2.0. OS_AUTH_URL is
# only for the Identity API served through keystone.
export OS_AUTH_URL={auth_url}
# With the addition of Keystone we have standardized on the term **project**
# as the entity that owns the resources.
export OS_PROJECT_ID={project_id}
export OS_PROJECT_NAME="{project_name}"
export OS_USER_DOMAIN_NAME="Default"
if [ -z "$OS_USER_DOMAIN_NAME" ]; then unset OS_USER_DOMAIN_NAME; fi
export OS_PROJECT_DOMAIN_ID="default"
if [ -z "$OS_PROJECT_DOMAIN_ID" ]; then unset OS_PROJECT_DOMAIN_ID; fi
# unset v2.0 items in case set
unset OS_TENANT_ID
unset OS_TENANT_NAME
# In addition to the owning entity (tenant), OpenStack stores the entity
# performing the action as the **user**.
export OS_USERNAME="{username}"
# With Keystone you pass the keystone password.
#echo "Please enter your OpenStack Password for project $OS_PROJECT_NAME as user $OS_USERNAME: "
#read -sr OS_PASSWORD_INPUT
export OS_PASSWORD={password}
# If your configuration has multiple regions, we set that information here.
# OS_REGION_NAME is optional and only valid in certain environments.
export OS_REGION_NAME="{region_name}"
# Don't leave a blank variable, unset it if it was empty
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION=3
"""
open(openrc_file, 'w').write(openrc_2)
assert DotEnoughOpenStack.openrc2clouds(
openrc_file, clouds_file) == expected['clouds']['production']
assert yaml.safe_load(open(clouds_file)) == expected
os.unlink(clouds_file)
expected['clouds']['production']['auth']['password'] = 'PLACEHOLDER'
openrc_3 = f"""
export OS_AUTH_URL={auth_url}
export OS_PROJECT_ID={project_id}
export OS_PROJECT_NAME="{project_name}"
export OS_USERNAME="{username}"
read -sr OS_PASSWORD_INPUT
export OS_PASSWORD=$OS_PASSWORD_INPUT
export OS_REGION_NAME="{region_name}"
"""
open(openrc_file, 'w').write(openrc_3)
assert DotEnoughOpenStack.openrc2clouds(
openrc_file, clouds_file) == expected['clouds']['production']
assert yaml.safe_load(open(clouds_file)) == expected
os.unlink(clouds_file)
assert DotEnoughOpenStack.set_missing_config_file_from_openrc(openrc_file, clouds_file) is True
assert DotEnoughOpenStack.set_missing_config_file_from_openrc(openrc_file, clouds_file) is False

48
tests/enough/common/test_init.py

@ -36,8 +36,7 @@ def test_clone_and_destroy(request, tmpdir):
original = Enough(original_config_dir, settings.SHARE_DIR,
domain=original_domain,
driver='openstack',
provider=request.config.option.provider)
driver='openstack')
assert original.openstack.o.server.list().strip() == ''
clone_domain = 'clone.com'
@ -55,7 +54,7 @@ def test_clone_and_destroy(request, tmpdir):
assert not os.path.exists(clone.config_dir)
def create_enough(provider, tmpdir, dotenough, **kwargs):
def create_enough(tmpdir, dotenough, **kwargs):
enough_domain = 'enough.com'
@ -70,14 +69,14 @@ def create_enough(provider, tmpdir, dotenough, **kwargs):
enough = Enough(config_dir, settings.SHARE_DIR,
domain=enough_domain,
driver='openstack',
provider=provider,
**kwargs)
return enough
def test_clone_clobber(request, tmpdir):
provider = request.config.option.provider
original = create_enough(provider, tmpdir, 'copy')
original = create_enough(tmpdir, 'copy')
shutil.copyfile('tests/enough/common/test_init/clone-clobber/clouds.yml',
f'{original.config_dir}/inventory/group_vars/all/clouds.yml')
clone_domain = 'clone.com'
clone = original.clone(clone_domain, 'clone', False)
assert os.path.exists(f'{clone.config_dir}')
@ -91,8 +90,8 @@ def test_clone_clobber(request, tmpdir):
assert not os.path.exists(stone)
def create_and_clone_server_and_volume(provider, tmpdir):
original = create_enough(provider, tmpdir, 'backup')
def create_and_clone_server_and_volume(tmpdir):
original = create_enough(tmpdir, 'backup')
original.set_args(name='sample', playbook='enough-playbook.yml')
original.service.create_or_update()
@ -106,9 +105,8 @@ def create_and_clone_server_and_volume(provider, tmpdir):
@pytest.mark.openstack_integration
def test_clone_create_service(request, tmpdir):
provider = request.config.option.provider
try:
(original, clone) = create_and_clone_server_and_volume(provider, tmpdir)
(original, clone) = create_and_clone_server_and_volume(tmpdir)
assert 'sample-host' in original.openstack.o.server.list()
assert 'sample-volume' in original.openstack.o.volume.list()
assert 'sample-host' in clone.openstack.o.server.list()
@ -122,9 +120,8 @@ def test_clone_create_service(request, tmpdir):
@pytest.mark.openstack_integration
def test_create_copy_host(request, tmpdir):
provider = request.config.option.provider
try:
enough = create_enough(provider, tmpdir, 'copy')
enough = create_enough(tmpdir, 'copy')
if 'copy-volume' not in enough.openstack.o.volume.list():
enough.openstack.o.volume.create('--size', '1', 'copy-volume')
ip = enough.create_copy_host('copy-from-host', 'some-volume', 'copy-volume')
@ -142,9 +139,8 @@ def test_create_copy_host(request, tmpdir):
@pytest.mark.openstack_integration
def test_create_volume_from_snapshot(request, tmpdir):
provider = request.config.option.provider
try:
e = create_enough(provider, tmpdir, 'backup')
e = create_enough(tmpdir, 'backup')
e.set_args(name='sample', playbook='enough-playbook.yml')
e.service.create_or_update()
e.openstack.backup_create(['sample-volume'])
@ -164,9 +160,8 @@ def test_create_volume_from_snapshot(request, tmpdir):
@pytest.mark.openstack_integration
def test_clone_volume_from_snapshot(request, tmpdir):
provider = request.config.option.provider
try:
original = create_enough(provider, tmpdir, 'backup')
original = create_enough(tmpdir, 'backup')
original.set_args(name='sample', playbook='enough-playbook.yml')
original.service.create_or_update()
ip = original.hosts.load().get_ip('sample-host')
@ -206,9 +201,8 @@ def test_clone_volume_from_snapshot(request, tmpdir):
@pytest.mark.openstack_integration
def test_create_service_matching_snapshot(request, tmpdir):
provider = request.config.option.provider
try:
enough = create_enough(provider, tmpdir, 'backup')
enough = create_enough(tmpdir, 'backup')
host = enough.create_service_matching_snapshot('2020-02-20-sample-volume')
assert host == 'sample-host'
assert 'sample-volume' in enough.openstack.o.volume.list()
@ -221,9 +215,8 @@ def test_create_service_matching_snapshot(request, tmpdir):
@pytest.mark.openstack_integration
def test_restore_remote(request, tmpdir):
provider = request.config.option.provider
try:
original = create_enough(provider, tmpdir, 'backup')
original = create_enough(tmpdir, 'backup')
original.set_args(name='sample', playbook='enough-playbook.yml')
original.service.create_or_update()
hosts = original.hosts.load()
@ -265,9 +258,8 @@ def test_restore_remote(request, tmpdir):
@pytest.mark.openstack_integration
def test_restore_local(request, tmpdir):
provider = request.config.option.provider
try:
e = create_enough(provider, tmpdir, 'backup')
e = create_enough(tmpdir, 'backup')
e.set_args(name='sample', playbook='enough-playbook.yml')
e.service.create_or_update()
hosts = e.hosts.load()
@ -297,9 +289,8 @@ def test_restore_local(request, tmpdir):
@pytest.mark.openstack_integration
def test_create_missings(request, tmpdir):
provider = request.config.option.provider
try:
enough = create_enough(provider, tmpdir, 'create-missings')
enough = create_enough(tmpdir, 'create-missings')
r = enough.create_missings(['bind-host'])
assert 'bind-host' in r
internal_dns = enough.openstack.o.subnet.show(
@ -337,8 +328,7 @@ def test_volume_from_snapshot():
def test_host_from_snapshot(request, tmpdir):
provider = request.config.option.provider
e = create_enough(provider, tmpdir, 'backup')
e = create_enough(tmpdir, 'backup')
host = 'sample-host'
snapshot = f'2010-07-08-sample-volume'
assert e.host_from_snapshot(snapshot) == host
@ -346,9 +336,8 @@ def test_host_from_snapshot(request, tmpdir):
@pytest.mark.openstack_integration
def test_volume_resize(request, tmpdir):
provider = request.config.option.provider
try:
enough = create_enough(provider, tmpdir, 'resize')
enough = create_enough(tmpdir, 'resize')
enough.set_args(name='sample', playbook='enough-playbook.yml')
enough.service.create_or_update()
# False because the size does not change
@ -368,9 +357,8 @@ def test_volume_resize(request, tmpdir):
#
@pytest.mark.openstack_integration
def test_host_create_or_update(request, tmpdir, openstack_name):
provider = request.config.option.provider
try:
enough = create_enough(provider, tmpdir, 'host')
enough = create_enough(tmpdir, 'host')
enough.set_args(name='new-host')
info = enough.host.create_or_update()
assert 'ipv4' in info

20
tests/enough/common/test_init/clone-clobber/clouds.yml

@ -0,0 +1,20 @@
---
clouds:
production:
auth:
auth_url: "https://auth.cloud.ovh.net/v3" # OS_AUTH_URL
project_name: "XXXXXX" # OS_PROJECT_NAME
project_id: "XXXXX" # OS_PROJECT_ID
user_domain_name: "Default" # OS_USER_DOMAIN_NAME
username: "XXXXX" # OS_USERNAME
password: "XXXXX" # OS_PASSWORD
region_name: "PROD" # OS_REGION_NAME
clone:
auth:
auth_url: "https://auth.cloud.ovh.net/v3" # OS_AUTH_URL
project_name: "XXXXXX" # OS_PROJECT_NAME
project_id: "XXXXX" # OS_PROJECT_ID
user_domain_name: "Default" # OS_USER_DOMAIN_NAME
username: "XXXXX" # OS_USERNAME
password: "XXXXX" # OS_PASSWORD
region_name: "CLON" # OS_REGION_NAME

16
tests/enough/common/test_openstack.py

@ -15,13 +15,13 @@ from enough.common.openstack import Stack, Heat, OpenStack
# Stack
#
@pytest.mark.openstack_integration
def test_stack_create_or_update(openstack_name, openstack_provider,
def test_stack_create_or_update(openstack_name, openstack_variables,
dot_openstack):
name = openstack_name
network = openstack_name
class_c = '10.101.30'
cidr = f'{class_c}.0/24'
d = Heat.hostvars_to_stack(openstack_name, openstack_provider)
d = Heat.hostvars_to_stack(openstack_name, openstack_variables)
d.update(
internal_network=network,
internal_network_cidr=cidr
@ -177,8 +177,8 @@ def test_backup_create_no_names(openstack_name, dot_openstack, caplog):
@pytest.mark.openstack_integration
def test_openstack_replace_volume(openstack_name, dot_openstack,
openstack_provider):
d = Heat.hostvars_to_stack(openstack_name, openstack_provider)
openstack_variables):
d = Heat.hostvars_to_stack(openstack_name, openstack_variables)
d.update({
'volumes': [
{
@ -217,9 +217,9 @@ def test_openstack_replace_volume(openstack_name, dot_openstack,
@pytest.mark.openstack_integration
def test_openstack_volume_resize_ok(openstack_name, dot_openstack,
openstack_provider):
openstack_variables):
size = 1
d = Heat.hostvars_to_stack(openstack_name, openstack_provider)
d = Heat.hostvars_to_stack(openstack_name, openstack_variables)
d.update({
'volumes': [
{
@ -253,8 +253,8 @@ def test_openstack_volume_resize_ok(openstack_name, dot_openstack,
@pytest.mark.openstack_integration
def test_openstack_volume_resize_no_volume(openstack_name, dot_openstack, openstack_provider):
d = Heat.hostvars_to_stack(openstack_name, openstack_provider)
def test_openstack_volume_resize_no_volume(openstack_name, dot_openstack, openstack_variables):
d = Heat.hostvars_to_stack(openstack_name, openstack_variables)
s = Stack(settings.CONFIG_DIR, d)
s.set_public_key(f'{settings.CONFIG_DIR}/infrastructure_key.pub')
s.create_or_update()

33
tests/run-tests.sh

@ -26,32 +26,21 @@ function prepare_repository() {
}
function prepare_inventory() {
if ! test -f inventory/group_vars/all/clouds.yml ; then
if test -z "$OS_AUTH_URL" ; then
echo OS_AUTH_URL is not set, source openrc.sh
exit 1
fi
cat > tests/clouds.yml <<EOF
---
if test "$GITLAB_CI" ; then
cat > tests/clouds.yml <<EOF
clouds:
production:
auth: &os_auth
auth_url: "$OS_AUTH_URL"
project_name: "${OS_PROJECT_NAME:-$OS_TENANT_NAME}"
project_id: "${OS_PROJECT_ID:-$OS_TENANT_ID}"
user_domain_name: "$OS_USER_DOMAIN_NAME"
username: "$OS_USERNAME"
password: "$OS_PASSWORD"
region_name: "$OS_REGION_NAME"
clone:
auth:
<<: *os_auth
region_name: "DE1"
auth_url: "https://auth.cloud.ovh.net/v3/"
project_name: "AAAAAAA"
project_id: "BBBBB"
user_domain_name: "CCCCCCC"
username: "DDDDDDD"
password: "EEEEEEEE"
region_name: "FFFFFF"
EOF
cp tests/clouds.yml inventory/group_vars/all/clouds.yml
else
cat inventory/group_vars/all/clouds.yml > tests/clouds.yml
fi
cp -a tests/clouds.yml inventory/group_vars/all/clouds.yml
if ! test -f inventory/group_vars/all/domain.yml ; then
cat > tests/domain.yml <<EOF
@ -112,7 +101,7 @@ function run_tests() {
local interactive=-ti
fi
container_name="$name-$(date +%s)"
docker run $interactive --rm --device=/dev/net/tun --name $container_name --user "${USER:-root}" -e PYTEST_ADDOPTS="${PYTEST_ADDOPTS-}" -e ENOUGH_PROVIDER=$ENOUGH_PROVIDER -e ENOUGH_API_TOKEN=$ENOUGH_API_TOKEN ${PASS_GITLAB_CI} --cap-add=NET_ADMIN $args --volume /var/run/docker.sock:/var/run/docker.sock $name "${@:-tox}"
docker run $interactive --rm --device=/dev/net/tun --name $container_name --user "${USER:-root}" -e PYTEST_ADDOPTS="${PYTEST_ADDOPTS-}" -e ENOUGH_API_TOKEN=$ENOUGH_API_TOKEN ${PASS_GITLAB_CI} --cap-add=NET_ADMIN $args --volume /var/run/docker.sock:/var/run/docker.sock $name "${@:-tox}"
}
function main() {

2
tox.ini

@ -27,7 +27,6 @@ commands = flake8 {posargs}
[testenv:{infrastructure,bind,authorized_keys,backup,certificate,postfix,icinga,openvpn,wekan,misc,pad,firewall,gitlab,api,wazuh,weblate,website,chat,cloud,enough,forum,packages,securedrop,jitsi,wordpress,openedx,psono}]
passenv =
ENOUGH_API_TOKEN
ENOUGH_PROVIDER
PYTEST_ADDOPTS
# REQUESTS_CA_BUNDLE is set in enough/common/data/base.dockerfile but inconvenient in a test environment
commands = env --unset=REQUESTS_CA_BUNDLE {envbindir}/py.test --log-cli-level INFO -s --ssh-identity-file=infrastructure_key --ansible-inventory={envdir}/.pytest_cache/d/dotenough/{envname}.test/inventory {posargs:playbooks/{envname}/tests}
@ -35,7 +34,6 @@ commands = env --unset=REQUESTS_CA_BUNDLE {envbindir}/py.test --log-cli-level IN
[testenv:enough_nginx]
passenv =
ENOUGH_API_TOKEN
ENOUGH_PROVIDER
PYTEST_ADDOPTS
# REQUESTS_CA_BUNDLE is set in enough/common/data/base.dockerfile but inconvenient in a test environment
commands = env --unset=REQUESTS_CA_BUNDLE {envbindir}/py.test --log-cli-level INFO -s --ssh-identity-file=infrastructure_key --ansible-inventory={envdir}/.pytest_cache/d/dotenough/enough-nginx.test/inventory {posargs:playbooks/enough-nginx/tests}

Loading…
Cancel
Save