Is this issue:

1. about Ansible? Are you trying to say that Ansible was wrong choice?
2. or about LOCK_ON_FILE in inventory/standard-inventory-qcow2 script?
3. or about: that we should not use add_host?

Quote: it's far easier to address this now, rather than later.
What do you mean under this?

As I understand this issue #150 is more like general recommendation. Right?
No specific bug is addressed in this issue. Right?

By the way, I checked your python code, yml files, etc, from your github project.
I could not manage to understand what it was all about.
What are you trying to show in your github project and this issue?

Please rename the issue's topic to more appropriate.

@cevich Thanks for your feedback. I don't have your in depth experience, so I'd be eager to hear more about the concrete changes you suggest making to the standard test roles.

Maybe we can also separate between conceptual / architectural proposed changes and ones proposed to the concrete implementation. I think @astepano is looking for more concrete issues to tackle in these "issues", while what you describe sounds more like an overarching approach.

Ansible was wrong choice?

Not necessarily. This is more about the way it's used expected/supported interactions by subject-matter playbooks.

I could not manage to understand what it was all about.

That project is just an example/demo of how ansible can be used with both inventory-encoded details, with playbooks application of desired state. So, state transitions, not imperative decisions/control. I understand/appreciate this is somewhat difficult to wrap your head around, it certainly was for me.

while what you describe sounds more like an overarching approach

Yes, this is the dialog I wanted to start. As a concrete-issue, I pointed out the example use-case of LOCK_ON_FILE + `add_host'. I'm happy to split/fork this into a separate issue, but wanted to get and provide more context first. I'm open that maybe this isn't a practically resolvable problem.

The core of the conceptual issue is: Ansible YAML is not imperative. It's a declarative syntax, meaning "document-like" or "statement of fact". Think "Makefile" vs "script".
It's very very easy to start tossing control structures and conditions all over the place.

In my experience this leads to a very big and difficult to fix, complex problems as playbooks and roles grow larger. It's far easier to stick any needed logic into scripts, and keep playbooks very matter-of-fact, or declarations-of-state.

So as for, how/what needs fixing. I'm not clear on what the easiest path would be. My recommendations would be:

1) Drop the $LOCK_ON_FILE implementations and add_host example.

-or-

2) Do all the subject creation by script, before the main testing-related play(s) run.

-or-

3) Re-think the need for inventory-state being tied (by monitor process) to the ansible-playbook pid. Use a different way to clean up, maybe from a higher layer.

Does that help clarify what/why I raised this?

How else can I help?

Couldn't sleep last night :frowning: , kept thinking about this issue. I believe I discovered a workable solution that should dramatically improve overall conditions:

Separate/modularize responsibilities across all the inventory scripts with three "library" modules (they always print {} when called):

• An inventory module that only provides state representation. Other modules use it's classes/functions that add/delete hosts and print inventory to stdout. Data is cached/persists centrally, in an $ARTIFACTS file.

• A cleanup module that keeps track of subjects, removing them from inventory when necessary, then exits (i.e. not a daemon). Monitoring targets are described by subject host-variables, discovered by running the ansible-inventory command. e.g monitor_pid: F00 or monitor_file: bar.

• An module that only provides debugging/logging functions/classes. Used by all others to coordinate message format and keep $ARTIFACTS log files uniform.

Lastly:

• Subject-creation inventory scripts, similar as-is, but they only do that task, relying on the above for everything else. Concluding with a call to the inventory module to produce the current inventory state on stdout.

Final cleanup is realized by execution of cleanup module at the conclusion of testing. This could be enforced by humans (require meta: refresh_inventory at end of every play in the CI-spec.) and/or automatically by a final call from within the infrastructure (outside of Ansible) or by human.

Benefits (tl;dr):

1. Host creation and management logic is kept in inventory and out of playbooks. Playbooks need only declare desired idempotent-state from one task to the next.

2. Discrete components with singular purpose and wide-reuse vastly limit wheel-reinvention.

3. Modules with clear boundaries are easy to test and debug.

4. Existing infrastructure compatibility is preserved, with optional final infrastructure (or human) call to cleanup module.

5. The subject-creation inventory scripts can largely remain as-is. Only the common elements need to be rewritten to utilize the library functions/classes.

6. No complicated, hard to test and debug daemons are required. Finalization choice/flexibility means either/both humans or infrastructure can decide when it's death-time.

7. Supports debugging by not finalizing, and simply re-running playbooks on the living subjects.

Whew! Thoughts? Questions?

Hi Christopher!

Couldn't sleep last night 😦 , kept thinking about this issue. I believe I discovered a workable solution that should dramatically improve overall conditions:
Separate/modularize responsibilities across all the inventory scripts with three "library" modules (they always print {} when called):

An inventory module that only provides state representation. Other modules use it's classes/functions that add/delete hosts and print inventory to stdout. Data is cached/persists centrally, in an $ARTIFACTS file.

1. I fill a bit sceptical about Separate/modularize responsibilities. This will force all inventory scripts to use same library, same Python, same approach. For me this is some kind of limitation. If we follow this approach than we do not need to implement many inventory scripts. There could be 1 script that does all logic.

2. I do not like idea of caching inventory. This will lead only to complications. Think about synchronising actual environment and old cache, for example. I would like to have stateless inventory, otherwise we will hit many hidden pitfalls, and complication in implementation.

A cleanup module that keeps track of subjects, removing them from inventory when necessary, then exits (i.e. not a daemon). Monitoring targets are described by subject host-variables, discovered by running the ansible-inventory command. e.g monitor_pid: F00 or monitor_file: bar.

This is OK for library/modularize approach.

An module that only provides debugging/logging functions/classes. Used by all others to coordinate message format and keep $ARTIFACTS log files uniform.

In general I like your ideas. But this will require a lot of modifications.
I think this is blocked now by 'localhost' host/group approach.
First step would be adapting of STR specification and play with : Host creation and management logic is kept in inventory and out of playbooks.

Lastly:

Subject-creation inventory scripts, similar as-is, but they only do that task, relying on the above for everything else. Concluding with a call to the inventory module to produce the current inventory state on stdout.

Final cleanup is realized by execution of cleanup module at the conclusion of testing. This could be enforced by humans (require meta: refresh_inventory at end of every play in the CI-spec.) and/or automatically by a final call from within the infrastructure (outside of Ansible) or by human.

Strongly agree. We should think if we want to give user ability to act on $ARTIFACTS during tests run.

Benefits (tl;dr):

Host creation and management logic is kept in inventory and out of playbooks. Playbooks need only declare desired idempotent-state from one task to the next.

YES!!

Discrete components with singular purpose and wide-reuse vastly limit wheel-reinvention.

I think this can lead to some limitations. But,,, on the other hand it will help us to keep all under control.

Modules with clear boundaries are easy to test and debug.

I would call 'Modules' as 'Inventory-modules'. Otherwise, hard to understand what is module.

Existing infrastructure compatibility is preserved, with optional final infrastructure (or human) call to cleanup module.

The subject-creation inventory scripts can largely remain as-is. Only the common elements need to be rewritten to utilize the library functions/classes.

I doubt that inventory scripts can remain as-is :) This propose suggests many changes.

No complicated, hard to test and debug daemons are required. Finalization choice/flexibility means either/both humans or infrastructure can decide when it's death-time.

Should we start this daemons from inventory-scripts?

Supports debugging by not finalizing, and simply re-running playbooks on the living subjects.

Whew! Thoughts? Questions?

Yes. LOCK_ON_FILE was introduced by: https://pagure.io/standard-test-roles/pull-request/119#
by @ttomecek

There is explanation why we need LOCK_ON_FILE https://pagure.io/standard-test-roles/blob/master/f/README.md

By default, the virtual machine is killed when the process (your shell) it invoked the script, is gone. This behavior may not be desirable when invoking the inventory script from an Ansible playbook (since Ansible spawns a long hierarchy of processes). For that purpose you can use environment variable LOCK_ON_FILE

I am going to close this ticket.
Please open tickets for each separate issue.
Otherwise, there is a big chance to get lost in solving many issues in one ticket, and loose some important idea.

I fill a bit sceptical about Separate/modularize responsibilities.

The inventory-scripts must all be python2/3 compatible, this is a requirement if python3-ansible is being used. So it's a very safe assumption to make. Having separate modules vastly improves re-use of code, making behaviors consistent, and makes unit testing far easier.

I do not like idea of caching inventory. This will lead only to complications.

Synchronizing actual environment with inventory is a vastly more complicated thing to do, and it's unnecessary - ansible will fail unreachable hosts, which is desired behavior.

Caching is more or less a requirement by the dynamic-inventory spec. Without it, there's no way to provide a uniform contents across multiple calls. Remember, inventory is an INPUT to ansible commands. Inputs should change during command execution.

In practice, it's an optimization to remove known-dead hosts, otherwise even that's not strictly needed. Only some sort of "add" mechanism is required.

I would call 'Modules' as 'Inventory-modules'. Otherwise, hard to understand what is module.

Sorry, my confusion. I ment "python modules" for "ansible inventory modules" :D

I am going to close this ticket.
Please open tickets for each separate issue.

Fair enough, and thanks for the discussion. I agree, fixing the higher-layer interface problems and 'localhost' group thing, is probably the most important thing.

Caching is more or less a requirement by the dynamic-inventory spec. Without it, there's no way to provide a uniform contents across multiple calls. Remember, inventory is an INPUT to ansible commands. Inputs should change during command execution.

@cevich I see, I am feeling that there is some very important thought.

Could you please explain more about Inputs should change during command execution.

Ansible inventory becomes test-environment.
We can define test_environment before any test run.
This test_environment should stay static.

I opened an issue for discussing caching:

https://pagure.io/standard-test-roles/issue/167

• Close this ticket. Thank you!

Oops, I mean "Inputs should not change during execution"