Comments from Taiga #405

Richard England 08 Dec 2021 11:34
*thunderbirdtr Checking in with you to see if you have an status update on this article.

Richard England11 Nov 2021 11:21
Thanks for the response. That sounds fine, *thunderbirdtr Do the “copr” version for now. We can either modify that article to include the RPM version or do a follow-on article that includes RPM and some more info/examples/whatever in the future. Your choice.

Onuralp SEZER11 Nov 2021 11:15
That link has evolved and become and can't be seen by public but status is I can complete the article and If that's okay I would like to do it after the release party and handle it probably in “sunday”

Problem was : copr packages was missing and I asked for help and that worked, I'm also working on bringing openfoam in to “fedora repo” that could take some time but when happened I can ask for revisit and change that part in to “copr" to “direct dnf install mod” without adding copr repo as well. If that's okay I wanna move on with “copr” repo and complete as it is at sunday and release it ?

Richard England 11 Nov 2021 08:42
*thunderbirdtr The “road bump” link was a 404 for me. I don't see anything in the repo at this time. Any update on the project and your article?

Gregory Lee Bartholomew 26 Oct 2021 14:22
Unless you see some reason to go with the Copr repo, I would wait for the rpm. I don't think we are in too big of a need for articles to publish at the moment.

Thanks.

Onuralp SEZER 26 Oct 2021 02:01
Tiny road bump : https://develop.openfoam.com/Development/openfoam/-/issues/2252

I can include via my copr repo but I wanna make article as clean as possible as well.

@thunderbirdtr Checking in with you to see if you have made any progress on this article.

When it is ready for review, please let us know by adding a WordPress preview link here in the comments.

Thanks

@thunderbirdtr I noticed that you have started your article in WordPress so I added the link to it on this card.

Let us know when you have it ready for review.

Thanks.

@thunderbirdtr Do you have a status update on this article?

@thunderbirdtr When you have this article ready for review, don't forget to leave a comment here on this card to let the editor know.

Thanks.

Hey @thunderbirdtr, are you still planning on writing this article?

Oh, sorry for late answers, yes I should finish as early as I can which most of them was almost done already. Let me clean-up and finish it soon.

thunderbirdtr It doesn't look like this article is finished yet. Please be sure to comment on the card to let us know when you think it is ready for review.

Good afternoon. It looks like an interesting article and although I used matlab and other tools in University I would be interested in giving this a try...

@romangherta I don't think there should be a problem with you taking this on but let's give @thunderbirdtr a chance to respond with their status/interest.

Perhaps wait a week to see what they say?

Agreed, btw if I work on this it would probably take me the entire Q1 2024... just saying, but it's somethin I always had on the todo list somewhere in the back of my head.

@romangherta I've made it official and assigned this article to you, since I see there has been significant work done on it in WordPress.

When you have it completed/polished to your liking, let the editors know. Do that by placing a link to the WP preview site for the article in a comment on this ticket.

Thank you for contributing.

Hi @rlengland I am still probably a few weeks away from finishing but I will try to update periodically. I have seen @thunderbirdtr 's github account. He is really busy atm. A nice week ahead.

Sorry for the delay, I want to finish reading the openfoam11 usermanual at least once... I have 5 more chapters remaining. It is an interesting field that covers just too many subjects. It is hard to delimit what a CFD engineer should know and what a systems admin should know.... I\ll get back with an update in a week. Right now I feel like I will still be working on this in April.

@romangherta how is this article coming along?

Good evening, it is harder than I thought. I just updated a small thing and made the openfoam user confined to selinux user_u . I need to double check again the tests and then revert to other bulletpoints I placed in the draft that are more related to the systems administration and less to differential equation and cfd working knowledge. I will submit some changes by the end of this weekend. Thanks for the reminder.

@romangherta How is the work on this article going?

Good morning @rlengland , @glb ... I am trying to figure out some confusion with respect to who is maintaining what. I can see there is Openfoam Foundation (openfoam.org) that is collaborating or maintained by this company called cfd.direct and its source code is located here https://github.com/OpenFOAM/OpenFOAM-dev . The distinctive feature of the source code is the presence of the generic modular solver foamRun.C that was introduced in latest 11 version to simplify and make code modular and extensible by a guy called Henry Weller that seems to be the first architect of openfoam. Openfoam Foundation is dedicated according to the webpage to providing libraries under gpl license.

Then we have openfoam.com that is maintained by a company called either OpenCFD Ltd and/or ESI Group.The source code is located here https://gitlab.com/openfoam/openfoam . There are no generic solvers here, more similar to versions prior to 2015, and everything is a little bit different, but there is a beautiful nice shell, openmpi, dnf copr package etc. And this is the source code used for the dnf copr package which is called ... openfoam.... From what I can read, they used to actively collaborate on the Openfoam Foundation libraries until 2015.

So I am trying to figure out if I should use the fedora nice packages that can be installed with dnf install openfoam maintained by ESI guys ... Or I need to use the openfoam foundation git repo, clone it and then build the packages and only then continue with tutorials.

It seems to me there is a drift in this community... I think this article provides some historical context better than I can explain
https://openfoam.org/ip-history/

All in all we have a github repo called openfoam, and another gitlab repo called openfoam, and a dnf copr package called... you guessed - openfoam.

Let me know your thoughts. Thank you

@romangherta This page
https://openfoam.org/
under the section called "OpenFOAM is developed by contributors, primarily from CFD Direct"
seems to indicate the code development is managed by CFD/ESI. The foundation appears to be there to ensure that the code is maintained and is available as open source, as far as I can see. I'm not certain why there are two source code repositories available....

https://www.openfoam.com/news/community-projects
Indicates that the ESI version conforms to GPLv3

CFD/ESI makes a COPR version available here:
https://develop.openfoam.com/Development/openfoam/-/wikis/precompiled/redhat

Unless you want to spend time in the article helping users build from source code, I would suggest using the COPR version. Note that you may receive some flak from those that don't care for COPR but that so it is.

I can't speak to the availability of solvers, unless there are 3rd party contributors that may help out with that.

@glb Your thoughts?

I will proceed with copr packages. I realize now I was reading the wrong user manual and looking for a missing solver. The openfoam foundation github repo has a solvers/legacy folder that is similar to openfoam.com , but based on what I read I think this is not just a version skew.

Good morning, as a small update. I think I am finished with the Heading Simulations . The only thing remaining is the hpc considerations and some talks about how parallelism is implemented and security, how it supports only cpus.. although I see these ESI guys have some articles about openfoam running on gpus, or its just a particular solver.... I am still looking into this, probably will finish tomorrow.

Good afternoon, I think I finished. Please take a look and let me know your thoughts or feel free to edit as you see fit, I might reply slower in the upcoming weeks.

@romangherta Your article is schedule for publication 7 June at 0800 UTC.
I've added a featured image which I hope will meet with you approval. I did some sentence structure changes for a bit more clarity and simplification.

This would be a good time to review the image and the article one last time. If there are changes let me and we can switch the article back to "draft" in WordPress so you can make them, or we can do it for you if they are minor.

Thank you for your hard work.

I made an editing pass on this one as well, so not all the changes/tweaks are Richard's fault. 🙂 It looks good to me. Thanks again for your hard work!

Since ParaView uses XServer, we need to temporarily grant “ofuser” the necessary permissions via the xhost command.

Do we want to note in that sentence that ParaView will also work under XWayland? E.g.:

Since ParaView uses XServer (or XWayland), we need to temporarily grant “ofuser” the necessary permissions via the xhost command.

If you don't, you might get a series of comments stating that some editions/spins of Fedora Linux no longer install X.Org by default.

@glb I've made that change :thumbsup:
@romangherta

Afternoon @glb

I used the sudo -iu ofuser from the beginning for rapid testing, but from what I tested, gnome logout/login also works, so I would add a "sudo passwd ofuser" right after useradd and instruct to use gnome login. This normally would be part of ldap account and sssd config, but for this small tutorial it may be too much information.

As for the type of graphics, this is way below my knowledge of what it uses. I notice some people are complaining online about paraview with wayland. I don't know what to say about that, if you give me some time I can test with wayland. I will leave you to decide on how to correct or proceed.

@rlengland thank you for your time, it looks great. I cannot wait to see in the comments which version of openfoam is the "real" openfoam.

@romangherta There is no reason we have to publish on the 7th so I have changed the status back to "draft" (unscheduled). Do the testing/editing you feel appropriate and let us know when you have any adjustments made and we are ready to go.

Hello again, I tested with Gnome on Xorgs, and Gnome default Wayland It seems to work in both cases, I also simplified a little bit the terminal commands, I removed sudo -iu and xhost and mentions of xserver.

One thing that I noticed is that in VM I have an error/warning when launching paraFoam/paraview, specificaly

ofuser$ paraview
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen

Same steps on my host laptop with F40 does not throw these errors, and the graphics animation is similar to the VM animation, this is all I can add. Maybe some of my VM settings are not properly set.

Mesa is for software rendering. Software rendering is used in the VM because it doesn't have access to your physical graphics card (unless you enable GPU passthrough, but that is non-trivial to accomplish).

I'm not familiar with that error. A quick google search suggests that it might be caused by a missing software package: https://github.com/BelledonneCommunications/linphone-desktop/issues/830

From Wikipedia -- Mesa_(computer_graphics):

Mesa also contains an implementation of software rendering called swrast that allows shaders to run on the CPU as a fallback when no graphics hardware accelerators are present.

Not this specific package, but I will double check dnf list install from my laptop and from vm, maybe I find the missing package...

I ll try to look into this weekend

I don't know how deep you want to get into SELinux, but you might mention that the sesearch command can be used to list what access a confined user will have to files with a given file label. For example, sesearch -A -s user_t -t lib_t.

Yep, good idea. I also want to add a few lines about saving data to
directly attached diskas for faster IO, stbled upon this in the docs
yesterday.

I LL get back in the weekend

On Fri, Jun 7, 2024, 05:58 Gregory Lee Bartholomew pagure@pagure.io wrote:

glb added a new comment to an issue you are following:
I don't know how deep you want to get into SELinux, but you might mention that the `sesearch` command can be used to list what access a confined user will have to files with a given file label. For example, `sesearch -A -s user_t -t lib_t`.

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-magazine-newsroom/issue/72

Hello again

I added a few lines about improved IO by writing to attached disks instead of homedirs, appended the selinux line and some extra lines for better user insights into access , replaced the paralllel run commands with mpirun equivalents.

I have some insights into the Xorg discussion from earlier today... So when I tested by choosing Gnome Xorg and Gnome Wayland from the gnome login page, it did not actually make a difference since apparently xorg is enabled by default for compatibility reasons even in wayland. This explains why my xhost command worked previously. In Fedora 41 xorg will be completely removed. as of now f40 we still have a few packages

localuser@fedora:~$ dnf list installed | grep -E "xorg|xhost"
abrt-addon-xorg.x86_64                               2.17.5-1.fc40                       @anaconda                                        
xorg-x11-drv-libinput.x86_64                         1.4.0-2.fc40                        @anaconda                                        
xorg-x11-server-Xorg.x86_64                          1.20.14-35.fc40                     @updates                                         
xorg-x11-server-Xwayland.x86_64                      24.1.0-1.fc40                       @updates                                         
xorg-x11-server-common.x86_64                        1.20.14-35.fc40                     @updates                                         
xorg-x11-xauth.x86_64                                1:1.1.2-6.fc40                      @anaconda                                        
xorg-x11-xinit.x86_64                                1.4.2-2.fc40                        @anaconda 
localuser@fedora:~$ rpm -q --whatrequires   xorg-x11-xinit
ibus-1.5.30-1.fc40.x86_64
gdm-46.2-1.fc40.x86_64

I uninstalled xorg-x11-server-Xorg but there are other packages that are required by gnome-shell...

As for the specific paraview error/warning that appears on VMM but not on host laptop with videocard... It is indeed using mesa3d like you said for software rendering. Zinc is a specific mesa driver that uses vulkan apis.... which are isntalled as I can see...

ofuser$ paraview
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
localuser@fedora:~$ dnf list installed | grep vulkan
mesa-vulkan-drivers.x86_64                           24.0.9-1.fc40                       @updates                                         
vulkan-loader.x86_64                                 1.3.283.0-1.fc40                    @updates 

Done the same steps with fedora40-kde. I see in dnf list it has same xorg packages and the warning/error persists.

According to mesa3d docs we have the following drivers
https://docs.mesa3d.org/systems.html

error seems refring to DRI drivers for x windows systems... It seems to try xorg then falls back to something.... If I run "paraview --disable-xdisplay-test" I get only one such pair of warnings not two... I found some instructions on how to build paraview with llvmpipe as backend but it would take forever. I can see this error is not scoped to paraview only

localuser@fedora:~$ gnome-control-center
MESA: error: ZINK: failed to choose pdev
libEGL warning: egl: failed to create dri2 screen

So I don't know what is happening here, probably related to package mesa-dri-drivers

If you want I can investigate more, but as of now this is all I have

ok, I think I am done editing this comment

Small update... looking into mesa environment variables, I found this

https://docs.mesa3d.org/envvars.html

LIBGL_ALWAYS_SOFTWARE=true gnome-control-center

This way windows are launching w/o any warnings/errors. I think it is some default behaviour where it looks for hardware acceleration and if not found - falls back to software rendering ...

I updated the article line to this one

LIBGL_ALWAYS_SOFTWARE=true paraFoam

Anyway, do review and let me know what you guys decide. From what I read it seems F41 will completely get rid of xorg, Clearly this paraview available in dnf was compiled with some X libraries but how much it relies on xorg, I don't know.

ldd $(which paraview)
libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f5aad3d5000)

A nice start of the week ahead

@romangherta
The "Case Structure" section mentions blockMeshDict but I don't see that in the diagram for the tree structure.
The "Mesh" section refers to system/blockMeshDict but it doesn't appear in the tree structure, either.
However, the "Workflow" describes where these come from so I've added a note in both the cases above indicating this is explained later in the "Workflow" section. Please feel free to elaborate or clarify that. Someone following along might be looking for that file and note that it doesn't show up in the tree output you provide.

@glb Please take another look, if you have time.

I will try to describe the m sh with more details but it might take a
weekend or two

On Mon, Jun 10, 2024, 23:41 Richard England pagure@pagure.io wrote:

rlengland added a new comment to an issue you are following:
``
@romangherta
The "Case Structure" section mentions blockMeshDict but I don't see that
in the diagram for the tree structure.
The "Mesh" section refers to system/blockMeshDict but it doesn't appear in
the tree structure, either.
However, the "Workflow" describes where these come from so I've added a
note in both the cases above indicating this is explained later in the
"Workflow" section. Please feel free to elaborate or clarify that. Someone
following along might be looking for that file and note that it doesn't
show up in the tree output you provide.

@glb Please take another look, if you have time.
``

To reply, visit the link below or just reply to this email
https://pagure.io/fedora-magazine-newsroom/issue/72

Good morning, as a small update....

I have made some modifications so that case structure seems as generic as possible. I am afraid too much information will make this article even longer. The Mesh... you can write a lot of stuff about converting geometries between programs and refining, and this is already coevered in the docs....

My surprise is when on a final verification I am trying to run again the same tests single and parallel.... I cannot reproduce the same results.

For example when I run the case ansysToFoam I receive now a warning I did not observe before

Creating cells
--> FOAM Warning : 
    From Foam::polyMesh::polyMesh(const Foam::IOobject&, Foam::pointField&&, const Foam::cellShapeList&, const Foam::faceListList&, const Foam::wordList&, const Foam::wordList&, const Foam::word&, const Foam::word&, const Foam::wordList&, bool)
    in file meshes/polyMesh/polyMeshFromShapeMesh.C at line 640
    Found 3268 undefined faces in mesh; adding to default patch defaultFaces

So clearly something is wrong with this flange now.... Or maybe it is just a warning, so I move on. When running MPI I see the following errors

ofuser$ head -20 log.laplacianFoam 
[fedora:03588] PMIX ERROR: PMIX_ERROR in file client/pmix_client_topology.c at line 352
--------------------------------------------------------------------------
Open MPI's OFI driver detected multiple equidistant NICs from the current process,
but had insufficient information to ensure MPI processes fairly pick a NIC for use.
This may negatively impact performance. A more modern PMIx server is necessary to
resolve this issue.
Note: This message is displayed only when the OFI component's verbosity level is
-1386277808 or higher.
--------------------------------------------------------------------------
[fedora:03587] PMIX ERROR: PMIX_ERROR in file client/pmix_client_topology.c at line 352
[fedora:03588] PMIX ERROR: PMIX_ERROR in file client/pmix_client_topology.c at line 352
[fedora:03587] PMIX ERROR: PMIX_ERROR in file client/pmix_client_topology.c at line 352
[fedora][[45497,1],1][btl_tcp_component.c:1020:mca_btl_tcp_component_create_listen] bind() failed: Permission denied (13)
[fedora][[45497,1],0][btl_tcp_component.c:1020:mca_btl_tcp_component_create_listen] bind() failed: Permission denied (13

Then I have a cascade of selinux errors... I don t know what the hell is happening. I need some time to look into this and understand why is mpi throwing selinux errors.

A small update postupdate... In the article I added ofuser to group systemd-journal to be able to use journalctl.

Regarding PMIX error above... I was clearly running 3 mpi processes on a VM with 2 CPUs oversubscribing my resources.

As for selinux error... I took a look at some openmpi parameters

ompi_info --level 9  --param btl tcp
MCA btl tcp: parameter "btl_tcp_port_min_v4" (current value: "1024", data source: default, level: 2 user/detail, type: int)
                          The minimum port where the TCP BTL will try to bind (default 1024)
MCA btl tcp: parameter "btl_tcp_port_range_v4" (current value: "64511", data source: default, level: 2 user/detail, type: int)
                          The number of ports where the TCP BTL will try to bind (default 64511). This parameter together with the port min, define a 

So apparently the default port value is unprivileged. The problem must come from somewhere else. Lookig at journalctl logs these lines seem related

Jun 15 23:30:01 fedora audit[4665]: AVC avc:  denied  { create } for  pid=4665 comm="laplacianFoam" scontext=user_u:user_r:user_t:s0 tcontext=user_u:user_r:user_t:s0 tclass=netlink_rdma_socket permissive=0
Jun 15 23:30:01 fedora audit[4666]: AVC avc:  denied  { create } for  pid=4666 comm="laplacianFoam" scontext=user_u:user_r:user_t:s0 tcontext=user_u:user_r:user_t:s0 tclass=netlink_rdma_socket permissive=0
Jun 15 23:30:01 fedora audit[4665]: AVC avc:  denied  { create } for  pid=4665 comm="laplacianFoam" scontext=user_u:user_r:user_t:s0 tcontext=user_u:user_r:user_t:s0 tclass=netlink_rdma_socket permissive=0
Jun 15 23:30:01 fedora audit[4666]: AVC avc:  denied  { create } for  pid=4666 comm="laplacianFoam" scontext=user_u:user_r:user_t:s0 tcontext=user_u:user_r:user_t:s0 tclass=netlink_rdma_socket permissive=0
Jun 15 23:30:01 fedora audit[4665]: AVC avc:  denied  { name_bind } for  pid=4665 comm="laplacianFoam" src=5555 scontext=user_u:user_r:user_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0
Jun 15 23:30:01 fedora audit[4666]: AVC avc:  denied  { name_bind } for  pid=4666 comm="laplacianFoam" src=5555 scontext=user_u:user_r:user_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0

So my source context user_u is trying to access a target file labeled unreserved_port_t .... Checking permissions:

romh@fedora:~$ sudo sesearch -A -s user_t -t unreserved_port_t
allow nsswitch_domain port_type:tcp_socket { recv_msg send_msg }; [ nis_enabled ]:True
allow nsswitch_domain port_type:udp_socket recv_msg; [ nis_enabled ]:True
allow nsswitch_domain port_type:udp_socket send_msg; [ nis_enabled ]:True
allow nsswitch_domain unreserved_port_t:tcp_socket name_bind; [ nis_enabled ]:True
allow nsswitch_domain unreserved_port_t:tcp_socket name_connect; [ nis_enabled ]:True
allow nsswitch_domain unreserved_port_t:udp_socket name_bind; [ nis_enabled ]:True
allow user_usertype port_type:tcp_socket { name_connect recv_msg send_msg };
allow user_usertype port_type:udp_socket { recv_msg send_msg };
allow user_usertype unreserved_port_t:udp_socket name_bind;
allow user_usertype unreserved_port_type:tcp_socket name_bind; [ selinuxuser_tcp_server ]:True
allow user_usertype unreserved_port_type:udp_socket name_bind; [ selinuxuser_udp_server ]:True
sudo sesearch -A -s user_t -c netlink_rdma_socket
allow user_usertype application_domain_type:netlink_rdma_socket getattr

I am not allowed to create unreserved ports nor create rdma sockets... unless I am misreading the output...

Digging a bit more openmpi docs I found this
https://docs.open-mpi.org/en/v5.0.x/launching-apps/localhost.html#

Apparently MCA is something that defines a modular architecture and components that should be loaded are specified with --mca

Then we have a component called BTL (byte transfer layer) that has a lot of parameters and subcomponents and is charged with transfering data between shared memory amongst processes running on a single node.

Above docs page details the mca components that are required when running on localhost as

• self
• sm

these components can be appended with --mca likeso

mpirun --mca  btl self,sm  -np 2 laplacianFoam -parallel

This way the process runs, I see no errors.... but... I had to ammend the selinux paragraph... Please review and let me know... I was hoping there will be some selinux booleans to allow me to switch rdma and unreserved ports on....

As for the mesh... I had great plans, and I still want to understand it betterin my personal time, but I feel like for this article it will be too much... I tried to add the most common formats and a reference to snappyHexMesh but things are getting again complicated...

Please review in your next meeting and let me know

A nice weekend ahead

For the unreserved port permission, it looks like there is a boolean that will allow user_t to do that:

$ getsebool -a | grep selinuxuser_tcp_server
selinuxuser_tcp_server --> off

So running sudo setsebool selinuxuser_tcp_server on might grant your confined user the needed ability.

Indeed, than you. The netlink_rdma_socket errors still stands however. I think there is a value for the user in seeing those mpirun extra options and no journalctl error logs... I added a few line in selinux paragraph about this bool but I will not write it in the installation paragraph as an admin might not want to allow this access by default unless it is porvisioning in hpc... In this article we are still focuse on a local fedora workstation, the rest are just good to know... Anyway, feel free to edit .

@romangherta I am scheduling your article for publication on 19 June, this Wednesday (0800 UTC). If you find something that needs to be changed we can retract that and change it back to "draft" for you to edit.

Thanks for all your work.

I am sorry for the update, but there is a very small change I would like to do...

I need to ammend this command likeso

mpirun --mca pml ob1 --mca btl self,sm -np 2 \
laplacianFoam -parallel > log.laParallel

This is mentioned in ompi docs but I skipped that line

I failed to add this ob1 component... on a verbose check I noticed it uses by default the mtl cm component some extra description can also be found in this doc

No need to move it to drafpt, if you could ammend above command with --mca pml ob1 it would be enough... Thank you in advance and a nice week ahead.

I moved it back to draft. It is easy to reschedule it.

I edited the above mentioned line, you can schedule it back, thanks