how to print the region of an existent mapset?

[veroandreo]

Hi,

I'm baby-step trying to grasp how to do some simple grass stuff in actinia. I want to get the computational region of an existent mapset printed. I did not find an endpoint for that, so I went the ace way via grass terminal. But if I do the following

ace location="latlong_wgs84" mapset="worldpop_south_america" grass_command="g.region -p"

I get:

https://actinia.mundialis.de//api/v1/locations/latlong_wgs84/mapsets/worldpop_south_america/processing_async
{'version': '1', 'list': [{'module': 'g.region', 'id': 'g.region_1804289383', 'flags': 'p'}]}
veroandreo
Resource status accepted
Polling: https://actinia.mundialis.de/api/v1/resources/veroandreo/resource_id-3839dbbf-7de1-4e0d-abec-76ea852f6624
Resource poll status: error
AsyncProcessError:  Mapset <worldpop_south_america> exists in the global dataset and can not be modified.

Only, this works: ace location="latlong_wgs84" grass_command="g.region -p" but that's not the region I am interested in (btw, there's no report of which mapset the reported region corresponds to, I assume PERMANENT). I must be misunderstanding something, but how can I then get the region of an existent mapset printed?

I apologize in advance if this is not the channel to post these questions. I will happily subscribe to the one you point out :)

[mmacata]

Hi @veroandreo,

the right channel for this would be https://github.com/mundialis/ace/issues but closely related ;)

Supposely it doesn't work currently with ace. With plain actinia HTTP REST calls you can query https://actinia.mundialis.de/api/v1/locations/latlong_wgs84/mapsets/worldpop_south_america/info
The problem is that an existing global mapset cannot be used in persistent processing, Actinia provides this endpoint to overcome this issue but it is not imlemented in ace.

[veroandreo]

Hi Carmen!

Thanks for transferring the issue to the right place and for pointing out the right endpoint in terms of actinia HTTP REST calls.

I have further questions. What do you mean by "an existing global mapset cannot be used in persistent processing"? Why global? And why persistent process? Aren't the terms ephemeral and persistent referring to storage only, and sync and async to processes? Sorry if I'm slow with this, my only intention is understanding it properly to then be able to use actinia and contribute.

In the ace manual it says that for persistent process I should use -p. However I see now that flag p is no longer in the list of flags, so the sentence about that seems a left over (i can contribute a PR to fix that, but I see the topic was not fully defined yet: #3). What kind of processes does then ace allow to run?

And how to create my own location and mapsets to be able to play around? Once that is done (if possible), what is the recommended procedure to be able to actually use all those southamerica data splitted in mapsets within the latlong location in my workflows?

[mmacata]

Hi Vero,

I have further questions. What do you mean by "an existing global mapset cannot be used in persistent processing"? Why global? And why persistent process? Aren't the terms ephemeral and persistent referring to storage only, and sync and async to processes? Sorry if I'm slow with this, my only intention is understanding it properly to then be able to use actinia and contribute.

I mean with that that it cannot be manipulated during processing. Of course data from it can be copied and queried.

With global I mean the difference between the persistent grassdb and the user grassdb. Inside the persistent grassdb, data is stored and in theory available to every user (therefore global, even though local data can be included as well, but mostly datasets which cover a large area are inside). This data is read-only. I use the word global to distinguish from the userdata but persistent would fit better.

The terms ephemeral and persistent refer to storage but also to processing. The "real" processing endpoints where a process chain is send to are all asynchronous. This means that jobs are started and still running, even though the response will be received already (the json containing the status url). Some endpoints - e.g. the mapset info endpoint from above - is synchronous, meaning that the response already contains the result.

The two processing endpoints adress either a location and a mapset (e.g. https://actinia.mundialis.de//api/v1/locations/latlong_wgs84/mapsets/worldpop_south_america/processing_async) or a location only (e.g. https://actinia.mundialis.de//api/v1/locations/processing_async_export). In the first case the processing is persistent because it would store the result in the named mapset. Because in this example it is a (global) persistent mapset, it is not allowed to change it. This decision is done before processing starts, so actinia wouldn't analyse that a g.region command won't change anything but just return an error. Usually it is possible to do this type of processing with a user mapset. If it doesn't exist, it will directly be created by adressing e.g. https://actinia.mundialis.de//api/v1/locations/latlong_wgs84/mapsets/my_new_mapset/processing_async endpoint. A map from the persistent grassdb can be adressed in the process chain by mymap@mymapset but for g.region I fear it doesn't make much sense...
The second type, ephemeral processing, will not write the results back to the user mapset but export it for download.

In the ace manual it says that for persistent process I should use -p. However I see now that flag p is no longer in the list of flags, so the sentence about that seems a left over (i can contribute a PR to fix that, but I see the topic was not fully defined yet: #3). What kind of processes does then ace allow to run?

Yes you are right - it is still open to reactivate the -p flag if it makes things more obvious. Currently ace decides which endpoint to use depending if a mapset is given in the command or not, likewise the above endpoints determine it by a given mapset or none.

And how to create my own location and mapsets to be able to play around? Once that is done (if possible), what is the recommended procedure to be able to actually use all those southamerica data splitted in mapsets within the latlong location in my workflows?

A new location can only be created by admin role or above. A new mapset can be created by a user, e.g by adressing one in the persistent processing. I fear there is an inconsistency and it is not allowed for a user to create a mapset with the "official" mapset management endpoint.
A reprojection on-the-fly within an actinia process chain is not possible yet. The reason is that the process chain is analysed for needed mapsets and they are merged - different locations are not taken into account. For calculations inside the latlong location, they can be adressed in mymap@mymapset style.

Long answer - I hope it helps! Where do you think can we explain this better? In some documentation? Are you aware of the tutorial which also explains some concepts? Where would you look for this kind of documentation?

[veroandreo]

Hi Carmen,

Thanks for the detailed explanation, and apologies for my late answer.

I have further questions. What do you mean by "an existing global mapset cannot be used in persistent processing"? Why global? And why persistent process? Aren't the terms ephemeral and persistent referring to storage only, and sync and async to processes? Sorry if I'm slow with this, my only intention is understanding it properly to then be able to use actinia and contribute.

I mean with that that it cannot be manipulated during processing. Of course data from it can be copied and queried.

Right, I think the problem was that with role=user, I was not able to create a mapset of my own either (actinia-org/actinia-core#266), so, where to copy the data then?

With global I mean the difference between the persistent grassdb and the user grassdb. Inside the persistent grassdb, data is stored and in theory available to every user (therefore global, even though local data can be included as well, but mostly datasets which cover a large area are inside). This data is read-only. I use the word global to distinguish from the userdata but persistent would fit better.

I see. Maybe a diagram or so with the different types of db's and permissions for each type would be useful guidance. I didn't know those data were read only.

The terms ephemeral and persistent refer to storage but also to processing. The "real" processing endpoints where a process chain is send to are all asynchronous. This means that job> s are started and still running, even though the response will be received already (the json containing the status url). Some endpoints - e.g. the mapset info endpoint from above - is synchronous, meaning that the response already contains the result.

The two processing endpoints address either a location and a mapset (e.g. https://actinia.mundialis.de//api/v1/locations/latlong_wgs84/mapsets/worldpop_south_america/processing_async) or a location only (e.g. https://actinia.mundialis.de//api/v1/locations/processing_async_export). In the first case the processing is persistent because it would store the result in the named mapset. Because in this example it is a (global) persistent mapset, it is not allowed to change it. This decision is done before processing starts, so actinia wouldn't analyse that a g.region command won't change anything but just return an error. Usually it is possible to do this type of processing with a user mapset. If it doesn't exist, it will directly be created by adressing e.g. https://actinia.mundialis.de//api/v1/locations/latlong_wgs84/mapsets/my_new_mapset/processing_async endpoint.

I see. But again, it will be created only if the user has the right permissions, no?

A map from the persistent grassdb can be adressed in the process chain by mymap@mymapset but for g.region I fear it doesn't make much sense... The second type, ephemeral processing, will not write the results back to the user mapset but export it for download.

In the ace manual it says that for persistent process I should use -p. However I see now that flag p is no longer in the list of flags, so the sentence about that seems a left over (i can contribute a PR to fix that, but I see the topic was not fully defined yet: #3). What kind of processes does then ace allow to run?

Yes you are right - it is still open to reactivate the -p flag if it makes things more obvious. Currently ace decides which endpoint to use depending if a mapset is given in the command or not, likewise the above endpoints determine it by a given mapset or none.

And how to create my own location and mapsets to be able to play around? Once that is done (if possible), what is the recommended procedure to be able to actually use all those southamerica data splitted in mapsets within the latlong location in my workflows?

A new location can only be created by admin role or above. A new mapset can be created by a user, e.g by adressing one in the persistent processing. I fear there is an inconsistency and it is not allowed for a user to create a mapset with the "official" mapset management endpoint.

I see. So, role=user can only create a mapset using the processing endpoint but not the endpoint to create a mapset? Can the inconsistency be solved somehow? Or maybe document it.

Long answer - I hope it helps! Where do you think can we explain this better? In some documentation? Are you aware of the tutorial which also explains some concepts? Where would you look for this kind of documentation?

I was not aware of that tutorial, I noted it when Anika pointed it out in a different issue. Some of the things you explained here, are already there. So, maybe add some graphic content there and add the link to the actinia_core repo README for reference would be a good starting point. What I also imagined is a sort of wiki like grass wiki with full workflow examples, e.g., download, import, process and classify a time series of Sentinel images, create a time series of x variable, do aggregations, temporal algebra and query with a vector. My idea was to contribute in that but I found myself pretty limited in what I was able to do as user only. I haven't found time in the last couple of weeks to test much the new role.

[mmacata]

Hi Vero,

now it is my turn to apologize for the late answer.

A bit in short:

• every user should be allowed to create a mapset (https://actinia.mundialis.de/tutorial/actinia_concepts.html#user-user-roles-and-user-groups)

• I see. So, role=user can only create a mapset using the processing endpoint but not the endpoint to create a mapset? Can the inconsistency be solved somehow? Or maybe document it.

  Yes that is the problem and seems to be a bug. I opened a ticket for it.

I think that solves most of your questions above.

Regarding the documentation -I fully agree. I think the wiki with vivid examples is a great idea. I added the documentation label here as I guess the open questions here are mostly due to lack of documentation.