Move away from Elasticsearch

[feanil]

Due to the fact that AWS is no longer supporting the latest versions of Elasticsearch, we are considering deprecating our usage of ES in favor of the AWS replacement, OpenSearch.

This deprecation is in the initial stages of discovery, so we wanted to solicit community feedback before moving too far along on it, so there currently is no acceptance date for this deprecation ticket.

Discussion Thread: [https://discuss.openedx.org/t/deprecation-removal-depr-170-move-from-elasticsearch-to-opensearch/5844](Discussion Thread on Discuss)

Comment from Diana:

We here at edx.org are going to look into what it would take to remove all ES or OS dependencies and then evaluate from there.

[dianakhuang]

Teams at edX/2U have done discovery on the work for this ticket, and we have decided to go forward with using OpenSearch for several key use cases. We will be removing usages of Elasticsearch or equivalent in favor of MySQL text search in all other use cases.

Switched to using OpenSearch:

• cs_comments_service
• blockstore
• edx-search (courseware search)

Removing usage of Elasticsearch:

• course-discovery
• edx-notes
• edx-analytics-api (endpoints are deprecated entirely. See [DEPR]: Remove Learner View in Insights, Data API, and Pipeline #36)

Because of this, I propose setting the acceptance date on this ticket for April 18, 2022 in order to give the community time to discuss this.

[jristau1984]

To be clear about Blockstore: Our investigation found that Blockstore does not currently leverage ElasticSearch at all. Our recommendation is to have the BD-14 Content Lib v2 project team implement another solution during the project which would remove the need for OpenSearch.

[jristau1984]

PT-CscommentsserviceESusage-170322-1900.pdf
PT-TNL-9545-ElasticsearchUsageandReplacement-170322-1900.pdf

These are the discoveries done by T&L and Infinity squads which led us to prefer an OpenSearch solution rather than a native MySQL solution.

[feanil]

@jristau1984 looking at the discoveries, it looks like courseware search of course content is not actually enabled in the new MFE. Is the reason for moving to opensearch that we're planning to port that feature to the MFE in the near future?

[jristau1984]

Yes, the plan is to re-implement this feature in the MFE versions of LMS and CMS when possible.

[ormsbee]

@dianakhuang, @feanil: Can this be moved to "Communicated" status, since there is a post about it?

[CodeWithEmad]

@feanil since we know what are the exact index names,

PT-CscommentsserviceESusage-170322-1900.pdf
PT-TNL-9545-ElasticsearchUsageandReplacement-170322-1900.pdf

could we modify them to be configurable? like an environment variable with the default value of the index name (or add prefix/suffix to the index name). I'm asking this because we talked here about Using one Elastic Cluster for different organizations.

[feanil]

@CodeWithEmad I'm not sure exactly what you're asking? I believe that it should be possible to update the code to make the index names configurable in a safe way. Are you asking how you should go about modifying the code to be able to make this configurable?(Most Open edX services are django and have an associated settings file so I would push for making the name be pulled from a Django setting rather than an environment variable for consistency with the rest of the system.)

[feanil]

@dianakhuang I'm gonna assign this ticket to you as the point person for this work that 2U is taking on.