Added Elasticsearch Emitter to index Frames into an ElasticSearch Index

[maheshbabugorantla]

ElasticSearch Emitter

Resolves Issue #392

Summary of Changes

1. Added Elasticsearch Emitter
2. Added UnitTest to test formatting of crawler frames into elasticsearch document format

---

Results

Environment Setup

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
e96b77434155        kibana:7.4.2          "/usr/local/bin/dumb…"   21 hours ago        Up 21 hours         0.0.0.0:5601->5601/tcp                           tender_dijkstra
a658b6357af5        elasticsearch:7.4.2   "/usr/local/bin/dock…"   6 months ago        Up 22 hours         0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   es_emitter_test

Crawling and Indexing frames in INVM Crawl Mode

$ sudo venv/bin/python crawler/crawler.py --url elastic://localhost:9200 --features os,disk,process,package --extraMetadata '{"iteration_number": 1, "hostname": "my_ubuntu_1804"}' --format json

$ sudo venv/bin/python crawler/crawler.py --url elastic://localhost:9200 --features os,disk,process,package --extraMetadata '{"iteration_number": 2, "hostname": "my_ubuntu_1804"}' --format json

Kibana Query (Filtering by extraMetadata fields)

hostname : "my_ubuntu_1804" and iteration_number : "1"

hostname : "my_ubuntu_1804" and iteration_number : "2"

[sahilsuneja1]

Plugin-specific dependencies should not be added to the global list, but installed inside the plugin itself.
Think about the case where one isn't using the emitter but yet ends up installing all the unnecessary packages.

[maheshbabugorantla]

@sahilsuneja1 Okay I made the change suggested and rolled back the global list of packages in requirements.txt file.

[sahilsuneja1]

Test environment setup failing. Maybe need to update https://apt.dockerproject.org/repo/dists/ubuntu-trusty/main/binary-amd64/ to https://download.docker.com/linux/ubuntu/dists/trusty/stable/binary-amd64/ in https://github.com/cloudviz/agentless-system-crawler/blob/master/.travis.yml#L22 as per moby/moby#16941 Will look into it later.

[sahilsuneja1]

I tried updating test environment setup but failed (#396). Error: /usr/bin/python: relocation error: /lib/x86_64-linux-gnu/libnss_compat.so.2: symbol __nss_database_lookup2, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference